Designing transparent assessment strategies for online learning in HE (English language)

1. Good practice examples at Vytautas Magnus University in Lithuania

At Vytautas Magnus university, teachers incorporate and use a variety of different Moodle tools to assess student learning outcomes. For example, the Moodle learning environment provides teachers with opportunities to perform assessments of the intended learning outcomes, e.g., teachers can create tests, quizzes, use interactive tools for assessment, (e.g., H5P tool (see Illustration 1), etc.

Illustration 1. Example of a Test Segment, created using H5P tool (VMU Moodle environment)

One of the advantages of using the H5P tool is that the students have an opportunity to listen to the recording of the lecture, see the lecture slides or any other related material or links for further information, and then perform a self-check test. Also, the H5P tool and tests created using this tool can be used as part of either a formative or summative assessment strategy, depending on what purpose the test is supposed to serve. Overall, it should be stated that this tool enables a more interactive approach to learning and assessment. 

Also, teachers can explore the Moodle environment for designing different tests for their students. The Moodle environment offers a wide range of test items that can be used to assess a student’s content knowledge, skills, and competence. Teachers can design various tests by using multiple choice questions, true/false statements, fill in the blank questions, drag and drop matching items, etc.

Illustration 2. Examples of different Question Items (VMU Moodle environment)

Moreover, Moodle equips teachers with possibilities to assess more intrinsic skills. For instance, by designing open, essay type questions teachers can assess a student’s understanding of the topic and writing skills as well as critical thinking skills (see Illustration No. 3 below).

Illustration 3. Example of an Essay Type Question Item (VMU Moodle environment)

Besides, if the test is not a suitable format for assessing a student's skills, knowledge and competence, the teacher has an option to assign tasks on Moodle and ask students to submit papers in the Moodle environment through the paper submission tool (see Illustration No. 4). The teacher can provide a precise description of the assignment, give evaluation criteria, and set the due dates. Once the student submits their work through the system, the teacher can grade and provide feedback on the paper in Moodle without any need to first download the submitted papers. 

Illustration 4. Example of Assignment Submission Tool (VMU Moodle environment)

In short, teachers can use Moodle tools to create tests to check students’ knowledge on a certain topic and to evaluate whether the intended learning outcomes have been reached. Here, just a couple of different options have been described, however, a note should be made that there are several different tools that can be used for both assessment of student learning as well as assessment of learning outcomes. Also, a fact should be considered that external tools can be compatible and integrated within the Moodle environment, thus, teachers have a wide selection of tools to implement the selected assessment strategy.

2. Using Chatbox and AI teaching assistant

Using a text-based or voice-based conversational interface to communicate with the user, chatbots can answer and ask questions, guide learners, and assist in problem solving. This means that, when a teacher is not available or cannot help, learners are still able to make some progress. Increasingly, chatbots use artificial intelligence techniques to understand human languages, voices, body language and behaviors, and to make sense of patterns in languages or behaviors. Chatbots have brought opportunities when tackling the contradiction between large-scale and high-quality in learning. They enable greater personalization by collecting data from dialogues and learners’ behaviors to provide support that is specifically tailored to each learner’s requirements, which might also help reduce educators’ workloads. These tools, based on artificial intelligence (AI), could enable language practice via simple activities like asking and answering questions, through to more advanced conversation designs, such as enabling a learner to participate in a story by responding to choices offered by the chatbot. Chatbots are currently mainly designed for individual interactions, but in future they could support more collaborative dialogues.

In educational practice, researchers from Georgia Institute of Technology have investigated how chatbots can be used in online classes. They developed an AI teaching assistant called ‘Jill Watson’ based on dialogues and learning data from previous courses. With these data, Jill could analyze learners’ questions and come up with immediate responses. The chatbot has been used on several courses to help learners with content-related questions and through meaningful dialogues. Researchers found that learners could not distinguish the chatbot from the teacher, which suggests that in some contexts chatbots could work well as online learning facilitators.

Researchers from the Advanced Innovation Center for Future Education (AICFE) at Beijing Normal University have investigated the role of a chatbot in moral education. In moral education, teachers focus on helping learners understand moral problems and cultivate morality. In this research, an AI-bot (AI-powered chatbot) was developed to detect learners’ moral problems via dialogues and provide learners with adaptive solutions. For example, when the learner expressed negative emotions, the AI-bot would diagnose the cause by chatting. After that the AI-bot would assess whether the learner had experienced unfair treatment. Then it would suggest some options and give examples of how to deal with their issue. The results showed that the AI-bot could mimic teachers with 8–9 years’ experience.

In the field of language learning, chatbots can be used for informal conversation and pronunciation practice, which some learners prefer as it enables them to try out different ways of saying something and to avoid feelings of embarrassment (affective filter) when speaking.

In a foreign language, researchers have also used chatbots in education to support collaboration. They designed and trialed a game-based collaborative problem-solving ability assessment tool, ‘Circuit Runner’, which demonstrated the potential of chatbots in assessing higher-level skills in education. Finally, researchers have shown that chatbots can also be creative. For example, a chatbot has been developed which could help generate high-quality quizzes based on existing materials.

Researchers investigating the role of chatbots in real learning contexts acknowledge that the application of chatbots is still at a very early stage. Chatbots cannot work effectively without understanding human learning mechanisms. To bridge the gap between the techniques behind the chatbots and human learning mechanisms, learning design is necessary. Most learning designs could be enhanced with more detailed information about learners and knowledge of learning or teaching. AI techniques have created chatbots with the capacity to collect information and explore learners’ requirements before providing the learners with smart learning environments and adaptive support. Chatbots could be a new way to achieve learner-centered instruction. Furthermore, this will help reduce educators’ workloads.

Compared to traditional learning and teaching, chatbots bring new opportunities, such as immediate problem diagnosis and interventions which make the learners feel they are not alone during the learning process. Learners might also be more relaxed and express themselves more freely as they are not interacting with humans who might judge them. This relaxed environment can be conducive to promoting learning.

3. Other Online Formative Assessment Tools

Bolton College moved beyond closed questions to explore whether students can provide answers and receive automated feedback based on model answers provided by teaching staff. Staff have been exploring the potential that natural language processing and natural language classifications platforms have to offer from the leading vendors in the field such as IBM, Amazon, Google and Microsoft. Initial results are promising, with positive feedback received from students and teachers. Students liked receiving real-time feedback as they responded to open-ended questions, and teachers stated that these services could lead to a reduction in marking workloads.

The emergence of this new assessment tool enabled Bolton College’s teachers to make use of a richer medium for assessing their students. Traditionally, online formative assessment activities are undertaken using closed questioning techniques such as yes/no questions, multiple-choice questions, or drag-and-drop activities. Their solution for formative assessment enabled teachers to pose open-ended questions which can be automatically analyzed and assessed by a computer. The ability to offer real-time feedback means that students can qualify and clarify their responses.

It is important to note that teachers play an important role. They train the classification models that underpin the open-ended questions that they want to present to their students. Teachers may also welcome the fact that the accuracy of the classification models improves as more students engage with each open-ended question and as the volume of training data rises.      

A clear concern is the extent to which technology might reduce student/academic staff interaction, and to what extent human judgement could or should be replaced entirely – hence the need for ‘appropriately’ automated assessment. Students at Bolton College expressed a preference for tutor feedback alongside the automated feedback.

For essay-writing students who want instant feedback on style, if not content, écree (ecree.com) describes itself as “your personal AI writing tutor” and provides real-time writing feedback, spelling, and grammar support, including whether work is well structured as an academic essay with developed points and a strong conclusion. The aim is to help students improve their essays before their tutor sees them, thereby saving time marking up basic errors and allowing more time to focus on feeding back on the content and learning points.

Similarly to Bolton College, the universities of Edinburgh, Glasgow and Manchester have used Adaptive Comparative Judgment (ACJ) technology. Comparative judgement works on the principle that the human brain finds it easier to compare two items and decide one is better or worse than the other than to make an objective assessment about the quality of either against a given rubric. ACJ uses technology to automate the comparison process. While scripts are initially compared randomly, the adaptive element comes in as the computer algorithm starts to select the pairs that will most improve the reliability of the ranking. Comparison between very good and very bad scripts is obvious and more effort goes into assessing those that are more closely matched. It speeds up the marking process and the ranking can be used to grade papers and determine grade boundaries. It also enables students to receive feedback from multiple different markers whether staff or peers.