Can computers really mark exams? Benefits of ELT automated assessments

app Languages
Hands typing at a laptop with symbols

Automated assessment, including the use of Artificial Intelligence (AI), is one of the latest education tech solutions. It speeds up exam marking times, removes human biases, and is as accurate and at least as reliable as human examiners. As innovations go, this one is a real game-changer for teachers and students. 

However, it has understandably been met with many questions and sometimes skepticism in the ELT community – can computers really mark speaking and writing exams accurately? 

The answer is a resounding yes. Students from all parts of the world already take AI-graded tests.  aԻ Versanttests – for example – provide unbiased, fair and fast automated scoring for speaking and writing exams – irrespective of where the test takers live, or what their accent or gender is. 

This article will explain the main processes involved in AI automated scoring and make the point that AI technologies are built on the foundations of consistent expert human judgments. So, let’s clear up the confusion around automated scoring and AI and look into how it can help teachers and students alike. 

AI versus traditional automated scoring

First of all, let’s distinguish between traditional automated scoring and AI. When we talk about automated scoring, generally, we mean scoring items that are either multiple-choice or cloze items. You may have to reorder sentences, choose from a drop-down list, insert a missing word- that sort of thing. These question types are designed to test particular skills and automated scoring ensures that they can be marked quickly and accurately every time.

While automatically scored items like these can be used to assess receptive skills such as listening and reading comprehension, they cannot mark the productive skills of writing and speaking. Every student's response in writing and speaking items will be different, so how can computers mark them?

This is where AI comes in. 

We hear a lot about how AI is increasingly being used in areas where there is a need to deal with large amounts of unstructured data, effectively and 100% accurately – like in medical diagnostics, for example. In language testing, AI uses specialized computer software to grade written and oral tests. 

How AI is used to score speaking exams

The first step is to build an acoustic model for each language that can recognize speech and convert it into waveforms and text. While this technology used to be very unusual, most of our smartphones can do this now. 

These acoustic models are then trained to score every single prompt or item on a test. We do this by using human expert raters to score the items first, using double marking. They score hundreds of oral responses for each item, and these ‘Standards’ are then used to train the engine. 

Next, we validate the trained engine by feeding in many more human-marked items, and check that the machine scores are very highly correlated to the human scores. If this doesn’t happen for any item, we remove it, as it must match the standard set by human markers. We expect a correlation of between .95-.99. That means that tests will be marked between 95-99% exactly the same as human-marked samples. 

This is incredibly high compared to the reliability of human-marked speaking tests. In essence, we use a group of highly expert human raters to train the AI engine, and then their standard is replicated time after time.  

How AI is used to score writing exams

Our AI writing scoring uses a technology called . LSA is a natural language processing technique that can analyze and score writing, based on the meaning behind words – and not just their superficial characteristics. 

Similarly to our speech recognition acoustic models, we first establish a language-specific text recognition model. We feed a large amount of text into the system, and LSA uses artificial intelligence to learn the patterns of how words relate to each other and are used in, for example, the English language. 

Once the language model has been established, we train the engine to score every written item on a test. As in speaking items, we do this by using human expert raters to score the items first, using double marking. They score many hundreds of written responses for each item, and these ‘Standards’ are then used to train the engine. We then validate the trained engine by feeding in many more human-marked items, and check that the machine scores are very highly correlated to the human scores. 

The benchmark is always the expert human scores. If our AI system doesn’t closely match the scores given by human markers, we remove the item, as it is essential to match the standard set by human markers.

AI’s ability to mark multiple traits 

One of the challenges human markers face in scoring speaking and written items is assessing many traits on a single item. For example, when assessing and scoring speaking, they may need to give separate scores for content, fluency and pronunciation. 

In written responses, markers may need to score a piece of writing for vocabulary, style and grammar. Effectively, they may need to mark every single item at least three times, maybe more. However, once we have trained the AI systems on every trait score in speaking and writing, they can then mark items on any number of traits instantaneously – and without error. 

AI’s lack of bias

A fundamental premise for any test is that no advantage or disadvantage should be given to any candidate. In other words, there should be no positive or negative bias. This can be very difficult to achieve in human-marked speaking and written assessments. In fact, candidates often feel they may have received a different score if someone else had heard them or read their work.

Our AI systems eradicate the issue of bias. This is done by ensuring our speaking and writing AI systems are trained on an extensive range of human accents and writing types. 

We don’t want perfect native-speaking accents or writing styles to train our engines. We use representative non-native samples from across the world. When we initially set up our AI systems for speaking and writing scoring, we trialed our items and trained our engines using millions of student responses. We continue to do this now as new items are developed.

The benefits of AI automated assessment

There is nothing wrong with hand-marking homework tests and exams. In fact, it is essential for teachers to get to know their students and provide personal feedback and advice. However, manually correcting hundreds of tests, daily or weekly, can be repetitive, time-consuming, not always reliable and takes time away from working alongside students in the classroom. The use of AI in formative and summative assessments can increase assessed practice time for students and reduce the marking load for teachers.

Language learning takes time, lots of time to progress to high levels of proficiency. The blended use of AI can:

  • address the increasing importance of formative assessmentto drive personalized learning and diagnostic assessment feedback 

  • allow students to practice and get instant feedback inside and outside of allocated teaching time

  • address the issue of teacher workload

  • create a virtuous combination between humans and machines, taking advantage of what humans do best and what machines do best. 

  • provide fair, fast and unbiased summative assessment scores in high-stakes testing.

We hope this article has answered a few burning questions about how AI is used to assess speaking and writing in our language tests. An interesting quote from Fei-Fei Li, Chief scientist at Google and Stanford Professor describes AI like this:

“I often tell my students not to be misled by the name ‘artificial intelligence’ — there is nothing artificial about it; A.I. is made by humans, intended to behave [like] humans and, ultimately, to impact human lives and human society.”

AI in formative and summative assessments will never replace the role of teachers. AI will support teachers, provide endless opportunities for students to improve, and provide a solution to slow, unreliable and often unfair high-stakes assessments.

Examples of AI assessments in ELT

At app, we have developed a range of assessments using AI technology.

Versant

The Versant tests are a great tool to help establish language proficiency benchmarks in any school, organization or business. They are specifically designed for placement tests to determine the appropriate level for the learner.

PTE Academic

The  is aimed at those who need to prove their level of English for a university place, a job or a visa. It uses AI to score tests and results are available within five days. 

Read more about the use of AI in our learning and testing here.

Explore our courseware and exams

More blogs from app

  • A young woman taking notes in a lecture theatre, she is sat by other young people.

    Preparing your learners for university study abroad

    By Richard Cleeve

    Whether your learners are going for a single semester, academic year or an entire university course, studying abroad is an excellent opportunity for them. They’ll have the chance to discover a new culture, develop new skills and make new friends.

    University study in another country also poses several challenges. But as a teacher, you can equip them for this experience and prepare them for future academic success.

    Why study abroad?

    Most people think that studying at university is hard enough, without the added difficulty of doing it overseas. But that doesn’t stop hundreds of thousands of university students from leaving the support of family and friends and relocating to a foreign country.

    People apply to study in another country for a range of reasons. A university program abroad might offer the student better tuition and a greater promise of future employment or simply represent better value for money. And in the case of very specialist university courses, studying abroad may be the only option.

    Whatever the reason, the decision to study in a foreign country is likely to involve a high level of proficiency in another language – and more often than not, that language is English.

    A move towards English language in higher education

    There has been a significant shift in higher education in the last ten years, as many European institutions look to internationalize their programs. As a result, across Europe, we have seen a sharp growth in the number of university courses taught in English. English-taught bachelor’s programs offered by universities in the have multiplied dramatically over the last decade.

    What challenges do learners face?

    Academic skills

    There are a whole range of academic skills that students are expected to know when they start university. From research and evaluation, to note-making and referencing, many learners will enter higher education lacking many of the essential skills they require.

    Studying in a foreign language

    Not only will they have to master new skills, but they may need to do them in a second language. What’s more, even everyday things that fluent speakers may take for granted, such as understanding lectures, reading academic papers, writing essays and even socializing with new friends, will take a lot more effort if English isn’t your first language.

    Administrative issues

    There are many potential pitfalls for a student in a new academic setting. From the administrative process and campus regulations to the types of lessons and assessments, there may be a lot of differences to deal with. Even understanding the etiquette of addressing and interacting with professors can be daunting.

    Problems integrating

    Another challenge is integrating into another culture. Even if the host country is culturally similar, adapting to new surroundings is not always straightforward. There can also be a certain amount of ghettoization, where international students might stick together and remain isolated from the local student population.

    Homesickness

    for international students to deal with. Depending on how far they travel to study, your learners may be unable to return home easily, visit their families and alleviate their homesickness.

    Mental health

    Moving abroad and living in a completely new place can be very stressful and overwhelming, and many factors can exacerbate/cause . Making it harder to do day-to-day tasks, socialize and study.

    Money worries

    Without a grant or a scholarship, studying abroad can be very expensive. If your learners currently live at home with their parents, the cost of accommodation may be formidable. The higher cost of living could mean they have to look for a part-time job to supplement their income. Understanding a country's can also be confusing and hard to calculate into their budget.

    What can you do to get your students ready?

    All of the challenges mentioned above have one thing in common. If a student cannot communicate effectively, these situations can be exacerbated. Language is key, whether it’s accessing support, communicating with professors or getting to grips with a new culture.

    Here are some things you can do to help your learners prepare for university life:

    1) Put them in touch with past students

    It’s important that your learners have a clear idea of what university study abroad entails. Creating a chance for them to speak to other students who have already gone through that experience can be extremely valuable.

    Students who have returned from studying abroad can help with your learners' doubts and put their minds at rest. They might be able to provide essential advice about a specific country or university or simply tell their story. Either way, it’s a great way to reassure and encourage your learners.

    2) Use appropriate authentic content

    In preparation for your learner’s time abroad, the language course that you teach should align with their future linguistic needs. One of the main aims should be to develop the language skills required to perform successfully and confidently in their new context.

    3) Teach them academic study skills

    Think back to when you were at university and what you struggled with. Group work, presentations, critical thinking and exam skills are all things which your learners will need to be proficient in, so the more you practice them in class the better.

    4) Promote autonomous learning

    Success at university is deeply rooted in a student’s ability to work independently and develop practical self-study skills. Giving your learners more choice in the language learning process is one way to encourage autonomy.

  • A business woman in a suit sat at a laptop

    6 tips for teaching business English to low level learners

    By Margaret O'Keeffe

    The CEFR describes A1 and A2 learners as ‘basic users’ of a language. So how can we help these students to develop their English for the workplace?

    Here are our six top tips:

    1. Focus on high-frequency vocabulary for work

    Learning English vocabulary for work context is the top priority for many low-level learners in business English classes. It helps them to communicate their message in a simple, effective way. This makes it important to teach common words and set expressions for everyday work situations.

    These include:

    • lexical sets (words related to the same topic or situation) – for example, days, months, numbers, verbs to describe work routines, verbs in the past.
    • common collocations with verbs and nouns (for example, manage a team, have meetings, place an order, solve a problem).
    • functional language and fixed phrases – greetings (How are you? Nice to meet you.) and offers (How can I help you? Would you like…?).

    2. Help students with vocabulary learning

    Teach vocabulary items in realistic contexts. For example, phone calls, to-do lists, short emails, text messages etc.

    While it might be tempting to give students lots of vocabulary to memorize, this can cause overload, be frustrating and ultimately demotivating for learners. Instead, you should aim to present eight to ten new words in a lesson as a general rule. This is an achievable number for working memory and helps to build learners’ confidence. The number of words can be a little higher if items are easy to show in images or there is repetition; for instance, the numbers 20 to 100.

    Have students make simple decisions about new words, as this helps with recall later. Start with simple tasks, such as matching words and pictures or verb and noun collocations they’ve seen in a short text (for example, managing a team, call customers, writing emails, etc.). Next, ask students to complete sentences using the target words and write their own sentences using these words.

    Getting students to personalize new vocabulary makes it more memorable, for instance writing sentences describing their work routines. Repetition also aids long-term memory, so make sure vocabulary is recycled in the materials in later lessons.

    Finally, make a list of vocabulary games to use for revision exercises, warmers and to finish classes.

    3. Maximize student speaking time

    Learners need to develop their English-speaking skills for work. The classroom is a safe, low-stakes environment for them to gain fluency and confidence.

    Use the audio and video scripts of short dialogues or an extract from a longer script. Students read the dialogue aloud in pairs or groups. Give feedback by drilling the stress and rhythm of any words or phrases which were difficult with the whole class. Back-chaining phrases – starting with the last sound and building up going backwards – is an excellent way to drill. Get students to swap roles and repeat the task.

    You can also use another technique called disappearing dialogue. Put a short dialogue on the board for students to practice in pairs. Then delete parts of the dialogue and ask them to repeat the task, swapping roles each time. Gradually delete more parts to increase the challenge. Students can reconstruct the dialogue as a final task.

    Moreover, surveys, questionnaires, true/false games, and information-gap exercises are ways to practice speaking in English, target structures, and vocabulary.

  • A Teacher sat with a child at a desk in a classroom helping them with their writing,

    5 ways to support students with dyslexia

    By Anna Hasper

    Children seem to be starting English lessons younger than ever, often before they can even read and write. This means that learning differences like dyslexia may not have yet made themselves apparent.

    While it’s not a language teacher’s role to diagnose specific learning needs, it is important for us to monitor our young learner students’ progress. If we think a student might be showing signs of dyslexia (or another learning difference), we should feel comfortable referring parents to the right place early on. This can make a huge difference in the learning process.

    There are many forms of dyslexia and it affects students in various ways. However, some may include the following:

    • having difficulty reading (especially aloud)
    • struggling with spelling
    • problems remembering the sequence of things
    • finding it hard to follow instructions
    • misbehaving or disrupting the class
    • being very quiet or shy (especially when doing reading or writing activities)
    • falling asleep in class.

    Dyslexia is not a learning disability; it’s a learning difference.

    What do Magic Johnson, Richard Branson and Tom Cruise have in common? They all have dyslexia. So learners with dyslexia are certainly not less capable; in fact, they often excel in spatial thinking and creativity. The difference is that their brain works differently, so they find visual processing and using their working memory challenging. For example, they may struggle to remember what was said and face challenges when trying to link sounds to letters.

    The most common issues are related to reading, spelling and writing, but dyslexia can also impact concentration span and planning skills. And all these challenges have a severe impact on learners’ self-esteem.

    Providing effective learning opportunities for young learners with dyslexia might require teachers to reframe how they see dyslexia. Avoid seeing it as a dis-ability, but rather as a form of neurodiversity: the brain functions and learns in different ways.

    Creating the conditions for learning

    Many – if not most – young learner teachers feel they are not appropriately trained to deal effectively with learners who have dyslexia in a classroom context.

    In an ideal world, all EAL and mainstream teachers would receive in-depth training to better deal with neurodiversity in the classroom. But for now, let’s explore some modifications that help create a more enabling learning environment in which all learners – with or without dyslexia – can progress.

    1. Getting to know them

    If we want all learners to progress to their next level, we need to get to know them. Only then can we provide learning opportunities that start where they are. Get to know their strengths, weaknesses and interests as well as their learning profile; where do they like to work, who do they work well with and what kinds of tasks engage them fully? These are the starting principles of differentiated teaching and all learners will profit from you taking the time to get to know them beyond their name.

    Top tip:

    Observations are an extremely useful tool to gain insight into learners’ levels and learning preferences. My favorite activity is to get young learners to create a personal profile.

    This can be done in their first language – at home with parents – or as a shared writing activity in class. You provide the stem sentences, and learners complete them with drawings or words. You can hang the profiles on the wall and use them to start talking about ‘differences and similarities’. Alternatively, you can have a learner present their buddy to the class based on their profile, depending on the level and age you teach.

    2. Creating a collaborative culture in the classroom

    If we want learners to help each other in class, we need to create a culture of ‘helping hands’. Focusing on developing good relationships in your classroom, between you and the learners but also between learners, is vital for a collaborative culture. Use activities that focus on building understanding through sharing ideas. Integrating collaborative learning activities will help to establish supportive relationships and makes struggling learners feel more confident in the classroom. They know they can first talk things through with others and ask them for help before completing a task independently. This will benefit all learners, not only learners with dyslexia.

    Top tip:

    Think-pair-share is a well-known collaborative activity and can easily be adapted to include some movement too in the form of HuSuPuWu!

    This activity will help learners share ideas and allow for differentiated thinking time. Ask your young learners a question you want them to respond to, give them thinking time and tell them to put their hand up when they are ready to talk (Hu).

    Encourage them to look around, find another person with their hand up and stand up (Su) to walk over and pair up (Pu).

    Together they share ideas before returning to their place and writing up their ideas (Wu).

    This will be especially beneficial for students who need more time to process, love to move and want to get confirmation or support.

    3. Providing multi-sensory tasks and activities

    Providing multi-sensory activities is already common practice in most young learner classrooms. It allows learners to process information using their stronger senses while strengthening their weaker areas.

    Multi-sensory teaching (MST) acknowledges that all brains learn in unique, different ways and is a well-known method used when working with dyslexic students in their mother tongue. So instead of only telling the story, find images that illustrate the events, draw a story path for learners to follow, or get them to visualize the story.

    Doing this increases the ‘routes of memory’ as Kormos (2017) calls it, and enables information to reach the brain via different pathways, visual and auditory, which strengthen the message.

    Top tip:

    When learning new words, break them into syllables by clapping when you say them. Then show the word and break it up visually (e.g. fri-end), and get them to make the word with playdough or in shaving foam as they say it. Get them to keep saying it as they write it and then check it.

    4. Setting clear, manageable instructions

    Because dyslexia often impacts working memory, following instructions can be even more challenging than it already is for young learners. We need to reduce the processing load by breaking up instructions into manageable, achievable steps.

    Focusing on just a small amount of information better enables learners with dyslexia (Kormos & Smith, 2012) and to be honest, all young learners – and our classroom management – can benefit from this.

    Also, check whether you need to ‘tell’ it or can you ‘show’ the instructions? Presenting instructions in a multisensory way where you, for example, use the whiteboard to visualize the instructions, and use gestures and body language to support your oral input will facilitate understanding.

    Top tip:

    Learners benefit from talking things through as talk plays an integral part in meaning-making. So why not get learners to turn to their elbow buddy and repeat what they need to do in their own words? Another effective way would be to record the instructions so they can listen back as many times as they need.

    5. Adapting your materials

    Being aware of what works best for the unique brains of learners with dyslexia allows us to tweak existing materials to make learning more accessible. Think about the color of paper you copy on or the background color of your slides. Learners with dyslexia cope better with colored backgrounds as it reduces word blurring. When learning to write new words in their workbook, use a highlighter to highlight the area between the middle lines where the body of the letters needs to be written.

    Top tip:

    Nowadays, many young learner coursebooks have audio resources available, but not always for readers or stories. Use assistive technology to get the selected reading text recorded. Struggling readers can listen to the audio as they read the text alone. In this way, they will feel that they are reading independently whilst working on letter sound correlation as well as the rhythm of the language.

    The English language classroom can be stressful for learners with specific learning needs. Now, we don’t need to – and can’t – ‘fix’ learners, but we should try to ‘fix’ the environment and provide an enabling, inclusive learning environment for all. By tweaking our teaching, we might better enable learners who face challenges, ensure they feel supported in their learning and allow them to bloom in our classroom.