Can computers really mark exams? Benefits of ELT automated assessments

app Languages
Hands typing at a laptop with symbols

Automated assessment, including the use of Artificial Intelligence (AI), is one of the latest education tech solutions. It speeds up exam marking times, removes human biases, and is as accurate and at least as reliable as human examiners. As innovations go, this one is a real game-changer for teachers and students. 

However, it has understandably been met with many questions and sometimes skepticism in the ELT community – can computers really mark speaking and writing exams accurately? 

The answer is a resounding yes. Students from all parts of the world already take AI-graded tests.  aԻ Versanttests – for example – provide unbiased, fair and fast automated scoring for speaking and writing exams – irrespective of where the test takers live, or what their accent or gender is. 

This article will explain the main processes involved in AI automated scoring and make the point that AI technologies are built on the foundations of consistent expert human judgments. So, let’s clear up the confusion around automated scoring and AI and look into how it can help teachers and students alike. 

AI versus traditional automated scoring

First of all, let’s distinguish between traditional automated scoring and AI. When we talk about automated scoring, generally, we mean scoring items that are either multiple-choice or cloze items. You may have to reorder sentences, choose from a drop-down list, insert a missing word- that sort of thing. These question types are designed to test particular skills and automated scoring ensures that they can be marked quickly and accurately every time.

While automatically scored items like these can be used to assess receptive skills such as listening and reading comprehension, they cannot mark the productive skills of writing and speaking. Every student's response in writing and speaking items will be different, so how can computers mark them?

This is where AI comes in. 

We hear a lot about how AI is increasingly being used in areas where there is a need to deal with large amounts of unstructured data, effectively and 100% accurately – like in medical diagnostics, for example. In language testing, AI uses specialized computer software to grade written and oral tests. 

How AI is used to score speaking exams

The first step is to build an acoustic model for each language that can recognize speech and convert it into waveforms and text. While this technology used to be very unusual, most of our smartphones can do this now. 

These acoustic models are then trained to score every single prompt or item on a test. We do this by using human expert raters to score the items first, using double marking. They score hundreds of oral responses for each item, and these ‘Standards’ are then used to train the engine. 

Next, we validate the trained engine by feeding in many more human-marked items, and check that the machine scores are very highly correlated to the human scores. If this doesn’t happen for any item, we remove it, as it must match the standard set by human markers. We expect a correlation of between .95-.99. That means that tests will be marked between 95-99% exactly the same as human-marked samples. 

This is incredibly high compared to the reliability of human-marked speaking tests. In essence, we use a group of highly expert human raters to train the AI engine, and then their standard is replicated time after time.  

How AI is used to score writing exams

Our AI writing scoring uses a technology called . LSA is a natural language processing technique that can analyze and score writing, based on the meaning behind words – and not just their superficial characteristics. 

Similarly to our speech recognition acoustic models, we first establish a language-specific text recognition model. We feed a large amount of text into the system, and LSA uses artificial intelligence to learn the patterns of how words relate to each other and are used in, for example, the English language. 

Once the language model has been established, we train the engine to score every written item on a test. As in speaking items, we do this by using human expert raters to score the items first, using double marking. They score many hundreds of written responses for each item, and these ‘Standards’ are then used to train the engine. We then validate the trained engine by feeding in many more human-marked items, and check that the machine scores are very highly correlated to the human scores. 

The benchmark is always the expert human scores. If our AI system doesn’t closely match the scores given by human markers, we remove the item, as it is essential to match the standard set by human markers.

AI’s ability to mark multiple traits 

One of the challenges human markers face in scoring speaking and written items is assessing many traits on a single item. For example, when assessing and scoring speaking, they may need to give separate scores for content, fluency and pronunciation. 

In written responses, markers may need to score a piece of writing for vocabulary, style and grammar. Effectively, they may need to mark every single item at least three times, maybe more. However, once we have trained the AI systems on every trait score in speaking and writing, they can then mark items on any number of traits instantaneously – and without error. 

AI’s lack of bias

A fundamental premise for any test is that no advantage or disadvantage should be given to any candidate. In other words, there should be no positive or negative bias. This can be very difficult to achieve in human-marked speaking and written assessments. In fact, candidates often feel they may have received a different score if someone else had heard them or read their work.

Our AI systems eradicate the issue of bias. This is done by ensuring our speaking and writing AI systems are trained on an extensive range of human accents and writing types. 

We don’t want perfect native-speaking accents or writing styles to train our engines. We use representative non-native samples from across the world. When we initially set up our AI systems for speaking and writing scoring, we trialed our items and trained our engines using millions of student responses. We continue to do this now as new items are developed.

The benefits of AI automated assessment

There is nothing wrong with hand-marking homework tests and exams. In fact, it is essential for teachers to get to know their students and provide personal feedback and advice. However, manually correcting hundreds of tests, daily or weekly, can be repetitive, time-consuming, not always reliable and takes time away from working alongside students in the classroom. The use of AI in formative and summative assessments can increase assessed practice time for students and reduce the marking load for teachers.

Language learning takes time, lots of time to progress to high levels of proficiency. The blended use of AI can:

  • address the increasing importance of formative assessmentto drive personalized learning and diagnostic assessment feedback 

  • allow students to practice and get instant feedback inside and outside of allocated teaching time

  • address the issue of teacher workload

  • create a virtuous combination between humans and machines, taking advantage of what humans do best and what machines do best. 

  • provide fair, fast and unbiased summative assessment scores in high-stakes testing.

We hope this article has answered a few burning questions about how AI is used to assess speaking and writing in our language tests. An interesting quote from Fei-Fei Li, Chief scientist at Google and Stanford Professor describes AI like this:

“I often tell my students not to be misled by the name ‘artificial intelligence’ — there is nothing artificial about it; A.I. is made by humans, intended to behave [like] humans and, ultimately, to impact human lives and human society.”

AI in formative and summative assessments will never replace the role of teachers. AI will support teachers, provide endless opportunities for students to improve, and provide a solution to slow, unreliable and often unfair high-stakes assessments.

Examples of AI assessments in ELT

At app, we have developed a range of assessments using AI technology.

Versant

The Versant tests are a great tool to help establish language proficiency benchmarks in any school, organization or business. They are specifically designed for placement tests to determine the appropriate level for the learner.

PTE Academic

The  is aimed at those who need to prove their level of English for a university place, a job or a visa. It uses AI to score tests and results are available within five days. 

Read more about the use of AI in our learning and testing here.

Explore our courseware and exams

More blogs from app

  • A group of university students outside smiling

    International students: Which UK university is best for you?

    By

    People study abroad for many reasons - to see new places, to experience a different culture, and, most importantly, to get access to student life and educational opportunities not available at home. If you want to study at a UK university, you’ll not only have access to some truly innovative teaching and research, you’ll also be joining one of the most multicultural student bodies in the world.

    The UK has a vast range of universities and courses to choose from. So, how do you choose between over 160 institutions and thousands of courses? that half of students who drop out of university early do so because they chose the wrong course, so it’s important to be completely happy with your choice.

    So, here's what you need to consider before you make your university application so you can make the right choice.

    Choose your subject

    The first step in choosing a university is deciding what you want to study. Your will be sent to all your university choices, so it should be relevant to all the courses you apply for. This means that your course choices must be the same or in a similar field.

    Think about the school subjects you’re good at, but also think about what you could spend the next three or even four years studying.

    Is there a subject that you want to continue and deepen your knowledge in? Or do you want to study something completely new?

    Do you want to study a subject like law or medicine related to a specific job? Or do you want to take a course like history that can be used in a wide range of careers? Think carefully about these questions and the rest of the process will become much more manageable.

    Do as many events and online taster sessions, and sign up for as many free online courses as possible. has a great range of taster courses from universities all over the world. This can help you decide where your interests lie. Above all, keep notes on what interests you and what doesn’t.

    If you think you want a broader degree, then you’ve always got the option of doing a joint honors degree. This means you combine two different courses. Many joint honors courses combine a language with another subject, allowing you to use your learning around the world or even do a study placement in another country.

    Consider the competition

    When thinking about what course you could do, it’s important to consider the competition. Some courses have many more applicants than places, but there are also hundreds of ‘hidden’ courses that get far fewer applicants. For example, the University of Oxford receives 16.9 applicants per place on its Economics course, but it’s closer to 3 applicants per Classics place. Classics covers a range of areas like politics, history, linguistics and social sciences, and really challenges your intellect.

    Economics might be competitive but Classics might be just as valuable. There are thousands of ‘hidden’ courses, which don't attract as many applicants just because they’re not well known. Check out the full range of courses on the site and the and do it with an open mind.

    Once you’ve narrowed down your list of courses, it’s time to think about where to study.

    Think about location and compare places

    The UK might be a small country but it’s incredibly diverse. There are large, international cities like London or Glasgow, smaller towns and cities like Cardiff, Belfast or Nottingham, rural universities like Lancaster, coastal towns like St. Andrews or historic cities like York, Oxford or Cambridge.

    So, when deciding on a UK university, you must first ask yourself: what’s important to you? Would you prefer to live in a busy city or the quiet countryside? Will you be doing a lot of exploring or will you stay near campus? These factors will make a difference to your decision.

    You’ll also need to think carefully about costs. Different parts of the UK have different living costs, depending on things like rent, transport and the cost of entertainment. Check out to see what rents are like, to get an idea of how expensive life will be.

    Finally, the type of university itself is important. Some universities have campuses where the whole of university life takes place, like the . Others are more spread out, such as the , which has buildings all over the city. Each has its own atmosphere and you should think about which you prefer.

    Look at the university rankings

    Find out how your preferred universities rank in the league tables. Rankings of the top universities in the UK don’t just measure teaching quality. They show data on student satisfaction, post-graduation employment rates and staff-to-student ratios.

    The best universities in the UK and courses with higher rankings will be more competitive to gain entry to. You can apply for up to five courses on your UCAS form, so choosing a range of courses and universities is a good idea. Include an ambitious choice which you may or may not get the grades for and safe choices that you know will accept you.

    You can check the , or the to see which universities are the most highly rated.

    That being said, student life in the UK is about balance. You need to consider both rankings and what’s right for you. It’s no use going to a top-ranking UK university in a location you hate, studying things that don’t interest you and where the culture on campus doesn’t appeal to you.

    Sometimes it might be better to consider a lower university ranking if it can offer you plenty of chances to thrive and achieve your personal goals.

    Remember, the reason that there are so many different kinds of universities is because everyone is unique. If you’re applying to somewhere you know matches your values and interests, you’ll be much more likely to succeed in your university career.

    Research the teaching style

    It’s a good idea to look at the style of teaching and assessment at your chosen university. Some courses focus more on final exams, others on coursework, projects, dissertations and independent research.

    See whether your university’s exams are open book or closed book. If they’re closed you have to remember a lot more, but if they’re open you will probably have to give more detailed answers in your exams.

    If you can attend open days and chat with other students and potential professors, do so. But if you can’t, remember that each university course website will tell you about the course content and how it is assessed. You need to make a choice based on what type of assessment will suit you best.

    Some courses are taught in large lectures and seminars; others, such as many colleges, will teach you in small groups or one-to-one with a tutor. The teaching style that works best for you will be crucial in your choice.

    Consider your hobbies

    Every university has its own culture and social life based around the student unions and societies. For local and international students, university life is about more than studying.

    You will have a more balanced student experience if you make time for your hobbies. You can look on the university website to learn about student societies and clubs and see which activities are available. But don’t worry if you don’t see anything that appeals - you can always be proactive and start your own society.

    Do you have a sport that you love to do? If you do and are really good at it, you may be eligible for a . There are all sorts of funding and awards available, so even if you think you won’t get anything, it’s a good idea to check.

    Make sure you meet the English language requirements

    You’ll need to pass a test of English for your UK student visa requirements. You can take any Secure English Language Test (SELT) approved by the Home Office, the UK government department responsible for immigration.

    PTE Academic can be used for UK student visas for degree-level courses, and to access 99% of universities in the country. The test takes two hours, covers reading, speaking, listening and writing, and is done on a computer. Results usually come back within 48 hours, making it convenient and reliable.

    Choosing a university requires a lot of thought. Luckily, choosing an English test to get a visa is easy.

  • a woman sat in a library in front of a laptop and open books

    The importance of antonyms: Expanding your vocabulary

    By

    Expanding and improving your English vocabulary can be done in various ways, but exploring antonyms can be a handy and unique approach. Antonyms, which are words that express opposite meanings, can help you enhance your language skills. Knowing a wide range of them can be incredibly helpful in everyday life and help you get your point across clearly.

    In today's language learning blog post, we share practical tips to help you harness the power of antonyms and improve your vocabulary.

  • A child sat at a desk with a pen in hand, looking up at their teacher and smiling

    Dyslexia and ELT: How to help young learners in the classroom

    By Joanna Wiseman

    When you’re teaching English to young learners, you might find that there are a few students in your class who are struggling. But sometimes it can be hard to tell why. Is it because their language level is low? Or are they finding classroom work difficult because of a general cognitive difference, like dyslexia?