Can computers really mark exams? Benefits of ELT automated assessments

app Languages
Hands typing at a laptop with symbols

Automated assessment, including the use of Artificial Intelligence (AI), is one of the latest education tech solutions. It speeds up exam marking times, removes human biases, and is as accurate and at least as reliable as human examiners. As innovations go, this one is a real game-changer for teachers and students.?

However, it has understandably been met with many questions and sometimes skepticism in the ELT community C can computers?really?mark speaking and writing exams accurately??

The answer is a resounding yes. Students from all parts of the world already take AI-graded tests.??and?Versant?tests C for example C provide unbiased, fair and fast automated scoring for speaking and writing exams C irrespective of where the test takers live, or what their accent or gender is.?

This article will explain the main processes involved in AI automated scoring and make the point that AI technologies are built on the foundations of consistent expert human judgments. So, lets clear up the confusion around automated scoring and AI and look into how it can help teachers and students alike.?

AI versus traditional automated scoring

First of all, lets distinguish between traditional automated scoring and AI. When we talk about automated scoring, generally, we mean scoring items that are either multiple-choice or cloze items. You may have to reorder sentences, choose from a drop-down list, insert a missing word- that sort of thing. These question types are designed to test particular skills and automated scoring ensures that they can be marked quickly and accurately every time.

While automatically scored items like these can be used to assess receptive skills such as listening and reading comprehension, they cannot mark the productive skills of writing and speaking. Every student's response in writing and speaking items will be different, so how can computers mark them?

This is where AI comes in.?

We hear a lot about how AI is increasingly being used in areas where there is a need to deal with large amounts of unstructured data, effectively and 100% accurately C like in medical diagnostics, for example. In language testing, AI uses specialized computer software to grade written and oral tests.?

How AI is used to score speaking exams

The first step is to build an acoustic model for each language that can recognize speech and convert it into waveforms and text. While this technology used to be very unusual, most of our smartphones can do this now.?

These acoustic models are then trained to score every single prompt or item on a test. We do this by using human expert raters to score the items first, using double marking. They score hundreds of oral responses for each item, and these Standards are then used to train the engine.?

Next, we validate the trained engine by feeding in many more human-marked items, and check that the machine scores are very highly correlated to the human scores. If this doesnt happen for any item, we remove it, as it must match the standard set by human markers. We expect a correlation of between .95-.99. That means that tests will be marked between 95-99% exactly the same as human-marked samples.?

This is incredibly high compared to the reliability of human-marked speaking tests. In essence, we use a group of highly expert human raters to train the AI engine, and then their standard is replicated time after time.??

How AI is used to score writing exams

Our AI writing scoring uses a technology called . LSA is a natural language processing technique that can analyze and score writing, based on the meaning behind words C and not just their superficial characteristics.?

Similarly to our speech recognition acoustic models, we first establish a language-specific text recognition model. We feed a large amount of text into the system, and LSA uses artificial intelligence to learn the patterns of how words relate to each other and are used in, for example, the English language.?

Once the language model has been established, we train the engine to score every written item on a test. As in speaking items, we do this by using human expert raters to score the items first, using double marking. They score many hundreds of written responses for each item, and these Standards are then used to train the engine. We then validate the trained engine by feeding in many more human-marked items, and check that the machine scores are very highly correlated to the human scores.?

The benchmark is always the expert human scores. If our AI system doesnt closely match the scores given by human markers, we remove the item, as it is essential to match the standard set by human markers.

AIs ability to mark multiple traits?

One of the challenges human markers face in scoring speaking and written items is assessing many traits on a single item. For example, when assessing and scoring speaking, they may need to give separate scores for content, fluency and pronunciation.?

In written responses, markers may need to score a piece of writing for vocabulary, style and grammar. Effectively, they may need to mark every single item at least three times, maybe more. However, once we have trained the AI systems on every trait score in speaking and writing, they can then mark items on any number of traits instantaneously C and without error.?

AIs lack of bias

A fundamental premise for any test is that no advantage or disadvantage should be given to any candidate. In other words, there should be no positive or negative bias. This can be very difficult to achieve in human-marked speaking and written assessments. In fact, candidates often feel they may have received a different score if someone else had heard them or read their work.

Our AI systems eradicate the issue of bias. This is done by ensuring our speaking and writing AI systems are trained on an extensive range of human accents and writing types.?

We dont want perfect native-speaking accents or writing styles to train our engines. We use representative non-native samples from across the world. When we initially set up our AI systems for speaking and writing scoring, we trialed our items and trained our engines using millions of student responses. We continue to do this now as new items are developed.

The benefits of AI automated assessment

There is nothing wrong with hand-marking homework tests and exams. In fact, it is essential for teachers to get to know their students and provide personal feedback and advice. However, manually correcting hundreds of tests, daily or weekly, can be repetitive, time-consuming, not always reliable and takes time away from working alongside students in the classroom. The use of AI in formative and summative assessments can increase assessed practice time for students and reduce the marking load for teachers.

Language learning takes time, lots of time to progress to high levels of proficiency. The blended use of AI can:

  • address the increasing importance of?formative assessment?to drive personalized learning and diagnostic assessment feedback?

  • allow students to practice and get instant feedback inside and outside of allocated teaching time

  • address the issue of teacher workload

  • create a virtuous combination between humans and machines, taking advantage of what humans do best and what machines do best.?

  • provide fair, fast and unbiased summative assessment scores in high-stakes testing.

We hope this article has answered a few burning questions about how AI is used to assess speaking and writing in our language tests. An interesting quote from Fei-Fei Li, Chief scientist at Google and Stanford Professor describes AI like this:

I often tell my students not to be misled by the name artificial intelligence there is nothing artificial about it; A.I. is made by humans, intended to behave [like] humans and, ultimately, to impact human lives and human society.

AI in formative and summative assessments will never replace the role of teachers. AI will support teachers, provide endless opportunities for students to improve, and provide a solution to slow, unreliable and often unfair high-stakes assessments.

Examples of AI assessments in ELT

At app, we have developed a range of assessments using AI technology.

Versant

The Versant tests are a great tool to help establish language proficiency benchmarks in any school, organization or business. They are specifically designed for placement tests to determine the appropriate level for the learner.

PTE Academic

The ?is aimed at those who need to prove their level of English for a university place, a job or a visa. It uses AI to score tests and results are available within five days.?

app English International Certificate (PEIC)

app English International Certificate (PEIC) also uses automated assessment technology. With a two-hour test available on-demand to take at home or at school (or at a secure test center). Using a combination of advanced speech recognition and exam grading technology and the expertise of professional ELT exam markers worldwide, our patented software can measure English language ability.

Read more about the use of AI in our learning and testing here, or if you're wondering which English test is right for your students make sure to check out our post 'Which exam is right for my students?'.

Explore our courseware and exams

More blogs from app

  • Children sat at desks in a classroom, with one in the middle smiling with her hand up

    Teaching young learners: all just fun and games?

    By Jill Leighton
    Reading time: 5 minutes

    We want lots of fun in our lessons, with a good balance of activities and games with strong educational value. But in my opinion, todays young learners need more than just fun and games to stay motivated. Fun and games can quickly become meaningless and boring for some students.

    How do we create fun and engaging lessons for young learners aged 6-14?

    • Have a strong relationship with your students. Know about their natural talents, their weaker points and where they are in their development.
    • Encourage them to be active learners and engage them in learning that is meaningful to all of them. E.g. through project work.
    • Provide opportunities for students to use technology and develop success skills. E.g. collaboration, communication, critical thinking, creativity and self-learning.

    Most teachers agree that we have to assume the responsibility of preparing students for life as well as helping them to acquire a language C and this is what I aim to demonstrate in this blog. Using , I want to demonstrate how you can learn more about your students and care for individual needs in one classroom.

    The students will also learn about themselves and their classmates. They will acquire vocabulary and language (at their level) in a dynamic way and create a project to explore their interests and demonstrate their talents. They will use technology and develop their success skills. Assessment will be fun and engaging. I hope you will be able to adapt the ideas and put them into practice with children aged 6-14.

    Howard Gardners Model of Multiple Intelligences

    prompts us to ask: How is this child intelligent? He identified eight different types of intelligence which guide the way students learn:

    • Verbal-Linguistic Intelligence (Word Smart)
    • Logical-Mathematical Intelligence (Maths Smart)
    • Spatial Intelligence (Picture Smart)
    • Musical Intelligence (Music Smart)
    • Bodily-Kinesthetic Intelligence (Body Smart)
    • Naturalistic-Environmentalist Intelligence (Nature Smart)
    • Interpersonal Intelligence (People Smart)
    • Intrapersonal Intelligence (Self-Smart)
    • (He later went on to identify Existential Intelligence C Life Smart, and Pedagogical Intelligence -Teacher Smart)

    Lesson 1: Outline the students learning goals

    Explain to the students that they are going to learn about Howard Gardners Model of Multiple Intelligences and that you want them to achieve learning goals on completion of the project. Write the goals on a poster and discuss them with your students. Take a digital photo of the poster to use during the assessment.

    Student learning goals C you will be able to:

    • Identify different intelligences and what they mean. Name eight different intelligences you have
    • Communicate ideas clearly through a project to demonstrate your talents and interests
    • Use your success skills more effectively.

    Tip: Video parts of lesson 1, especially discussion of the learning goals, to use as part-assessment and reflection on completion of the project.

    What do you know about the Multiple Intelligences? How are you smart?

    • Ask children what they know about multiple intelligences. Teach key language and vocabulary as necessary, depending on level.
    • Ask, What does it mean be intelligent? (They will probably say, good grades, be good at maths, English, reading, writing, computers...)
    • Tell them being intelligent or smart (get the younger children to repeat the word smart several times) isnt only about getting good grades. Ask them to think of more ways to be smart.
    • Elicit the eight ways to be smart according to Gardner. (You may need to mime). E.g. mime playing the piano or clap a rhythm to elicit Music Smart.
    • Discuss what the different intelligences mean. (E.g. Word Smart (Linguistic Intelligence): you like reading, writing or speaking, you are probably good at languages)
    • Teach career vocabulary associated with the different intelligences. (E.g. Word Smart: journalist, teacher, lawyer, editor, TV announcer, web editor.)

    Lesson 2: Find out your smarts quiz

    Tell students they are going to do a quiz to discover how they are smart. Model each stage of the activity and do the quiz with them to find out about your own strengths. Give each student a piece of paper.

    • Take your paper and show the students how to fold it into eight sections, then unfold it and draw lines along the folds to make a grid.
    • Write the different smarts in each section. (Use small handwriting to leave room to illustrate each smart with a picture).
    • Give a picture dictation to illustrate each smart. Here are some examples:

    Word Smart:?Draw a dictionary and children reading, writing and speaking.

    Logic/Number Smart: Draw sums on a computer, a scientist with test tube.

    Music Smart: Draw children singing and playing musical instruments.

    Body Smart: Draw children playing a sport, dancing or cooking.

    Nature Smart:?Draw trees, animals, insects, child watering a plant.

    Spatial/Picture Smart: Draw children drawing, painting or taking photos and a pilot in a plane.

    People Smart:?Draw a child helping or leading a group or a group of children holding hands.

    Self-Smart: Draw children keeping a journal, researching on a computer, or meditating.

    Encourage students to order their smarts from 1-8. For example, if you love music, write number 1 in the Music Smart section and continue to 8 in order of preference. (You may wish to model this first and order your smarts from 1-8 and then encourage the children to order their smarts.)

    After the quiz

    • Encourage students to compare and discuss their results. Collect the papers and make notes about each students results. This will help you reach all your students when planning activities.
    • Explain that we have all the intelligences in different degrees and that all of the intelligences are equal (no intelligence is better than another). Also point out that it is important to know our strengths in order to help in all subjects. (E.g. a music smart student who finds maths challenging may want to sing multiplication tables). Remind students that we usually use several intelligences to do something and we can explore and develop all our smarts.

    Tip: Video lesson 2 to use as part of assessment.

    Lesson 3: Beginning the project

    Encourage the students to create a project for enjoyment. Explain that you want them to collaborate in groups that share the same smarts and interests and using their creativity you want them come up with an interesting topic to explore. (Remind students to concentrate on developing their success skills when they are working with others and mention that you will also be monitoring this as part of the assessment).

    Organize the students into groups of no more than five students in each. Give students time to brainstorm in their groups and come up with the best topic for the project, using their critical thinking skills. (E.g. Picture Smart students may decide to create a project about a famous artist, such as Salvador Dali).

    You may wish to give each student a ?(Know, Want-to-know, and Learned) to complete during the project process, asking questions such as: What do you know? What do you want to know? What have you learned?

    Lesson 4: Planning the project

    Inform students about the timing of the project. Encourage each group to make a project mind map, which will encourage them to be more creative and organized.

    Remind all the groups to keep the following questions in mind:

    • What are you going to do/make? (Presentation, PowerPoint, website, video, posters.)
    • How will you research it? (Internet, tech tools, library.)
    • How will you delegate responsibilities?
    • How will you check that your audience has understood the message of your project? What questions will you ask?

    Decide on how many lessons are needed to prepare the project and how much will be done in school or at home, depending on the age groups and timing.

    Encourage the students to share their work with the rest of the class (or in assembly.)

    Tip: Video the different groups sharing their work.

    Can assessment also be fun and engaging?

    Yes, it can; here are some tips and suggestions.

    • Show the videos you have recorded and ask the students to compare and contrast their knowledge in Lesson 1 and how it developed over the lessons. Encourage them to observe and comment on their success skills.
    • Display the photo of the Student Learning Goals poster from Lesson 1. Get the students to self-assess and decide whether they have achieved the goals that were set in Lesson 1. (You may wish to give the students three small pieces of colored paper: red to represent I understand quite well, orange to represent I understand well and green to represent I understand very well). Ask: Can you identify the different intelligences and say what they mean? Encourage the students to hold up a colored piece of paper according to their understanding. (Make a mental note of all red pieces of paper to be ready to give extra help to those students). Check understanding by getting several students to answer the question.
    • Get the children to reflect on the learning experience. What have they learned about the different intelligences? How can they develop weaker points using their strengths to help them? Can they use all eight intelligences inside and outside school? Did they manage to get along well with their classmates? Did they communicate the message of their project so that the audience understood?
    • Give individual feedback to each student. E.g. congratulate them on their attitude and effort or identify areas for improvement: You managed to use vocabulary and language effectively when you shared your project, we understood your message perfectly. Or You need to work on being more collaborative. You werent on task during the project. How do you think you can improve that?
    • Ask students to give you feedback on the activities they enjoyed. Get them to draw happy and sad face cards. Go through all the activities and get the students to show a happy or sad face according to whether or not they liked the activity. E.g. say Did you like the Find out your smarts quiz? and ask them to hold up the happy or sad face depending on whether they liked the activity or not.

    Create fun lessons to engage all your students keeping this model in mind: traditional activities such as short fun activities and games + Howard Gardners model of Multiple Intelligences + PBL (Project Based Learning) + success skills + meaningful assessment. Enjoy the results with your students.

    How the Global Scale of English can help

    The?Global Scale of English (GSE) Learning Objectives for provides ready-made learning objectives that can help with planning curriculums and lesson and benchmarking learners progress. They are great for young learners because they describe language functions in a granular way, enabling educators to give their learners credit for small achievements. They also clearly show the language functions to target next in order to take learners to the next level.

  • Expressing love around the world: Interesting facts and how to say "I love you" in different languages

    By Sam Colley
    Reading time: 5 minutes

    Love is a universal language that transcends borders, cultures and languages. For those learning another language, understanding how to express love in various tongues can be both fascinating and useful. Whether you're planning to travel, connect with friends from different backgrounds, or simply expand your linguistic repertoire, knowing how to say "I love you" in different languages as well as the cultural context, can be a beautiful way to show appreciation and affection. Let's take a look at some of the many ways to express this timeless sentiment and some interesting facts.

    1. Cultural nuances:

    In many cultures, the way you express love can carry different weights and meanings. For example, in Japanese, "ۤƤ" (aishiteru) is a very strong expression of love, often reserved for serious relationships, whereas "ä" (daisuki) is more commonly used among friends and family.
    However, French, often called the language of love, uses "je t'aime" to express love as a romantic phrase that can be used for both partners and close family members.

    2. Gender differences:

    Some languages have gender-specific ways of saying "I love you." For instance, in Arabic, "????" (a?ibbuka) is used when a woman says "I love you" to a man, and "????" (a?ibbuki) is used when a man says it to a woman. Similarly, in Hindi, men say "??? ????? ????? ???? ???" (main tumse pyaar karta hoon) and women say "??? ????? ????? ???? ???" (main tumse pyaar karti hoon).

    3. Formal vs. informal:

    Some languages have formal and informal ways of expressing love. In Spanish, "te amo" is more formal and used for romantic love, while "te quiero" is more casual and can be used for friends and family.
    ?

  • A teacher sat at the front of the class holding a book reading it to a class of young children who are sat on the floor

    Romantic Readers and classroom ideas to celebrate Valentines day

    By Iram Ahmed
    Reading time: 3 minutes

    Love is in the air on February 14th, with hearts, red roses and boxes of chocolates everywhere. Were all set for the big day, and to help you prepare as well, here are six of our favorite love stories for learners of all ages and levels. There are also plenty of fun ideas for your classroom.

    1.?Cinderella?(Level 1)

    Poor Cinderella lives with her stepmother and two horrible stepsisters. They treat her badly, making her cook and clean, while they relax and wear beautiful clothes. However, one night, Cinderella gets the chance to sneak away and attend a fabulous ball. She meets a handsome princebut must run away before he discovers her true identity. With only a glass shoe to help him, will the prince ever find Cinderella?

    How to use this Reader in class:

    Its no surprise that the story has a happy ending. After reading the story, have the learners draw a picture and write about Cinderellas new life in the castle, with her loving new family.

    2.?Lady and the Tramp?(Level 3)

    This classic Disney story tells the tale of two young dogs who come from different backgrounds. Lady lives in a house while Tramp lives on the street. However, one day, Lady is forced to leave the house and the life she knows. Tramp teaches her about life on the streets and the two dogs fall in love but with the dog-catcher always chasing them, will the story have a happy ending?

    How to use this reader in class:

    After reading the story, why not watch the animated version with your students? They can choose their favorite character from the movie and write a description of them.

    Looking for more romantic Readers for young learners? How about?The Little Mermaid?(Level 2) or?Aladdin?(Level 5)? Theres something for everyone?in our catalog.

    3.?Pirates of the Caribbean: The Curse of the Black Pearl?(Level 2)

    If your teenagers are looking for more adventure than romance, this Reader has both. Follow crazy Jack Sparrow, handsome Will and brave Elizabeth as they fight pirates and uncover the curse of the Black Pearl. But will Elizabeth and Will survive the dangerous life on the seven seas?

    How to use this Reader in class:

    After reading the story, why not ask your students to write a review of it? Its a great way for them to summarize the plot and share their opinion.

    4.?Pride and Prejudice?(Level 5)

    A timeless classic,?Pride and Prejudice?tells the story of young Elizabeth Bennett. As the oldest daughters in a family of five, she and her sister Jane are both expected to find good husbands and marry soon. But will Elizabeth find love, or will no man be good enough for her?

    How to use this Reader in class:

    While reading the story, learners can keep a diary, writing entries in the voice of Elizabeth Bennett. This helps them to demonstrate their understanding of the novel and encourages them to have a more empathetic reaction to the storys heroine. They can either write the diary entries using typical language from Jane Austens day, or updating it to more modern language.

    Heres an example:

    I had the pleasure of meeting Mr Darcy today and found him rather rude and reserved. I imagine he feels this country life is inferior and that the locals are uncultured simpletons!

    or

    Met Mr Darcy today C what a stuck-up dork! He just stood there, looking down his nose at everyone you can see he thinks hes better than us locals.

    If you think your teenage learners would enjoy a different modern or classic tale of romance, check out our readers?. There are plenty to choose from, including?Romeo and Juliet?(Level 3),?Love Actually?(Level 4) and?Madame Bovary?(Level 6).

    5.?Tales from the Arabian Nights?(Level 2)?

    What would you do if you needed to fascinate a king to save your life? This is the problem Sheherezade faces when she marries a king who has killed all his previous wives. So she tells her new husband exciting, mesmerizing stories every night and always ends the tale with the king wanting to hear more. She tells him stories of adventure, love and mystery and you can read her stories too in this Level 2 Reader.

    How to use this reader in class:

    After reading the book, have students vote on their favorite story from Sheherezade. They can then work in groups to present a short play of one of the stories, working together to write a short script.

    6.?Wuthering Heights?(Level 5)?

    Unfortunately, the path of true love isnt always easy. Set in 19th Century Yorkshire, this tragic romance follows the story of Catherine and Heathcliff. The two childhood friends grow up together very closely. However, although she loves him deeply, Catherine cant marry Heathcliff because of his lower social status. Instead, she marries another man, Edgar. How will Heathcliff react to this news? Will Catherine and Edgar be happy together?

    How to use this reader in class:

    At one point in?Wuthering Heights, Heathcliff runs away from home because he thinks Catherine is embarrassed by him and doesnt love him. Have the learners write a letter from Catherine to Heathcliff at that point, expressing her regrets about why they cant be together. It provides students with the opportunity to use phrases such as?If only,?I wish?and?I should have?whilst demonstrating their understanding of the novel so far.

    If Yorkshire moors and Arabian nights arent quite what your students are looking for, there are more tales of romance on our site. Choose from modern-day love stories such as?Notting Hill?(Level 3) and?Lisa in London?(Level 1) or classics like?Anna Karenina?(Level 6) and?Persuasion?(Level 2).

    If you enjoyed this post here are some more practical posts to get your students engaged in reading, 'How to bring Shakespeare to life in the classroom'?and 'How to motivate reluctant readers'.