Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • Children walking in a neighbourhood wearing costumes

    10 creepy cryptids you should know about

    Por

    Cryptids are creatures that are often unseen and mysterious. They are shrouded in legends and stories that have been passed down for generations, making them a fascination for humans for centuries. If you're looking to add a little more creativity to your story writing, learning about these elusive beings can be a great way to do so. In today's post, we'll take a closer look at some examples of cryptids, to get your imagination racing.

    What are cryptids?

    Cryptids are mythical creatures or beings whose existence cannot be proven by science. Some may claim to have seen them but there's usually no solid proof of the encounter. They exist in folklore, mythology and urban legends. Cryptids can be found in cultures all around the world, from the Loch Ness Monster in Scotland to the in Latin America.

    Here are ten cryptids you'll want to learn about this Halloween:

    Barghest

    The Barghest is a ghostly black dog cryptid that appears in the folklore of Yorkshire and Lancashire. It is often associated with misfortune, and sightings of this ominous creature continue to be reported.

    Owlman

    The Owlman is a humanoid creature with owl-like features such as red eyes, wings and feathers. Sightings of this mysterious creature have been reported around the village of Mawnan Smith in Cornwall, adding an eerie twist to local legend.

    The Kraken

    The Kraken is a legendary sea monster of gigantic size and octopus-like appearance, said to dwell in the deep sea and feasting on ships that are unfortunate enough to come across it.

  • A girl in a costume running up the stairs to the front door of a house that is covered in Halloween decorations

    Eerie English idioms and phrases

    Por

    As the leaves turn golden and the air becomes crisp, it's not only the ghosts and ghouls that come out to play. Halloween may happen only once a year, but learning about spooky idioms and phrases can add an exciting twist to your language journey throughout the year. So, grab your torch and let's delve in.

  • Children in halloween costumes stood in a hallway with a adult

    5 spooky ideas for your primary classes this Halloween

    Por Joanna Wiseman

    It’s almost Halloween, and the ghosts and vampires will soon be coming out to play. Did you know that although we often associate Halloween with pumpkin carving and eating candy, the festival has much older origins?

    is an ancient Gaelic festival that celebrates the end of the harvest and the start of winter. This is why people often associate the colors of orange and black with Halloween: orange is the color many leaves turn in autumn and black is the color of the darker winter months.

    People used to believe that spirits walked the Earth on the night of Samhain. The tradition of dressing up as ghosts and demons started as a way to hide from the spirits who walked the streets. Similarly, people used to leave treats outside their houses for the spirits and from this came the tradition of trick-or-treating.

    So to help get your younger students in the Halloween spirit, here are five spooky ideas to try in your primary classes.

    1. ‘Pumpkin’ oranges

    Pumpkin carving is fun - but it’s also messy and pumpkins can be really heavy. Instead, bring in an orange for each student and give them a black marker pen. Get them to draw a scary face on their orange and then write a short text describing it.

    My pumpkin orange, Ghoulie, has two big eyes. He’s got a small nose and a big mouth, with lots of teeth. This Halloween, he’s going to sit outside my house. He’s going to scare people but he doesn’t scare me. I think he’s very funny.

    2. Bat fishing

    This is a great way to practice questions and review language with your younger students. Have your students cut out bat shapes on card and tell them to write a question on the back of each one. They can write personal information questions, such as ‘What do you eat for breakfast?’ or questions related to topics you’re studying at the moment, like ‘How do you spell dinosaur?’

    Attach a paper clip to each bat and put them on the floor, with the questions face down. Then attach a magnet to a piece of string.

    Divide the class into teams and have students take turns to fish a bat from the floor. When they catch a bat using the magnet, a student from another team asks them the question written on the bat. If the team can answer correctly, they keep the bat. If they don’t answer correctly, the bat goes back on the floor.

    When all the bats have been fished, the team with the most wins.

    3. Haunted house dictation

    This is a good activity to review prepositions of place and house vocabulary. Before you start, elicit some scary things from the students, such as ghost, spider, witch, zombie. If these words are new for your students, draw a picture dictionary on the board for them to refer to in the next stage.

    Next, give students an outline of a house with the rooms labeled, but without any furniture. Then dictate a sentence to the students and have them draw what you say on their individual houses. For example, ‘In the kitchen, there’s a big cupboard. In the cupboard, there’s a witch.’ Or, ‘In the living room, there’s an old sofa. A zombie is sitting on the sofa.’

    You can then divide the class into pairs or small groups and have them take turns dictating sentences to each other. When they finish, they can compare their pictures and then write a short story about their haunted houses.

    4. Trick-or-treat board game

    Draw a 7x5 grid on card and add Start and Finish squares. Number the other squares so the students know what direction to move in. Then, on some of the squares write Trick and on some of the other squares write Treat. Finally, prepare a set of ‘trick’ and ‘treat’ cards for each group. (There are some ideas for tricks and treats below).

    Before students play, teach them some phrases to use while playing the game. For example:

    • Whose turn is it?
    • It’s my turn.
    • Roll the dice.
    • Who’s winning?

    Then divide the class into groups of four and give each group a board, a set of ‘trick-or-treat’ cards, a dice and a counter. Have them take turns to roll the dice and move. If they land on a Trickor Treat square, they have to take a card and do what it says. Then they put the card at the bottom of the pile.The winner is the first person to reach the Finish square.

    Ideas for ‘trick’ cards

    • Go back 3 squares
    • Miss a turn
    • Go back to the start
    • Count down from 10 to 1 in English
    • Say the alphabet backwards (Z, Y, X…)
    • Laugh like a witch
    • Pretend to be a ghost

    Ideas for ‘treat’ cards

    • Go forward two spaces
    • Roll again
    • Go forward five spaces
    • Choose someone to miss a turn

    5. Spooky stories

    Are your students bored of celebrating Halloween every year? Mix things up with stories or readers. Allowing their imagination to run wild. There are lots of you can use or get inspiration from, creating your own. If you want your pupils more involved you could also have them make or take part in your very own 'create your own adventure' spooky story.

    After reading the story, have your students create comic strips of different parts of the book and display them around the classroom. If your students prefer theatrics, get them to act out or sing parts of the story.