Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • teenage boy studying with headphones on and  with a laptop

    Why should I learn English?

    By Steffanie Zazulak

    English is the second most widely spoken language – it is estimated that nearly two billion people worldwide can speak English at a useful level. That means they can hold a conversation with other English-speaking people.

    A report by the attests the importance of the English language to the world, and says thatsecond-language English speakers far outnumber fluent English speakers. It also recognizes how being able to speak English can give individuals a competitive edge over others. If you're asking the question, "Why should I learn English?", read on to find out more...

  • A boy wearing headphones smiling listening to music

    11 great English language song lyrics

    By Steffanie Zazulak

    What is it about music that helps boost your English skills, confidence and pronunciation? A song can provide an emotional connection between the music and the listener, providing learners with new ways to express their feelings. Music and rhythm have also been shown to benefit memorization, which is a key component of learning.

    Here are some of our favorite lyrics to some of our favorite songs:

    1. The Beatles – Blackbird

    The Beatles are the best band to help you learn English. There are many Beatles songs with catchy melodies and simple lyrics, but Blackbird captures the Fab Four at their most poetic:

    Blackbird singing in the dead of night
    Take these broken wings and learn to fly
    All your life
    You were only waiting for this moment to arise

    2. The Cure – Friday I’m In Love

    This song is a great way to help learn the days of the week (that may be obvious). Love is also a very popular English word, so this one is for all the romantics out there.

    Always take a big bite
    It’s such a gorgeous sight
    To see you eat in the middle of the night

    3. Ed Sheeran – Thinking Out Loud

    Another one for the lovers, Ed’s heartfelt lyrics are huge in the mainstream pop charts. Here, he tells the sweet story of long-time love in this ballad and he’s becoming one of the world’s most sought-after songwriters.

    Take me into your loving arms
    Kiss me under the light of a thousand stars
    Place your head on my beating heart

    4. The Smiths – How Soon Is Now?

    This classic from Morrissey and co was voted runner-up in VH1’s Top Lyrics poll, for the lyrics: So you go and you stand on your own, and you leave on your own, and you go home and you cry and you want to die’ but it’s the opening lines of the song that are the most intriguing. Firstly, they’re adapted from George Eliot’s novel Middlemarch and include some rather clever double meanings – namely ‘son’ (sun) and ‘heir’ (air).

    I am the son
    And the heir
    Of a shyness that is criminally vulgar

    5. Neil Young – Heart of Gold

    This was Neil Young’s only number-one hit single from 1972’s Harvest album. He uses simple lyrics and melodies to tell his story of searching for true love.

    I’ve been in my mind
    It’s such a fine line
    That keeps me searchin’ for a heart of gold
    And I’m gettin’ old

    6. U2 – City of Blinding Lights

    U2 is the second-best band to help you learn English and frontman Bono is the second-best male artist to help you learn (edged out by Justin Timberlake). The chorus was inspired by a moment during a performance in New York City, when Bono saw the audience lit up and shouted, “Oh, you look so beautiful tonight!”

    And I miss you when you’re not around
    I’m getting ready to leave the ground
    Oh you look so beautiful tonight
    In the city of blinding lights

    7. The Police – Every Breath You Take

    These lyrics are a good use of repetition and rhyme, which is excellent for helping memorization. Most people consider this a love song, but that’s a common mistake. In 1983, Sting was interviewed for New Musical Express and explained: “I think it’s a nasty little song, really rather evil. It’s about jealousy and surveillance and ownership.”

    Every move you make and every vow you break
    Every smile you fake, every claim you stake, I’ll be watching you
    Every move you make, every step you take, I’ll be watching you

    8. Bob Dylan – Mr Tambourine Man

    Dylan’s whimsical, poetic lyrics might be difficult for an English language learner to interpret, but that doesn’t mean they’re not a pleasure to listen and sing along to.

    Hey! Mr Tambourine Man, play a song for me
    I’m not sleepy and there is no place I’m going to
    Hey! Mr Tambourine Man, play a song for me
    In the jingle jangle morning I’ll come followin’ you

    9. Madness – Our House

    This song takes you through the daily goings-on in a typical household in the 80s. This song has been around for a while now but is a story that most people can still connect with. This is an excellent song for understanding the concept of nostalgia!

    I remember way back then when
    everything was true and when
    we would have such a very good time
    such a fine time

    10. Otis Reading – (Sittin’ on) The Dock of the Bay

    Sadly, Otis Redding lost his life in a plane crash shortly after this song was recorded, and it was released after his death. His lyrics are quite reflective, provoking both contentment and sadness. You can really imagine yourself sitting on the dock with his simple yet descriptive words.

    Sittin’ in the morning sun
    I’ll be sittin’ when the evening comes
    Watching the ships roll in
    Then I watch them roll away again

    11. Leonard Cohen – Everybody Knows

    No list of standout turns of phrase would be complete without Leonard Cohen, a man whose songwriting process is so painstaking he’ll often spend years on the same song. When quizzed on his process, Cohen once said it often took so long because, “After a while, if you stick with the song long enough it will yield.” It’s interesting to note then that, even for a master of the English language, the words don’t always come easily. Everybody Knows remains a firm favorite among fans, with the majority of lines starting with the words ‘Everybody knows…’ It’s a lengthy song, but for the sheer beauty of its words and phrasing, the opening lines are a highlight:

    Everybody knows that the dice are loaded
    Everybody rolls with their fingers crossed
    Everybody knows that the war is over
    Everybody knows that the good guys lost

  • A couple laughing together while having drinks

    10 English words and slang terms you should know

    By Steffanie Zazulak

    Not everything can be taught in the classroom – that’s why we’ve put together a list of 10 English words, slang terms and figures of speech for you to sprinkle into your English conversations. Not sure how to use them? Don’t worry, we’ve included definitions and examples of how to use them in a sentence.

    Earworm

    A song or tune that, once you’ve heard it, is stuck in your head.

    Use it: “That new Taylor Swift song is such an earworm!”

    Staycation

    Spending your holiday in your hometown rather than traveling abroad. Often spent relaxing around the house or doing activities in the local area.

    Use it: “I’m having a staycation this summer, as I’m trying to save money.”

    Webisode

    An episode or series created exclusively for online viewing. This can be part of a web series or used to promote a television series.

    Use it: “There’s a new Breaking Bad webisode online, have you seen it?”

    Crowdfunding

    Funding a project or business venture by asking a large number of people (typically online) to invest a small amount of money – usually via websites such as Kickstarter and Indiegogo.

    Use it: “We raised enough money to launch our business using crowdfunding.”

    Glamping

    Glamorous camping! Avoiding any rough conditions with luxurious facilities and accommodation, such as a yurt or cabin.

    Use it: “We’re going to be glamping at Glastonbury this year.”

    Lookalike

    Something or someone that closely resembles something else, often used to describe someone who looks very similar to a celebrity.

    Use it: “Have you seen the Prince William and Kate Middleton lookalikes over there?”

    Handover

    This word has a more political significance, in terms of gaining land or ownership. However, its modern usage is commonly in the workplace: The act of passing over control or responsibility to another person while you are off work on holiday or leaving a job.

    Use it: “I’ll send my notes in a handover email so you can continue the project while I’m away.”

    Meh

    When something is a little uninspiring or dull. Also a word you could use to describe your lack of interest or indifference. It’s basically a verbal shrug.

    Use it: “The film was a bit meh.”

    Binge-watch

    Watching multiple episodes of a TV show, one after another, in a single sitting. Usually, with a DVD box set or using online streaming.

    Use it: “I’m planning to binge-watch the entire series of Game of Thrones this weekend!”

    Spill the tea

    When someone 'spills the tea' they are telling you all the latest news or gossip.

    Use it: “Please spill the tea about last week's party!"