Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A person smiling outdoors with headphones in their ears

    Can music help you learn languages?

    Por

    Learning a new language can be daunting, requiring time, effort, and consistent practice to master. One learning tool that’s been researched and examined over the years for its effectiveness, is music and sound.
    Music is sometimes overlooked or forgotten about, but it can be a beneficial tool when learning a new language. Today we will explore how music can aid language learning and what benefits it can provide. Here’s how:

    Recollection and memory

    Song is a powerful tool that can help us memorize new words and phrases. When listening to a song, recollecting the lyrics and melody long after it's finished can often happen. This is because our brains retain information better when associated with a melody/rhythm.

    Research has found that exposure to a language's rhythm leaves a lasting impact on a person's ability to segment speech patterns[1]. Listening to songs in a foreign language can help learners memorize new words, patterns and phrases more easily.

    Pronunciation and tone

    Music can improve our pronunciation and tone. When singing along to music, listeners may mimic the singer's pronunciation and cadence when listening to a song. This can help us develop a more natural and authentic accent when speaking the language.

    Additionally, it can broaden our knowledge and comprehension of diverse dialects and accents. Exposure to different accents and pronunciations can make real-world interactions with a foreign language less daunting or confusing.

    Two people may speak the same language but pronounce words very differently due to accents, a great example of this is in the UK and how the English language varies throughout areas.

    In the UK, someone from the north may sound completely different to someone from the south, despite speaking the same language. So if you were trying to learn English with music and listened to English songs from different parts of the country, you'd be exposing yourself to different accents. Making yourself more familiar with the varied pronunciation.

    Cultural understanding

    It can expose us to the culture of the language we are learning. Music can be a deep-rooted part of a culture, and hearing music in a different language can provide insight into that culture's beliefs, practices, and heritage. This can give us a deeper understanding of the language and help us communicate more effectively with fluent speakers.

    Often, songs use informal language and slang words we may not otherwise know or make us aware of a culture's situation or troubles. Academics Mark Gant & Dr. Carmen Fonseca-Mora discuss how although no one type of music works the best, foreign protest songs are a good example.

    They're typically motivational, clear, and repetitive and give an insight into a country's culture without causing problems for the learners. Listeners may even feel inclined to discuss or research that situation – which could be a good learning opportunity.

    Listening skills

    In addition, music can improve our listening skills. When we listen to a song in a foreign language, we’re often compelled to pay close attention to the lyrics, melody, and rhythm. This can help us develop our listening skills and train our ears to distinguish different sounds and accents in the language.

    Being exposed to foreign languages in any output can be beneficial; having the added layers of music adds that little extra challenge for your ears and brain. It’s almost like immersing yourself in the language without the need for traveling.

    An enjoyable way to learn

    Finally, music can make language learning more enjoyable. Learning a new language can be challenging and sometimes even frustrating. However, listening to music in a foreign language can make learning more enjoyable and engaging.

    Singing along with the lyrics can be a fun way to practice the language and boost motivation and confidence. Pairing those with interesting and engaging music videos can also make it more enjoyable. Check out some examples in our post11 great English language song lyrics.

    So when you find yourself taking on a new linguistic challenge, give music some 'airtime' – and watch how it helps you expand your vocabulary and improve your language skills.

  • A boy wearing headphones smiling listening to music

    11 great English language song lyrics

    Por

    What is it about music that helps boost your English skills, confidence and pronunciation? A song can provide an emotional connection between the music and the listener, providing learners with new ways to express their feelings. Music and rhythm have also been shown to benefit memorization, which is a key component of learning.

    Here are some of our favorite lyrics to some of our favorite songs:

    1. The Beatles – Blackbird

    The Beatles are the best band to help you learn English. There are many Beatles songs with catchy melodies and simple lyrics, but Blackbird captures the Fab Four at their most poetic:

    Blackbird singing in the dead of night
    Take these broken wings and learn to fly
    All your life
    You were only waiting for this moment to arise

    2. The Cure – Friday I’m In Love

    This song is a great way to help learn the days of the week (that may be obvious). Love is also a very popular English word, so this one is for all the romantics out there.

    Always take a big bite
    It’s such a gorgeous sight
    To see you eat in the middle of the night

    3. Ed Sheeran – Thinking Out Loud

    Another one for the lovers, Ed’s heartfelt lyrics are huge in the mainstream pop charts. Here, he tells the sweet story of long-time love in this ballad and he’s becoming one of the world’s most sought-after songwriters.

    Take me into your loving arms
    Kiss me under the light of a thousand stars
    Place your head on my beating heart

    4. The Smiths – How Soon Is Now?

    This classic from Morrissey and co was voted runner-up in VH1’s Top Lyrics poll, for the lyrics: So you go and you stand on your own, and you leave on your own, and you go home and you cry and you want to die’ but it’s the opening lines of the song that are the most intriguing. Firstly, they’re adapted from George Eliot’s novel Middlemarch and include some rather clever double meanings – namely ‘son’ (sun) and ‘heir’ (air).

    I am the son
    And the heir
    Of a shyness that is criminally vulgar

    5. Neil Young – Heart of Gold

    This was Neil Young’s only number-one hit single from 1972’s Harvest album. He uses simple lyrics and melodies to tell his story of searching for true love.

    I’ve been in my mind
    It’s such a fine line
    That keeps me searchin’ for a heart of gold
    And I’m gettin’ old

    6. U2 – City of Blinding Lights

    U2 is the second-best band to help you learn English and frontman Bono is the second-best male artist to help you learn (edged out by Justin Timberlake). The chorus was inspired by a moment during a performance in New York City, when Bono saw the audience lit up and shouted, “Oh, you look so beautiful tonight!”

    And I miss you when you’re not around
    I’m getting ready to leave the ground
    Oh you look so beautiful tonight
    In the city of blinding lights

    7. The Police – Every Breath You Take

    These lyrics are a good use of repetition and rhyme, which is excellent for helping memorization. Most people consider this a love song, but that’s a common mistake. In 1983, Sting was interviewed for New Musical Express and explained: “I think it’s a nasty little song, really rather evil. It’s about jealousy and surveillance and ownership.”

    Every move you make and every vow you break
    Every smile you fake, every claim you stake, I’ll be watching you
    Every move you make, every step you take, I’ll be watching you

    8. Bob Dylan – Mr Tambourine Man

    Dylan’s whimsical, poetic lyrics might be difficult for an English language learner to interpret, but that doesn’t mean they’re not a pleasure to listen and sing along to.

    Hey! Mr Tambourine Man, play a song for me
    I’m not sleepy and there is no place I’m going to
    Hey! Mr Tambourine Man, play a song for me
    In the jingle jangle morning I’ll come followin’ you

    9. Madness – Our House

    This song takes you through the daily goings-on in a typical household in the 80s. This song has been around for a while now but is a story that most people can still connect with. This is an excellent song for understanding the concept of nostalgia!

    I remember way back then when
    everything was true and when
    we would have such a very good time
    such a fine time

    10. Otis Reading – (Sittin’ on) The Dock of the Bay

    Sadly, Otis Redding lost his life in a plane crash shortly after this song was recorded, and it was released after his death. His lyrics are quite reflective, provoking both contentment and sadness. You can really imagine yourself sitting on the dock with his simple yet descriptive words.

    Sittin’ in the morning sun
    I’ll be sittin’ when the evening comes
    Watching the ships roll in
    Then I watch them roll away again

    11. Leonard Cohen – Everybody Knows

    No list of standout turns of phrase would be complete without Leonard Cohen, a man whose songwriting process is so painstaking he’ll often spend years on the same song. When quizzed on his process, Cohen once said it often took so long because, “After a while, if you stick with the song long enough it will yield.” It’s interesting to note then that, even for a master of the English language, the words don’t always come easily. Everybody Knows remains a firm favorite among fans, with the majority of lines starting with the words ‘Everybody knows…’ It’s a lengthy song, but for the sheer beauty of its words and phrasing, the opening lines are a highlight:

    Everybody knows that the dice are loaded
    Everybody rolls with their fingers crossed
    Everybody knows that the war is over
    Everybody knows that the good guys lost

  • A group of Young adult students in a library, looking in front of them smiling, some with their hands raised

    10 things to know about studying in Canada

    Por app Languages

    Have you considered studying abroad? Moving to Canada to study? With more than 100 world-class universities across the country, there’s good reason to. From coast to coast to coast, it offers many international students a safe and supportive learning environment and the opportunity to work after graduation.

    Want to learn more? In this guide, we cover ten things you should know about studying in Canada: