Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A young woman sat in a library with headphones around her neck reading a book

    Does progress in English slow as you get more advanced?

    By Ian Wood
    Reading time: 4 minutes

    Why does progression seem to slow down as an English learner moves from beginner to more advanced skills?

    The journey of learning English

    When presenting at ELT conferences, I often ask the audience – typically teachers and school administrators – “When you left home today, to start your journey here, did you know where you were going?” The audience invariably responds with a laugh and says yes, of course. I then ask, “Did you know roughly when you would arrive at your destination?” Again the answer is, of course, yes. “But what about your students on their English learning journey? Can they say the same?” At this point, the laughter stops.

    All too often English learners find themselves without a clear picture of the journey they are embarking on and the steps they will need to take to achieve their goals. We all share a fundamental need for orientation, and in a world of mobile phone GPS we take it for granted. Questions such as: Where am I? Where am I going? When will I get there? are answered instantly at the touch of a screen. If you’re driving along a motorway, you get a mileage sign every three miles.

    When they stop appearing regularly we soon feel uneasy. How often do English language learners see mileage signs counting down to their learning goal? Do they even have a specific goal?

    Am I there yet?

    The key thing about GPS is that it’s very precise. You can see your start point, where you are heading and tell, to the mile or kilometer, how long your journey will be. You can also get an estimated time of arrival to the minute. As Mike Mayor mentioned in his post about what it means to be fluent, the same can’t be said for understanding and measuring English proficiency. For several decades, the ELL industry got by with the terms ‘beginner’, ‘elementary’, ‘pre-intermediate’ and ‘advanced’ – even though there was no definition of what they meant, where they started and where they ended.

    The CEFR has become widely accepted as a measure of English proficiency, bringing an element of shared understanding of what it means to be at a particular level in English. However, the wide bands that make up the CEFR can result in a situation where learners start a course of study as B1 and, when they end the course, they are still within the B1 band. That doesn’t necessarily mean that their English skills haven’t improved – they might have developed substantially – but it’s just that the measurement system isn’t granular enough to pick up these improvements in proficiency.

    So here’s the first weakness in our English language GPS and one that’s well on the way to being remedied with the Global Scale of English (GSE). Because the GSE measures proficiency on a 10-90 scale across each of the four skills, students using assessment tools reporting on the GSE are able to see incremental progress in their skills even within a CEFR level. So we have the map for an English language GPS to be able to track location and plot the journey to the end goal.

    ‘The intermediate plateau’

    When it comes to pinpointing how long it’s going to take to reach that goal, we need to factor in the fact that the amount of effort it takes to improve your English increases as you become more proficient. Although the bands in the CEFR are approximately the same width, the law of diminishing returns means that the better your English is to begin with, the harder it is to make further progress – and the harder it is to feel that progress is being made.

    That’s why many an English language-learning journey gets abandoned on the intermediate plateau. With no sense of progression or a tangible, achievable goal on the horizon, the learner can become disoriented and demoralised.

    To draw another travel analogy, when you climb 100 meters up a mountain at 5,000 meters above sea level the effort required is greater than when you climb 100 meters of gentle slope down in the foothills. It’s exactly the same 100 meter distance, it’s just that those hundred 100 meters require progressively more effort the higher up you are, and the steeper the slope. So, how do we keep learners motivated as they pass through the intermediate plateau?

    Education, effort and motivation

    We have a number of tools available to keep learners on track as they start to experience the law of diminishing returns. We can show every bit of progress they are making using tools that capture incremental improvements in ability. We can also provide new content that challenges the learner in a way that’s realistic.

    Setting unrealistic expectations and promising outcomes that aren’t deliverable is hugely demotivating for the learner. It also has a negative impact on teachers – it’s hard to feel job satisfaction when your students are feeling increasingly frustrated by their apparent lack of progress.

    Big data is providing a growing bank of information. In the long term this will deliver a much more precise estimate of effort required to reach higher levels of proficiency, even down to a recommendation of the hours required to go from A to B and how those hours are best invested. That way, learners and teachers alike would be able to see where they are now, where they want to be and a path to get there. It’s a fully functioning English language learning GPS system, if you like.

  • A woman on her laptop smiling and working

    The science behind Smart Lesson Generator: Making teaching easier with AI

    By Thomas Gardner
    Reading time: 4 minutes

    It's 6 AM on a Monday morning. Ms. Lopez wakes up early to prepare for the day ahead. She spends the morning reviewing lesson plans, making sure everything is ready for her students. By lunchtime, she is preparing for the afternoon, grabbing a quick bite between classes... but it doesn’t stop there. The school day finishes but Ms. Lopez stays late marking assignments. Finally, on Sunday night, she sits at her kitchen table, surrounded by papers, course books and lesson plans.

    Does this sound familiar? You are not alone.

    The challenge teachers face

    In 2024, app research found that76% of teachers spend at least one hour of their personal time on lesson planning each week, with 43% spending more than three hours. This is a lot of time that could be spent on other important tasks. Teachers need a solution that helps them plan lessons fast, is connected to their course books and is built by learning experts.

  • Children sat at desks in a classroom, one is smiling and looking to the front of the class

    English: the best second language for your child to learn

    By Steffanie Zazulak
    Reading time: 2 minutes

    As adult learners, our very motivation for learning English can sometimes hinder our progress because we are focusing too much on the end result. The informal way in which children learn English – through music, games and fun activities – offers an environment where they can learn and practise without worrying about the importance of it all. This relaxed attitude, in turn, gives them confidence in learning English and sets them up for more opportunities in their academic pursuits and future career options.

    the positive impact bilingualism has on a child’s cognitive development. Catherine Ford, head teacher of Moreton First Prep School, says that children : “Before children become self-conscious they can try out their newly acquired languages without fear of embarrassment”.

    Starting the English learning process at a young age will provide the head start that most parents are keen to give their children in life, education and career. More than 77% of parents who were interviewed as part of said they would consider sending their child to study at a university abroad, which involves studying in English.

    Educational benefits

    The number of students pursuing postgraduate studies overseas continues to rise, reflecting the global nature of education. According to the seeking diverse academic experiences and cultural immersion. One crucial factor in this journey is having the right level of English skills, especially when applying to universities in popular destinations such as the US, UK, and Australia.

    Learning English from a young age provides a solid foundation, enabling students to tackle more complex language skills tailored to their academic goals. Traditional English teaching often emphasizes reading, writing, and grammar, but studying abroad offers a unique opportunity to immerse oneself in an English-speaking culture, enhancing speaking and listening skills.

    Future career benefits

    Mastering English at an early age can be a transformative asset for future career success. English is the lingua franca of business, opening doors to global opportunities and enabling individuals to pursue diverse career paths across borders. As the most widely used language in business worldwide, proficiency in English is a powerful motivator for students aspiring to join global companies.

    Bilingualism is becoming increasingly advantageous in the job market, improving employability and making candidates more appealing to employers. , underscoring the competitive edge that language skills provide.

    Empowering the next generation

    The benefits your children are given by learning English at a young age are invaluable and as they go through life, the possibilities for advancement in their academic and business careers will be wide open. Children are fortunate to have intuitive language learning capabilities from a young age and this is certainly something to capitalize on.