Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A Teacher sat with a child at a desk in a classroom helping them with their writing,

    5 ways to support students with dyslexia

    Por Anna Hasper

    Children seem to be starting English lessons younger than ever, often before they can even read and write. This means that learning differences like dyslexia may not have yet made themselves apparent.

    While it’s not a language teacher’s role to diagnose specific learning needs, it is important for us to monitor our young learner students’ progress. If we think a student might be showing signs of dyslexia (or another learning difference), we should feel comfortable referring parents to the right place early on. This can make a huge difference in the learning process.

    There are many forms of dyslexia and it affects students in various ways. However, some may include the following:

    • having difficulty reading (especially aloud)
    • struggling with spelling
    • problems remembering the sequence of things
    • finding it hard to follow instructions
    • misbehaving or disrupting the class
    • being very quiet or shy (especially when doing reading or writing activities)
    • falling asleep in class.

    Dyslexia is not a learning disability; it’s a learning difference.

    What do Magic Johnson, Richard Branson and Tom Cruise have in common? They all have dyslexia. So learners with dyslexia are certainly not less capable; in fact, they often excel in spatial thinking and creativity. The difference is that their brain works differently, so they find visual processing and using their working memory challenging. For example, they may struggle to remember what was said and face challenges when trying to link sounds to letters.

    The most common issues are related to reading, spelling and writing, but dyslexia can also impact concentration span and planning skills. And all these challenges have a severe impact on learners’ self-esteem.

    Providing effective learning opportunities for young learners with dyslexia might require teachers to reframe how they see dyslexia. Avoid seeing it as a dis-ability, but rather as a form of neurodiversity: the brain functions and learns in different ways.

    Creating the conditions for learning

    Many – if not most – young learner teachers feel they are not appropriately trained to deal effectively with learners who have dyslexia in a classroom context.

    In an ideal world, all EAL and mainstream teachers would receive in-depth training to better deal with neurodiversity in the classroom. But for now, let’s explore some modifications that help create a more enabling learning environment in which all learners – with or without dyslexia – can progress.

    1. Getting to know them

    If we want all learners to progress to their next level, we need to get to know them. Only then can we provide learning opportunities that start where they are. Get to know their strengths, weaknesses and interests as well as their learning profile; where do they like to work, who do they work well with and what kinds of tasks engage them fully? These are the starting principles of differentiated teaching and all learners will profit from you taking the time to get to know them beyond their name.

    Top tip:

    Observations are an extremely useful tool to gain insight into learners’ levels and learning preferences. My favorite activity is to get young learners to create a personal profile.

    This can be done in their first language – at home with parents – or as a shared writing activity in class. You provide the stem sentences, and learners complete them with drawings or words. You can hang the profiles on the wall and use them to start talking about ‘differences and similarities’. Alternatively, you can have a learner present their buddy to the class based on their profile, depending on the level and age you teach.

    2. Creating a collaborative culture in the classroom

    If we want learners to help each other in class, we need to create a culture of ‘helping hands’. Focusing on developing good relationships in your classroom, between you and the learners but also between learners, is vital for a collaborative culture. Use activities that focus on building understanding through sharing ideas. Integrating collaborative learning activities will help to establish supportive relationships and makes struggling learners feel more confident in the classroom. They know they can first talk things through with others and ask them for help before completing a task independently. This will benefit all learners, not only learners with dyslexia.

    Top tip:

    Think-pair-share is a well-known collaborative activity and can easily be adapted to include some movement too in the form of HuSuPuWu!

    This activity will help learners share ideas and allow for differentiated thinking time. Ask your young learners a question you want them to respond to, give them thinking time and tell them to put their hand up when they are ready to talk (Hu).

    Encourage them to look around, find another person with their hand up and stand up (Su) to walk over and pair up (Pu).

    Together they share ideas before returning to their place and writing up their ideas (Wu).

    This will be especially beneficial for students who need more time to process, love to move and want to get confirmation or support.

    3. Providing multi-sensory tasks and activities

    Providing multi-sensory activities is already common practice in most young learner classrooms. It allows learners to process information using their stronger senses while strengthening their weaker areas.

    Multi-sensory teaching (MST) acknowledges that all brains learn in unique, different ways and is a well-known method used when working with dyslexic students in their mother tongue. So instead of only telling the story, find images that illustrate the events, draw a story path for learners to follow, or get them to visualize the story.

    Doing this increases the ‘routes of memory’ as Kormos (2017) calls it, and enables information to reach the brain via different pathways, visual and auditory, which strengthen the message.

    Top tip:

    When learning new words, break them into syllables by clapping when you say them. Then show the word and break it up visually (e.g. fri-end), and get them to make the word with playdough or in shaving foam as they say it. Get them to keep saying it as they write it and then check it.

    4. Setting clear, manageable instructions

    Because dyslexia often impacts working memory, following instructions can be even more challenging than it already is for young learners. We need to reduce the processing load by breaking up instructions into manageable, achievable steps.

    Focusing on just a small amount of information better enables learners with dyslexia (Kormos & Smith, 2012) and to be honest, all young learners – and our classroom management – can benefit from this.

    Also, check whether you need to ‘tell’ it or can you ‘show’ the instructions? Presenting instructions in a multisensory way where you, for example, use the whiteboard to visualize the instructions, and use gestures and body language to support your oral input will facilitate understanding.

    Top tip:

    Learners benefit from talking things through as talk plays an integral part in meaning-making. So why not get learners to turn to their elbow buddy and repeat what they need to do in their own words? Another effective way would be to record the instructions so they can listen back as many times as they need.

    5. Adapting your materials

    Being aware of what works best for the unique brains of learners with dyslexia allows us to tweak existing materials to make learning more accessible. Think about the color of paper you copy on or the background color of your slides. Learners with dyslexia cope better with colored backgrounds as it reduces word blurring. When learning to write new words in their workbook, use a highlighter to highlight the area between the middle lines where the body of the letters needs to be written.

    Top tip:

    Nowadays, many young learner coursebooks have audio resources available, but not always for readers or stories. Use assistive technology to get the selected reading text recorded. Struggling readers can listen to the audio as they read the text alone. In this way, they will feel that they are reading independently whilst working on letter sound correlation as well as the rhythm of the language.

    The English language classroom can be stressful for learners with specific learning needs. Now, we don’t need to – and can’t – ‘fix’ learners, but we should try to ‘fix’ the environment and provide an enabling, inclusive learning environment for all. By tweaking our teaching, we might better enable learners who face challenges, ensure they feel supported in their learning and allow them to bloom in our classroom.