Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • a young boy and a young girl sat at desks in a classroom, smiling and looking at another child in front of them

    Five ways to promote friendship in your English language classroom

    By Joanna Wiseman

    There is a strong link between well-being and friendship, which is just as accurate for children as it is for adults. that children with stable friendships are happier, more able to cope with stress, and have higher self-esteem. Moreover, too, and children who experience friendship adjust more easily to school and perform better academically. School friendships are also a valuable way of learning social skills like sharing, resolving conflict, and engaging with peers positively.

    Having friends is an important part of school life, and teachers can play a significant role in creating a positive classroom culture and helping children and young people to form friendships. How can you promote friendships between your students? Here are some ideas:

    1. Make friendship a central theme in your classes

    If there is an example of a good friendship in a book you are reading with students or in your lesson materials, draw your students’ attention to it. For instance, The Jungle Book is an excellent example of a story about friendship. Encourage your students to think critically about the friendships that they read about. You can ask questions like:

    • What are some things a good friend does?
    • What are the qualities of a good friend?
    • What words do you associate with friendship?

    By discussing friendship regularly in the classroom, your students will learn about the behavior and characteristics of being a good friend.

    2. Create opportunities for interaction

    You can create lots of opportunities for students to build friendships with one another during class time. Design activities that call for pair or group work, and include at least one or two in every lesson. Focusing on a common goal or problem is a great way to encourage your students to bond with one another.As a bonus, this collaborative way of working promotes important 21st century skills like communication and problem-solving.

    Another way of promoting friendships between different groups of students is moving them around. Adjust your seating plan regularly to ensure that all your students interact with and work alongside different classmates, to promote positive, friendly classroom vibes.

    3. Discuss and model good behavior

    Friendship is a social skill that children need to learn – and you can help by teaching them. Be explicit about exactly what you expect from them. Saying, “Be kind” is an abstract, vague concept that children might struggle to put into practice. Instead, give them concrete examples of behavior, such as:

    • take turns when playing
    • speak respectfully
    • share pens or other resources

    When you see a student engaging in these behaviors, draw attention to them and praise them.

    But to really convey the importance of school friendships and kind behavior, it’s not enough to tell them. As the teacher, you must model the behavior you’d like to see in your students. Be consistently positive and upbeat in your interactions with students, thank your learners when they offer help with something, and try to refrain from sounding impatient (even when you are!). All these steps will help to build a positive classroom culture where students feel secure, supported and happy.

    4. Help to navigate difficult situations

    Conflict is an inevitable part of school life, but you can help students to navigate arguments and other difficult situations in the classroom. If two students argue, you can help them to resolve it with the following steps:

    • Separate the students to give them time to calm down.
    • Have a conversation with them. Ask for their perspective on the conflict, and find out how their day has been in general. Sometimes, you can better understand a student’s reaction when you look at the bigger picture. Ask them how they feel, and how they could resolve the situation.
    • Bring the students together again to discuss things and find a resolution together. Reconciliation is the goal.

    Following these simple steps teaches children a framework for resolving conflict and gives them the tools to process strong emotions.

    5. Do some friendship-focused activities

    An excellent way to promote a positive classroom culture is to do friendship-focused activities with your students:

    You could do a Venn diagram activity, where students work in pairs to discuss and write down things they have in common and areas of difference. It’s good to pair students who don’t spend much time together for this activity.

    Another positive activity is a 'Honey Roast'. Give students enough squares of paper to have one for each of their classmates. Get them to write their classmates’ names on the paper, and then, on the other side, write down a compliment or something they like about that person, for example, “I like your handwriting”, or “You always use kind words”. Collect the pieces of paper and read out some nice examples to the class. Alternatively, give them directly to the students to read to themselves.