Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A group of children sat in a park high fiving each other

    Back to school: 5 team building activities to help break down boundaries

    Por Anna Roslaniec

    At the beginning of term adolescent learners are often shy, embarrassed and awkward. They are reluctant to speak English in front of their peers or show enthusiasm in class, often suffering from social pressure and lack of self-confidence. It can take weeks or months for students to get to know each other and form bonds. However, don’t despair if your teen learners are like this. There are plenty of team-building games and activities you can do to help students build relationships that will allow them to feel comfortable and relaxed in the classroom.

  • Two school girls getting onto. a schoolbus

    4 steps to make sure the transition back to school goes well

    Por Donatella Fitzgerald MBE

    As we start thinking about returning to school, the big question for teachers and parents is: How can we help our children get off to a smooth start?

    After the long break, students might be keen to see their friends again. But it's not always easy to get back into the class routine. It's especially difficult when students are moving into a newclass orare facing important exams.

    So what can you do to support children in transitioning back to school at the beginning of the academic year?

    Here are 4 steps to help them get off to a great start.

    1. Be organized and create routines

    Organizing is what you do before you do something, so that when you do it, it's not all mixed up. – A. A. Milne

    Have a "family meeting" to brainstorm the routines for the school year,e.g., tidying room, getting school clothes/uniform/bag ready the evening before, homework routines, familymeal times and exercise. Ensure each member of the family has input into the routine too. Create a chart with the routines so everyone can see what has been agreed upon and how they are being adhered to.

    2. Start bedtime routines early

    Early to bed and early to rise, makes a man healthy, wealthy, and wise. – Benjamin Franklin

    Agreeing on firm bedtimes during the school week is very important for everyone in the family.

    If children and teenagers do not get enough sleep, it can negatively impact their healthand academic achievement. It's important that they are aware of this too.

    In an ideal world, we should ensure that adults and children are not exposed to any form of screen time for at least 30 minutes to one hour before bedtime. This will help everyone sleep better.

    Start the back-to-school sleep transition gradually. To help your child adjust, move bedtime up by 30 to 60 minutes at a time over the course of a few days or a week before the start of school so the transition from a later bedtime to an earlier one is progressive.

    3. Talk about homework and teach organizational skills

    When it's obvious that the goals cannot be reached, don't adjust the goals, adjust the action steps. – Confucius

    Organization needs to be taught and practiced. As children get older, homework usually increases. Take an active interest in your child's homework.

    Parents can be supportive by demonstrating organizational skills and helping children with time management. Assist them with creating a plan for their homework and encourage your child (whatever age) to have a study plan of some sort and to set goals for their homework which are SMART:

    • Specific – Describe in detail what activities they are going to do.
    • Measurable – How will they know when they are progressing/finished?
    • Achievable – Do they have the skills and resources to get to their goal?
    • Relevant – How does this goal connect to short and long-term targets?
    • Time-bound – Set a concrete deadline.

    Teach them how to approach homework with a "strategy". How much homework do they have? How long will it take to do it? Prioritize urgent homework and do more difficult things when they are less tired.

    Make sure there's adequate space in your house and set up a homework-friendly area that is welllit, andhas a table with enough room to put their pens and books. Of course, this is preferably somewhere quiet.

    Also, it's important that parents are motivators and monitors and that they try to make themselves available for advice. Always praise children for their work and efforts. If you spot any problems, try and address them. Keep distractions to a minimum at home while they are doing their homework.

    4. Make mealtimes quality family time: listen and share one good thing about each day

    Kids who grow up having family dinners, when they're on their own, tend to eat more healthily. – Anne Fishel 

    A meal around the table can bring to the entire family and be an important opportunity for daily interaction. Sitting down to eat as a family provides the opportunity to have an influence over both short and long-term familyhealth, andcan help children establish resilience and the ability to cope with the demands of life as we know it now and in the future. It can also be an opportunity to introduce mindful eating too – being more aware of what and how we are eating.

    Additionally, this time together allows for members to talk and share things about their day and also offers an opportunity to establish a strong and powerful bonding experience. How can we make sure family mealtimes are quality time?

    • Be attentive and offer undivided attention during this time.
    • Turn all modern technology off during the meal so everyone is focused on each other.
    • Talk to each other about topics such as: What lessons do you have today (at breakfast)? What did you enjoy about today? What did you have for lunch (while sharing the evening meal). Tell me one thing you learned today. What made you laugh today? What made you happy today?
    • Listen mindfully to your child's thoughts and worries (if any).
    • Encourage each member to talk about one good thing that has happened to them that day. This lets them know you are there for them.
    • Assign mealtime jobs to involve everyone, e.g.: setting and clearing the table and putting away the dishes.

    However, sometimes it's not possible to share meal times during the week so plan at least one on the weekend if possible.

    The benefits of any small moment of time can have long-lasting positive influences on your child's mental and physical health. Children model adultbehaviorand if they see you eating and engaging positively with them and others, they will carry this into their own lives.

    With a bit of preparation, the leadup and transition back to school can be smooth and enjoyable not only for children but also for the rest of the family.

  • friends chatting around a laptop

    Realistic practice scenarios to boost your English skills

    Por app Languages

    It's essential to engage in real-life conversations to truly become proficient in a language. By immersing yourself in realistic situations, you can really boost your English skills. Of course, it can be difficult to find people to speak to if you aren’t in an area with many English speakers. By practicing different possible scenarios, when the time or situation arises, you know you’ll be ready to take on the challenge.

    You might be preparing to work or study abroad, trying to connect with a new friend or improving your workplace communication. Let’s explore some realistic practice scenarios that can help you feel more confident and fluent when speaking English in your everyday life.