Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • Children running outside together with balloons

    5 quick and easy ESL games for teaching young learners

    By Joanna Wiseman

    Can we play a game? How many times have you been asked this in class? And how often do you say Yes? Young learners love to play games, and if you choose the right ones, they can have a hugely beneficial impact on their learning.

    As well as being fun, games can provide learners with necessary language practice, as well as lowering the affective filter (i.e. anxiety, fear, boredom and other negative emotions that can all impact learning). Games also foster a positive, relaxed environment.

    So are you ready to play? Here are a few tried and tested games that work especially well in the primary classroom. Each game is designed to consolidate and review the language students have been learning, and take from 5 to 15 minutes. The primary games are flexible enough for you to adapt them to different learner levels, age groups and skills.

  • Two young girls sat at a school desk reading a book

    8 first lesson problems and solutions for young learner classes

    By Joanna Wiseman

    The first class with a new group of young learners can be a nerve-wracking experience for teachers, old and new. Many of us spend the night before thinking about how to make a positive start to the year, with a mixture of nerves, excitement, and a desire to get started. However, sometimes things don’t always go as expected, and it is important to set a few ground rules in those early lessons to ensure a positive classroom experience for all throughout the academic year.

    Let’s look at a few common problems that can come up and how best to deal with them at the start of the school year.

  • A class of students sat at desks in a classroom looking at their teacher stood at the front

    5 ways to reinspire your students after the summer holidays

    By Joanna Wiseman

    The new academic year is here and we're getting ready to head back to the English classroom. Yet, after a long and relaxing summer holiday, some students may feel unmotivated to return to the same class routine, especially if they have been learning English for several years. So, how can we reinspire students to keep learning and reconnect with English? By bringing in new resources, learning approaches and targets, we are sure you'll be able to rekindle their love of learning.

    So let's look at five ways to reinspire your English students in the coming academic year.

    1. Set new goals

    Students may lose interest in classes or feel discouraged when they don't have a clear target to work towards. If this is the case with your class, have them write up a list of five new goals they'd like to achieve.

    These goals must be SMART: Specific, Measurable, Achievable, Relevant and Timely. So rather than just saying "I'd like to learn more vocabulary", have students make it SMART.

    For example:

    Specific: "I'd like to learn new advanced vocabulary to use in my writing."

    Measurable: "I'll test myself to see if I can define and use 20 new words in sentences."

    Achievable: "I will dedicate 2 hours a week to studying the definitions and writing example sentences in context."

    Relevant: "This will help me get a good score in myas I struggle with formal academic language."

    Timely: "I will learn 20 new words by the end of September."

    If learners find it difficult to think of goals, ask them to write one for each language skill: listening, reading, writing and speaking. You can also refer to the GSE Teacher Toolkit, which has hundreds of learning objectives organized by age, level, skill type and more.

    The idea is to encourage them to set clear objectives, giving them an exciting new challenge to work towards for the year ahead.

    2. Encourage students to find conversation partners

    Students may lose interest in improving their English if they've only been studying in a classroom. They may see it as something boring and unrelated to their real lives.

    A great way to tackle this is by encouraging them to talk with English speakers outside of class. By doing this, they'll pick up new vocabulary and expressions, giving them more confidencein their language abilities.

    Suggest that they attend a language exchange.andaregreat platforms to find regular language exchange events in their local area. While this is suitable for intermediate learners and above, it may be a bit daunting for beginners.

    In this case, the appmay be a suitable alternative. Similar to a language exchange, learners can connect with people from around the world. They can choose people with a similar level as them and either write messages, send short audios, or do video calls, depending on their ability and confidence.

    Communicating with real people is a fun and encouraging reason for your learners to want to improve.

    3. Introduce interesting new vocabulary

    Students may become disheartened if they've been learning for years but aren't seeing much progress. A simple and effective way to help them improve their level is by encouraging them toexpand their vocabulary.

    They already have to study a lot of vocabulary from their textbooks, so why not give it a more personal twist and ask for suggestions of topics that interest them?

    Maybe they are gamers and want to learn how to communicate better with other players around the world. Select vocabulary about styles of games, turn-taking, and strategizing that they could use – they can practice in class and be thrilled to be given homework.

    Perhaps some of your students want to study or work abroad. This may be a common topic, but one thing that is not frequently discussed is how to deal with the paperwork of living in another country. For example, getting into more specific language about banking, housing rentals, or setting up wifi will help them feel more confident about their move. Though these things differ between countries, there is a lot of overlapping vocabulary and roleplaying will do wonders to reassure and excite them about their upcoming adventures.

    By allowing your students to take control of their learning, their motivation is naturally higher and you too will enjoy finding out specific language about their interests.

    4. Work on specific problem areas

    Language learners may become frustrated and lose motivation if they continue to make the same mistakes. It may cause them to feel disheartened in their abilities and want to give up, especially for those who aim to sit exams. You can help them level up by identifying specific problem areas and tailoring your classes to work on these.

    Tests can help your learners discover their weaknesses and avoid the frustration of sitting and not passing an exam. They'll be able to pinpoint what they need to work on, and you can dedicate your classes to exactly what they need, rather than cover areas they may not have problems with.

    For example, if students are experiencing difficulties with reading comprehension, you could try introducing more varied reading materials. Ask them to bring in blog posts, magazines and news articles on topics that they find interesting. Highlight keywords in the text to enhance their understanding of the piece and create comprehension questions similar to the test format they'll take.

    By giving a little extra attention to fixing problem areas, learners will soon start to see their progress, encouraging and inspiring them to keep going.

    5. Change your class format

    Sometimes learners become demotivated simply because they have become too used to the format of the classes. If this is the case, you might want to take a break from the textbook and try more creative language learning methods. For example:

    Use interactive games

    Suitable for all levels, you can use platforms such asor to test your learners. They offer a new dimension to the class, encouraging students to have fun with the language. Divide them into teams to add an element of competition – there's nothing like a friendly game to excite students!

    Set project work

    Put your class into small groups and have them work on a project to present to the rest of the group. Choose topics they might cover in their textbooks, such as occupations, travel or cultural traditions. Or even better – let students come up with their own! This activity can be modified to suit all levels and offers a challenge as learners will need to push their language limits.

    Hold class debates

    More suitable for intermediate learners and above, class debates get everyone talking. You can ask students to brainstorm topics they're interested in. You can offer prompts such as climate change, the advertising of junk food or the impacts of social media. They'll be happy to talk about things that concern them.

    Throw in some unexpected activities to bring students' attention back to class and spark their interest in learning again.