Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A group of children stood at a table with their teacher watching her write something down on paper

    Build success beyond the classroom: Critical thinking and assessment

    By Christina Cavage
    Reading time: 4 minutes

    There are some common myths related to critical thinking and assessment. Many people believe that it’s impossible to assess critical thinking, especially in classes where language is limited. However, it can be done! Here, the key to success is crafting tasks and rubrics that allow you to separate language skills and cognitive skills. After all, a low language level doesn’t necessarily reflect your student’s ability to think critically.

    So, how can we measure how a student knows rather than just what they know?

    How to measure critical thinking

    Well, we first have to consider two types of assessment—formal and informal. Formal assessments tend to happen at the end of a task, lesson or skill-building activity and usually focus on the work the student has produced. Then, we have informal assessments. Those are the assessments that involve on-the-spot interactions. These types of assessments play a crucial role in measuring critical thinking.

  • Two friends outside laughing together

    Exploring unusual and funny English sayings

    By Charlotte Guest
    Reading time: 3 minutes

    It's April Fool's Day today and we’re diving into the lighter side of the English language. English is full of quirky sayings and odd words that can leave you scratching your head or laughing out loud. So enjoy this collection of funny English sayings and words that are sure to bring a smile to your face and probably make you question whether this is an April Fool’s Day joke.

    1. Bob’s your uncle

    Let’s start with a classic British saying: “Bob’s your uncle.” This phrase is used to indicate that something will be simple or straightforward. For example, “Just add water, and Bob’s your uncle”. But who is Bob, and why is he your uncle? The origins are unclear, but it’s a fun way to say, “And there you have it”.

    2. The bee’s knees

    When something is the “bee’s knees,” it means it’s excellent or of high quality. But why the knees of a bee? This saying likely originated in the 1920s as part of a trend of using animal anatomy in slang. Other similar phrases include “the cat’s pajamas” and “the snake’s hips”.

    3. More holes than Swiss cheese

    If something has “more holes than Swiss cheese,” it means it’s full of flaws or problems. This saying is self-explanatory if you’ve ever seen a slice of Swiss cheese. It’s a humorous way to point out that something isn’t quite up to par.

    4. Bamboozle

    To “bamboozle” someone means to trick or confuse them. This word itself sounds like a bit of a trick, doesn’t it? It’s believed to have originated in the 17th century, and it’s a perfect word to use on April Fool's Day when you’re pulling pranks on your friends.

    5. Fuddy-duddy

    A “fuddy-duddy” is someone who is old-fashioned or fussy. This term is often used affectionately to describe someone who is a bit behind the times. It’s a funny-sounding word that’s sure to bring a smile to your face.

    6. Gobbledygook

    “Gobbledygook” refers to language that is meaningless or hard to understand, often used to describe overly complex jargon. It was coined during World War II by a Texas congressman who compared confusing bureaucratic language to the sound of a turkey’s gobble. It’s a perfect word for describing those moments when someone is talking but you have no idea what they’re saying.

    7. Mad as a hatter

    If someone is “mad as a hatter,” they’re completely crazy. This saying dates back to the 19th century when hat makers (hatters) often suffered from mercury poisoning due to the chemicals used in hat production, leading to erratic behavior. It’s a whimsical way to describe someone who’s a bit off their rocker.

    8. Raining cats and dogs

    When it’s “raining cats and dogs”, it’s raining very heavily. The origins of this saying are unclear, but it’s a vivid and amusing way to describe a downpour. Just imagine cats and dogs falling from the sky!

    9. Lollygag

    To “lollygag” means to dawdle or waste time. It’s a playful word that perfectly captures the essence of goofing off. So, if you find yourself procrastinating today, just tell everyone you’re lollygagging.

    10. The whole shebang

    “The whole shebang” means everything or the entire thing. The origins of “shebang” are uncertain, but it’s a fun and quirky way to refer to the entirety of something. Whether you’re talking about a project, a meal or a party, this phrase adds a bit of flair.

    11. Cat got your tongue?

    When someone is unusually quiet or at a loss for words, you might ask “Cat got your tongue?” This odd saying dates back to the 19th century and is a humorous way to prompt someone to speak up.

    12. As cool as a cucumber

    If someone is “as cool as a cucumber,” they’re very calm and composed, even in stressful situations. This saying is amusing because it compares a person’s demeanor to a vegetable known for its refreshing coolness. It’s a fun way to compliment someone’s ability to stay unruffled.

    How many unusual English sayings do you know?

    English is a wonderfully weird language, full of sayings and words that can leave us amused and bewildered. On this April Fool's Day, we hope you’ve enjoyed this lighthearted look at some of the funniest expressions English has to offer. So go ahead, sprinkle these sayings into your conversations, and share a laugh with your friends. After all, laughter is the best medicine, and today is the perfect day to embrace the humor in our language.

  • A teacher stood next to a young students desk, the student sat at the desk is high fiving his teacher

    What is the app Smart Lesson Generator?

    By Thomas Gardner
    Reading time: 3 minutes

    How much of your valuable time is spent on planning and administrative tasks instead of teaching? While 93% of educators start their careers to positively impact students' lives, over 75% end up overwhelmed by these non-teaching duties. But we have a tool to help lighten the load.

    Introducing app Smart Lesson Generator– a fast, integrated and efficient solution to streamline lesson planning, allowing you to focus on teaching. Let’s explore the Smart Lesson Generator:

    Meet the new app Smart Lesson Generator

    The app Smart Lesson Generator allows you to create engaging and appropriate lesson plans in seconds. It is connected with app courses designed by experts and aligned with the Global Scale of English (GSE) to ensure effectiveness.