Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A woman with glasses thinking with her hand to her mouth, stood in front of a pink background

    5 of the strangest English phrases explained

    By Steffanie Zazulak

    Here, we look at what some of the strangest English phrases mean – and reveal their origins…

    Bite the bullet

    Biting a bullet? What a strange thing to do! This phrase means you’re going to force yourself to do something unpleasant or deal with a difficult situation. Historically, it derives from the 19th century when a patient or soldier would clench a bullet between their teeth to cope with the extreme pain of surgery without anesthetic. A similar phrase with a similar meaning, “chew a bullet”, dates to the late 18th century.

    Use it: “I don’t really want to exercise today, but I’ll bite the bullet and go for a run.”

    Pigs might fly

    We all know that pigs can’t fly, so people use this expression to describe something that is almost certain never to happen. It is said that this phrase has been in use since the 1600s, but why pigs? An early version of the succinct “pigs might fly” was “pigs fly with their tails forward”, which is first found in a list of proverbs in the 1616 edition of John Withals’s English-Latin dictionary, A Shorte Dictionarie for Yonge Begynners: “Pigs fly in the ayre with their tayles forward.” Other creatures have been previously cited in similar phrases – “snails may fly”, “cows might fly”, etc, but it is pigs that have stood the test of time as the favored image of an animal that is particularly unsuited to flight! This phrase is also often used as a sarcastic response to mock someone’s credulity.

    Use it: “I might clean my bedroom tomorrow.” – “Yes, and pigs might fly.”

    Bob’s your uncle

    Even if you don’t have an uncle called Bob, you might still hear this idiom! Its origin comes from when Arthur Balfour was unexpectedly promoted to Chief Secretary for Ireland by the Prime Minister of Britain, Lord Salisbury, in 1900. Salisbury was Arthur Balfour’s uncle (possibly his reason for getting the job!) – and his first name was Robert. This phrase is used when something is accomplished or successful – an alternative to “…and that’s that”.

    Use it: “You’re looking for the station? Take a left, then the first right and Bob’s your uncle – you’re there!”

    Dead ringer

    This phrase commonly refers to something that seems to be a copy of something – mainly if someone looks like another person. The often-repeated story about the origin of this phrase is that many years ago, people were sometimes buried alive because they were presumed dead – when actually they were still alive. To prevent deaths by premature burial, a piece of string would supposedly be tied to the finger of someone being buried – and the other end would be attached to a bell above ground. If the person woke up, they would ring the bell – and the “dead” ringer would emerge looking exactly like someone buried only a few hours ago! Other stories point to the practice of replacing slower horses with faster horses – “ringers”. In this case, “dead” means “exact”.

    Use it: “That guy over there is a dead ringer for my ex-boyfriend.”

    Off the back of a lorry

    This is a way of saying that something was acquired that is probably stolen, or someone is selling something that’s stolen or illegitimate. It can also be used humorously to emphasize that something you bought was so cheap that it must have been stolen! “Lorry” is the British version – in the US, things fall off the back of “trucks”. An early printed version of this saying came surprisingly late in The Times in 1968. However, there are many anecdotal reports of the phrase in the UK from much earlier than that, and it is likely to date back to at least World War II. It’s just the sort of language that those who peddled illegal goods during and after WWII would have used.

    Use it: “I can’t believe these shoes were so cheap – they must have fallen off the back of a lorry.”

  • Two ladies in a pottery studio, one with a clipboard, both looking at a laptop together

    11 ways you can avoid English jargon at work

    By Steffanie Zazulak

    From “blue-sky thinking” to “lots of moving parts”, there are many phrases used in the office that sometimes seem to make little sense in a work environment. These phrases are known as ‘work jargon’ – or you might hear it referred to as ‘corporate jargon’, ‘business jargon’ or ‘management speak’. It’s a type of language generally used by a profession or group in the workplace, and has been created and evolved over time. And whether people use this work jargon to sound impressive or to disguise the fact that they are unsure about the subject they are talking about, it’s much simpler and clearer to use plain English. This will mean that more people understand what they are saying –both fluent and second-language English speakers.

    The preference for plain English stems from the desire for communication to be clear and concise. This not only helps fluent English speakers to understand things better, but it also means that those learning English pick up a clearer vocabulary. This is particularly important in business, where it’s important that all colleagues feel included as part of the team and can understand what is being said. This, in turn, helps every colleague feel equipped with the information they need to do their jobs better, in the language they choose to use.

    Here, we explore some of the most common examples of English jargon at work that you might hear and suggest alternatives you can use…

    Blue-sky thinking

    This refers to ideas that are not limited by current thinking or beliefs. It’s used to encourage people to be more creative with their thinking. The phrase could be confusing as co-workers may wonder why you’re discussing the sky in a business environment.

    Instead of: “This is a new client, so we want to see some blue-sky thinking.”

    Try saying: “This is a new client, so don’t limit your creativity.”

    Helicopter view

    This phrase is often used to mean a broad overview of the business. It comes from the idea of being a passenger in a helicopter and being able to see a bigger view of a city or landscape than if you were simply viewing it from the ground.Second-language English speakers might take the phrase literally, and be puzzled as to why someone in the office is talking about taking a helicopter ride.

    Instead of: “Here’s a helicopter view of the business.”

    Try saying: “This is a broad view of the business.”

    Get all your ducks in a row

    This is nothing to do with actual ducks; it simply means to be organized. While we don’t exactly know the origin of this phrase, it probably stems from actual ducklings that walk in a neat row behind their parents.

    Instead of: “This is a busy time for the company, so make sure you get all your ducks in a row.”

    Try saying: “This is a busy time for the company, so make sure you’re as organized as possible.”

    Thinking outside the box

    Often used to encourage people to use novel or creative thinking. The phrase is commonly used when solving problems or thinking of a new concept. The idea is that, if you’re inside a box, you can only see those walls and that might block you from coming up with the best solution.

    Instead of: “The client is looking for something extra special, so try thinking outside the box.”

    Try saying: “The client is looking for something extra special, so try thinking of something a bit different to the usual work we do for them.”

    IGUs (Income Generating Units)

    A college principal alerted us to this one – it refers to his students. This is a classic example of jargon when many more words are used than necessary.

    Instead of: “This year, we have 300 new IGUs.”

    Try saying: “This year, we have 300 new students.”

    Run it up the flagpole

    Often followed by “…and see if it flies” or “…and see if anyone salutes it”, this phrase is a way of asking someone to suggest an idea and see what the reaction is.

    Instead of: “I love your idea, run it up the flagpole and see if it flies.”

    Try saying: “I love your idea, see what the others think about it.”

    Swim lane

    A visual element – a bit like a flow chart –  that distinguishes a specific responsibility in a business organization. The name for a swim lane diagram comes from the fact that the information is broken up into different sections – or “lanes” – a bit like in our picture above.

    Instead of: “Refer to the swim lanes to find out what your responsibilities are.”

    Try saying: “Refer to the diagram/chart to find out what your responsibilities are.”

    Bleeding edge

    A way to describe something that is innovative or cutting edge. It tends to imply an even greater advancement of technology that is almost so clever that it is unbelievable in its current state.

    Instead of: “The new technology we have purchased is bleeding edge.”

    Try saying: “The new technology we have purchased is innovative.”

    Tiger team

    A tiger team is a group of experts brought together for a single project or event. They’re often assembled to assure management that everything is under control, and the term suggests strength.

    Instead of: “The tiger team will solve the problem.” 

    Try saying: “The experts will solve the problem.” 

    Lots of moving parts

    When a project is complicated, this phrase is sometimes used to indicate lots is going on.

    Instead of: “This project will run for several months and there are lots of moving parts to it.”

    Try saying: “This project will run for several months and it will be complicated.”

    A paradigm shift

    Technically, this is a valid way to describe changing how you do something and the model you use. The word “paradigm” (pronounced “para-dime”) is an accepted way or pattern of doing something. So the “shift” part means that a possible new way has been discovered. Second-language English speakers however, might not be familiar with the meaning and might be confused about what it actually means.

    Instead of: “To solve this problem, we need a paradigm shift.”

    Try saying: To solve this problem; we need to think differently.”