Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • Woman with a open book to her face in a bookshop

    Books to improve your English: Sci-fi and fantasy edition

    Por

    Are you a fan of science fiction and fantasy books? These genres are not only very entertaining but can also help you improve your English skills. You can enhance your language proficiency by reading books with captivating storytelling, unique vocabulary and imaginative concepts.

    Following on from 9 great novels to help improve your English, we've gathered a list of great sci-fi and fantasy books that will transport you to amazing worlds and improve your language skills.

  • Buisness people in a workplace stood around a table laughing

    Business English idioms: Understanding corporate lingo

    Por app Languages

    When it comes to business, communication is key. Knowing how to use business English idioms can really help you express yourself, make connections and achieve success. These colorful expressions add depth and nuance to our conversations, making it easier to communicate confidently and effectively.

    Today, we explore several commonly used idioms you might come across when using English in the workplace, providing examples of how they can be used and shedding light on their meanings.

    Ballpark figure

    A ballpark figure means an estimate or rough guess of a quantity or value. It's a handy way to give a general idea or range without getting too into the details. This can be really helpful during negotiations or planning when you don't need exact numbers. Ballpark figures give you a sense of the bigger picture without bogging you down with specifics.

    In the red

    If someone says they are "in the red", it means they are having a tough time financially and are experiencing losses or negative cash flow. This happens when expenses are more than the money coming in, resulting in a negative balance or deficit. The term originated from using red ink in financial records to indicate negative numbers.

    Blue-sky thinking

    A way to describe a really creative and outside-the-box approach to problem-solving or brainstorming. Basically thinking or brainstorming without any boundaries or limits. A manager might encourage a team to do some blue-sky thinking, allowing them to come up with some very imaginative ideas and concepts.

    Square the circle

    An idiom for attempting an impossible task, originating from a math problem where you can't create a square with the same area as a circle using just a compass and straight edge. Someone may say trying to reach a solution all clients were happy with was like trying to square the circle, implying that it was a difficult and almost impossible task.

    Don't put all your eggs into one basket

    This saying suggests that relying on just one strategy or investment is not wise. It emphasizes the importance of diversification and spreading risks for long-term success. To achieve sustained success, a financial advisor may suggest to their client to diversify their investment portfolio and not solely rely on one option.

    Make ends meet

    The meaning of making ends meet involves being able to complete a task or project using the existing income or resources. It requires efficient management of finances to cover necessary expenses, despite having a limited budget or resources.For instance, if someone is working on a project with a restricted budget, they may need to make ends meet as they cannot obtain additional funds to support it.

    Cover all bases

    This means considering every possible factor and handling them carefully to prevent problems or risks. Making sure everything is examined, and anticipating any potential challenges or situations. A business doing a report or analysis may want to "cover all the bases" to prevent any surprise issues or gaps.

    The origin of this is likely a reference to baseball. When playing defense, the goal is to protect or 'cover' your bases and stop the other team from getting points.

    Get down to brass tacks

    To solve a problem or task, by focusing on the important details and practical steps needed for success. Avoiding unnecessary distractions and concentrating on what's essential.For instance, a project manager might say, "Let's focus on the necessary steps we need to take to complete this project on time by getting down to brass tacks."

    Playing hardball

    "Playing hardball" means being tough and unwavering during negotiations or business deals. It shows that someone is assertive and firm in defending their interests. If the other party is unyielding, a skilled negotiator may suggest "playing hardball" to get a better deal.

    These are just a few of a long list, but by adopting idioms and English phrases like these into your professional interactions, you can strengthen relationships with colleagues and showcase your understanding of the business environment more clearly. Plus, it can give you a boost of confidence in your language learning and in navigating the complex world of corporate English.

    If you want to refine your business English skills even further, make sure to check out our blog post5 ways to politely say no in Business English. You can also check out Mondly, an app developed to help you learn languages for various realistic scenarios.

  • Two parents sat with their two children, writing in a workbook togeher

    5 essentials every child needs when you're teaching English

    Por

    The educational choices available to children are evolving rapidly with apps, online courses, digital games, recordings and videos becoming easily accessible. However, amidst this technological advancement, human evolution has not suddenly accelerated, and the primary aim for teachers remains unchanged - helping children make sense of the world and leaving their mark on it.

    Here are five essential ways we can achieve that for every child, regardless of their circumstances, whether it's teaching English or fostering everyday learning and education.