Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • Friends studying together around one big table

    8 things you should try to avoid when learning English

    By
    Reading time: 4 minutes

    Learning a new language is an exciting and rewarding journey, but it can also come with its fair share of challenges. As English learners, it's important to recognize and overcome the common pitfalls that could hinder your progress. Here, we provide advice and guidance on what not to do to make your English learning experience as smooth and successful as possible.

  • Students sat at a table with a teacher stood with them interacting with them

    Why don’t my students speak English in class?

    By Silvia Minardi
    Reading time: 3 minutes

    Last year, I contributed to a national research project with an article titled “?”. The title originated from a concern expressed by a language teacher involved in the project, highlighting a common challenge faced by numerous language teachers. The difficulty of developing learners’ production and interaction skills is a well-known issue in language education.

    Large and increasingly diverse classes, limited time, and learners’ reluctance to speak in class are significant hurdles. During pair and group work, students often revert to their first language (L1), they lack confidence in speaking activities and end up avoiding all interaction in English. These observations are consistent with recent Global Scale of English (GSE) research findings, which indicate that 52% of English learners leave formal education without confidence in their speaking skills.

    Factors contributing to learners’ reluctance

    Several factors contribute to students’ reluctance to speak English in class. Psychological barriers such as lack of motivation, shyness, low self-confidence, fear of making mistakes, anxiety and concerns about negative evaluation play a crucial role. Linguistic challenges, including limited vocabulary, poor pronunciation, and insufficient grammatical skills, further exacerbate the problem.

    Task-related issues can also hinder speaking, especially when tasks are not well-matched to the learner’s proficiency level or focus more on accuracy than communication. Additionally, the classroom environment may not always be conducive to speaking, particularly for learners who need more time to formulate their thoughts before speaking.

    Positive teacher impact

    Fortunately, teachers can positively influence these intertwined factors. By creating a supportive classroom atmosphere and implementing well-designed tasks that prioritize communication over perfection, teachers can encourage reluctant students to participate more actively in speaking activities.

    Leveraging technology: Mondly by app

    One effective tool that can help address these challenges is Mondly by app. This learning companion is especially beneficial for learners who are hesitant to speak in class. Mondly by app offers over 500 minutes of speaking practice, encouraging learners to use English in real-life situations and tasks that prioritize action and communication over accuracy. This approach allows for mistakes - they are part of the game - thus fostering a positive mindset, which is essential if we want to enhance our learners’ speaking skills.

    AI-powered conversations

    A standout feature of Mondly by app is its AI-powered conversation capability, thanks to advanced speech recognition software. This flexibility helps build self-confidence and allows students to experiment with various production and interaction strategies.

    Comprehensive skill development

    Mondly by app is designed not only for speaking but also to develop all four language skills—listening, reading, writing and speaking—and is aligned with the Global Scale of English. The vocabulary for each topic is selected from the GSE vocabulary database, ensuring that learners are exposed to level-appropriate words and phrases.

    Integration into classroom teaching

    To facilitate the integration of Mondly by app into classroom teaching, three GSE mapping booklets have recently been published. These booklets cater to different proficiency levels:

    • Beginner (GSE range: 10-42 / CEFR level: A1-A2+)
    • Intermediate (GSE range: 43-58 / CEFR level: B1-B1+)
    • Advanced (GSE range: 59-75 / CEFR level: B2-B2+)

    These resources provide practical guidance on how to incorporate Mondly by app into lesson plans effectively, ensuring that the tool complements classroom activities and enhances overall language learning.

    Conclusion

    Encouraging students to speak English in class is a multifaceted challenge, but it is not insurmountable. By understanding the various factors that contribute to learners’ reluctance and leveraging innovative tools like Mondly by app, teachers can create a more engaging and supportive learning environment. This approach not only boosts students’ confidence in their speaking abilities but also fosters a more inclusive and interactive classroom atmosphere.

    Embracing technology and aligning teaching practices with modern educational standards, such as the Global Scale of English, can lead to significant improvements in language proficiency and student engagement.

  • Children working together outdoors picking up litter

    How to teach students to be global citizens

    By
    Reading time: 4.5 minutes

    As teachers, weallwant ourstudents towork toward making the world a better place. Through focusing onglobal citizenship,this drive to change the world is something we can help foster every day in the classroom.In this post, we’ll explore how.

    What are global citizens?

    A global citizen is someone who knows that they are part of a worldwide community. They understand that there are people who have completely different lifestyles, appearances, cultures and routines but with whom we share common values and responsibilities. Global citizenship encourages tolerance and understanding, and learning about it helps children become open-minded adults.

    In a primary English classroom, helping students become aware of themselves as citizens of the world will introduce them to a global way of thinking. We can do this while also helping them become familiar with, and proficient in, English.

    How can we introduce the concept?

    Beforestudents put themselves in a global context, they should get to know themselves as individuals. But they should also get to know themselves as people whoare part oftheir immediate communities.

    In the classroom, this can be done by encouraging students to think about something personal, such as their likes and dislikes. We can then encourage students to look a little further: What kinds of homes do they see in their communities? What makes a house a home to them? What about people working in their communities — what important jobs do they do, and how do they make an impact?

    For language teachers,the idea is tocombine vocabulary and grammar structures with a slowly widening view of our world.Simply by introducing the conceptthat we are part of a worldwide communitycantake the children out of their own experiences and help them start to consider others.

    Tips and activities

    Social media makes it possibleforteachers to contact each other across borders and to collaborate between their schools. Something simple, like organizing a class video call for students after lunchtime and encouraging students in different countries to discuss what they ate in English,canhelp learners become more globally aware.