Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A young child smiling in a classroom with a crayon in his hand.

    Young learners of English deserve more

    Por Ehsan Gorji
    Reading time: 3 minutes

    Imagine a class of English language students aged 8– 9 taught by a dynamic teacher they love. The young learners sit together for two hours, three times a week to learn English as a Foreign Language (EFL). The vibe they bring with them to the class, plus the dynamic teacher and the creativity she develops in her lesson plans, is fantastic.

    I have been observing trends in teaching EFL to young learners, and it is clear to me that school directors, syllabus generators, teachers, parents and learners are all satisfied with this image… “Hooray! Young learners sit together for two hours, three times a week to learn English as a Foreign Language. And the teacher is able to manage the class. Bravo!” But is it enough?

    What causes the lack of focus?

    It all begins with the coursebooks. If you take a coursebook for young learners and thumb through the ‘Scope and Sequence’ pages, you’ll see holistic definitions of language input in each unit. The school authorities then design a course based on the coursebook, and the snowball effect happens, whereby they design a course without specific details on what exactly to focus on.

    It is the teacher’s turn now. The creative and dynamic teacher provides an excellent classroom experience through which young learners can learn English together. She also assigns a piece of homework: write an email to a friend and tell her about your last holiday.

    When the teacher reviews the emails, she smiles as she finds many uses of the simple past tense—both in affirmative and negative forms. She then drafts an email thanking everyone and praising them generously. She includes a link to a PDF of other exercises to reinforce the grammar (the next day in class, they will review the completed handouts).

    This hardworking teacher tries to blend her style with digital literacy and applies creativity along the way. Everything seems perfect in her class, and she regularly receives emails from parents thanking her. Nevertheless, some questions remain: What was the task? What was the learning outcome? Which learning objective should have been tracked?

    Let’s reconsider the task – this time with our critic’s hat on – and analyze what has been taking place in this class. It is very nice that young learners sit together to learn English, and the teacher is able to manage the class successfully, but having fun and ease alone is not enough. We should aim for “fun, ease and outcomes”.*

    *Assessing Young Learners of English: Global and Local Perspectives,Dr Marianne Nikolov, 2016.

    Which important dynamics should be considered?

    The assigned piece of homework said: write an email to a friend and tell her about your last holiday. However, what actually occurred was a shift from this task to the students’ best performance in producing simple past-tense sentences. There are other important dynamics that have migrated out of the teacher’s focus. Did the students begin their emails appropriately? Was the tone appropriate? Did they pay attention to organizing their thoughts into sentences and paragraphs? Was the punctuation correct? Did they end their emails in the right way?

    If the coursebook had been equipped with clear and concrete learning objectives, the course directors would have employed them while designing study syllabuses, and the teacher would have used them when lesson planning. Consequently, the student’s formative and summative progress would have been evaluated against those detailed learning objectives rather than according to what some did better than the average.

    How can learning objectives be applied to tasks?

    With the Global Scale of English (GSE), publishers, course designers, teachers, and even parents can access a new world of English language teaching and testing. This global English language standard provides specific learning objectives for young learners that can be applied to tasks.

    For example, for our task, the GSE suggests the following learning objectives:

    • Can write short, simple personalemails/letters about familiar topics, given prompts or a model.(GSE 40/A2+)
    • Can use appropriate standard greetings and closings in simple, informal personal messages (e.g., postcards or emails). (GSE: 37/A2+)

    By applying language learning chunks – learning objectives, grammar and vocabulary – and identifying the can-do mission each one is supposed to accomplish, teaching and testing become more tangible, practical and measurable. Going back to my original scenario, it is excellent that young learners sit together for two hours, three times a week to learn English as a Foreign Language – provided that we know in detail which learning objectives to focus on, which skills to grow and what learning outcomes to expect.

  • A business woman and man sat at a long table discussing with eachother

    Improving employee engagement: The crucial role of language learning in business

    Por Samantha Ball
    Reading time: 8 minutes

    The ways we approach employee engagement are rapidly evolving and changing. For HR professionals and global business leaders, understanding these trends is essential to encourage a motivated, productive, and loyal workforce. A key yet often overlooked aspect of this engagement is the role of language learning and cultural understanding. Failure to adapt to the international market doesn’t just hinder growth—it can lead to significant financial losses.

    This blog post will delve into current employee engagement trends, provide suggestions for improvement, and talk about the importance of language learning and company culture in fostering a thriving global workforce through an effective employee engagement strategy.

  • Children working together outdoors picking up litter

    How to teach students to be global citizens

    Por
    Reading time: 4.5 minutes

    As teachers, weallwant ourstudents towork toward making the world a better place. Through focusing onglobal citizenship,this drive to change the world is something we can help foster every day in the classroom.In this post, we’ll explore how.

    What are global citizens?

    A global citizen is someone who knows that they are part of a worldwide community. They understand that there are people who have completely different lifestyles, appearances, cultures and routines but with whom we share common values and responsibilities. Global citizenship encourages tolerance and understanding, and learning about it helps children become open-minded adults.

    In a primary English classroom, helping students become aware of themselves as citizens of the world will introduce them to a global way of thinking. We can do this while also helping them become familiar with, and proficient in, English.

    How can we introduce the concept?

    Beforestudents put themselves in a global context, they should get to know themselves as individuals. But they should also get to know themselves as people whoare part oftheir immediate communities.

    In the classroom, this can be done by encouraging students to think about something personal, such as their likes and dislikes. We can then encourage students to look a little further: What kinds of homes do they see in their communities? What makes a house a home to them? What about people working in their communities — what important jobs do they do, and how do they make an impact?

    For language teachers,the idea is tocombine vocabulary and grammar structures with a slowly widening view of our world.Simply by introducing the conceptthat we are part of a worldwide communitycantake the children out of their own experiences and help them start to consider others.

    Tips and activities

    Social media makes it possibleforteachers to contact each other across borders and to collaborate between their schools. Something simple, like organizing a class video call for students after lunchtime and encouraging students in different countries to discuss what they ate in English,canhelp learners become more globally aware.