Explaining computerized English testing in plain English

蜜桃app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between 蜜桃app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

蜜桃app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with 蜜桃app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using 蜜桃app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor? (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor? (IEA), an automated scoring tool powered by 蜜桃app’s state-of-the-art Knowledge Analysis Technologies? (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from 蜜桃app

  • Childten sat at a desk staring up at a tablet

    Embracing AI for the new school year

    提交者 Thomas Gardner
    Reading time: 2 minutes

    Back-to-school season is the perfect time to introduce something new into your teaching toolkit. With advancements in technology, Artificial Intelligence (AI) is becoming increasingly important for educators to incorporate into their workflow and beneficial for students. Here are some practical ways AI can enhance your classroom experience this new school year.

    AI saves valuable time

    One of the biggest challenges teachers face is finding the time to plan and create engaging lessons. AI can maximize your time by generating word sets, images and activities. Imagine having a virtual assistant that can suggest fresh ideas for your lessons, create tailored resources and even develop entire lesson plans. By automating these time-consuming tasks, AI allows you to focus more on teaching and interacting with your students.

    How AI can help

    • Lesson planning: Generate detailed lesson plans and activities.
    • Lesson inspiration: Get new and creative ideas for classroom activities.
    • Resource creation: Develop word sets, images and other teaching materials.

    As you prepare for the upcoming school year, consider how integrating AI tools can elevate your teaching practice, making your lessons more engaging while freeing up precious time for interactions with your students.

    Personalized learning and targeted feedback

    Every student learns differently, and tailoring lessons to meet individual needs can be challenging. AI can help by providing personalized learning experiences and targeted feedback. Using data-driven insights, AI tools can identify areas where learners are struggling and offer customized recommendations to help them improve. This not only enhances the learning experience but also ensures that no student is left behind.

    Benefits of AI in personalization

    • Personalized lessons: Adapt lessons to suit individual learning styles.
    • Targeted feedback: Provide specific feedback to help students improve.

    Creating an inclusive classroom

    ?AI can play an important role in creating an inclusive classroom environment. AI can generate visuals, audio, videos and text, or a combination of all four, catering to different learning preferences. This ensures that all learners, regardless of their learning styles or abilities, have access to the same educational content.

    AI in inclusion

    • Multimodal content: Use various formats to cater to different learning preferences.
    • Accessibility: Ensure that all students can engage with the material.

    Supporting independent learning

    Encouraging students to take charge of their own learning is essential for their development. AI can support independent learning by providing platforms that facilitate self-study at home. These platforms offer a range of resources and tools that help students learn at their own pace, making education more flexible and accessible.?Explore how supports independent learning with immersive AI-powered conversations.

    AI tools for self-study

    • Independent learning: Foster a culture of self-directed learning.
    • Self-study resources: Provide tools and platforms for students to use at home.

    Integrating AI into your teaching practice can revolutionize the way you educate and engage with your students. From saving time on lesson planning to creating an inclusive classroom and supporting independent learning, the benefits are significant.

    Want more AI tips and tricks for the classroom? Join us on the page for the next video in our series.

  • A smiley woman and one of her friends/students working in a cafe.

    English Teacher Awards 2024: Understanding the categories

    提交者 Thomas Gardner
    Reading time: 4 minutes

    Teachers shape every aspect of our learning experience, especially when it comes to language learning. Great teachers give learners not only the skills but the confidence to go out in the world, start speaking up and discovering new opportunities.?

    We’re celebrating those exceptional educators with the 蜜桃app English Teacher Awards 2024.?

    With five different categories and a Gold, Silver and Bronze winner in each, there are 15 chances to take home thousands of pounds worth of top prizes for the winning teachers and their schools.?

    Find out more about who can enter and the different categories in this article.?