Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • Two parents sat with their two children, writing in a workbook togeher

    5 essentials every child needs when you're teaching English

    Por

    The educational choices available to children are evolving rapidly with apps, online courses, digital games, recordings and videos becoming easily accessible. However, amidst this technological advancement, human evolution has not suddenly accelerated, and the primary aim for teachers remains unchanged - helping children make sense of the world and leaving their mark on it.

    Here are five essential ways we can achieve that for every child, regardless of their circumstances, whether it's teaching English or fostering everyday learning and education.

  • A businessman smiling outside

    Upskilling and re-skilling for work: Future-proofing your career

    Por Claire Chettiar

    In today's rapidly evolving work landscape, the importance of both upskilling and re-skilling has never been more evident. The skills required in the workplace are continuously shifting due to technological advancements, changing job roles, and increasing global competition. Learning new skills is the key to remaining relevant, competitive, and future proofing your career.

    A recentapp skills outlook reportshows how Indian workers are keen to improve their skills for career growth (with a focus on certain skills in technology and business English) to keep up with progression, and to stay relevant to employers. Following this, in today's post, we explore the significance of upskilling and re-skilling and how they can empower you to thrive in the dynamic world of work.

    Staying relevant in a changing job market

    The job market is in a constant state of flux, with traditional roles being disrupted and new ones emerging. Upskilling allows you to stay ahead of these changes in your current career, and reskilling allows you to adapt and remain relevant in the face of evolving skill requirements. By continuously updating and expanding your skill set, you enhance your employability and increase your chances of securing meaningful employment opportunities, even in the face of automation and job market fluctuations.

    Enhancing professional growth and advancement

    Upskilling is not just about maintaining your current job; it is also a catalyst for professional growth and advancement. By acquiring new skills, you broaden your horizons, open doors to new career possibilities, and position yourself for promotions and higher-level positions.

    Employers value employees who demonstrate a commitment to self-improvement and possess a diverse range of skills, making upskilling a powerful tool for career progression.

    Navigating technological advancements

    Technology continues to reshape the workplace, and upskilling is essential for effectively navigating these advancements. Automation, artificial intelligence, and digitalization are transforming industries, and workers must adapt to remain relevant. Re-skilling in areas such as data analysis, coding, cybersecurity, or digital marketing equips you with the expertise needed to leverage technology and work alongside it, rather than being displaced by it.

    Increasing job satisfaction and motivation

    Learning and growth are fundamental drivers of job satisfaction and motivation: research has shown that it is as significant a factor as low pay in peoples' decision to stay with a particular company or job role1. Upskilling not only expands your knowledge but also stimulates your intellectual curiosity and passion for your work. When you invest in your professional development, you can align your skills with your passions and find greater fulfilment in your chosen career path.

    Nurturing adaptability and resilience

    Upskilling fosters adaptability and resilience, two essential qualities in today's dynamic work environment. The estimates that a quarter of all jobs will be disrupted in the next five years alone; meaning that everyone needs to be prepared for change and uncertainty.

    By continuously learning and developing new skills, you become more adaptable to changing circumstances and can readily embrace new challenges. Upskilling enhances your problem-solving abilities, critical thinking skills, and ability to learn quickly, making you better equipped to handle unexpected situations and thrive in an ever-evolving workplace.

    Remaining competitive in a global economy

    As the world becomes increasingly interconnected, global competition for jobs is intensifying. By staying abreast of industry trends, acquiring in-demand skills, and demonstrating a commitment to self-improvement, you differentiate yourself from other candidates and increase your chances of securing coveted job opportunities, both domestically and internationally.

    The importance of upskilling and re-skilling for work cannot be overstated. In a rapidly changing world, they are the keys to remaining relevant, adaptable, and competitive. By continuously learning and developing new skills, you future-proof your career, enhance your employability, and open doors to new opportunities.

    Embrace skills development as a lifelong journey: by doing so, you empower yourself to thrive in the dynamic world of work and build a fulfilling, successful career.

  • Two students sat on stairs smiling at a laptop

    How to keep your students motivated during summer intensive courses

    Por Richard Cleeve

    Summer is here and so are English language intensive courses. Adults and teens are lining up at your classroom doors, full of enthusiasm and ready to learn English. But soon they’re staring out the windows, looking up at the bright blue skies and wishing they were on the beach, or - frankly - doing anything but studying English for the next 100 hours.

    So how can you turn it around and keep your learners on track and motivated this summer?