Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A server standing and smiling at a group of ladies smiling sat at a table

    Language and employability skills: Critical thinking, creativity, and communication

    Por

    Why learn a language?

    For most, it's part of academic studies. For some, it's a fun social opportunity. But for many, language learning is aimed at getting a job.

    Language teachers didn't always consider the reasons students were motivated to learn a language. Instead, they focused solely on the central parts of language learning: phonology (sounds of letters and words), morphology (the meaning of parts of words), lexicon (vocabulary), grammar (word order) and to a lesser extent, discourse (the intent of language).

    But today, beyond the mechanical aspects of language teaching and learning, language teachers and their teaching and learning materials try to align with students' motivations. This includes exploring a wide variety of social issues from global warming to racism to homelessness. Reasons for teaching these issues are based on the notion that language is culture, and students want to learn broad topics and be able to contribute to conversations about the issues of the day.

    Employability skills

    A related challenge facing students is employability skills. In the past, students were largely taught the types of language expected of factory workers: giving and responding to simple instructions. Most students learning via the audio-lingual method would consider the question "How are you?" to always be answered with the response, "I'm fine, thank you." The reality, of course, is that you might just as well say, "I'm okay." "Can't complain!" "Not too bad." or even the little-used but truthful, "I feel terrible!"

    The Communicative Approach challenged this pre-programmed speech and reflected changes in the workplace. As robots and artificial intelligence agents take over more and more factory work, today's language students are graduating into jobs that require critical thinking, creativity, and broad communication skills. What are these skills and how do they relate to employability?

    Critical thinking is about examining problems to better understand them. Sometimes critical thinking helps students make choices between one or more alternatives. Like creativity and communication, critical thinking is vital in both academic and employment situations where, for example, staff might try to decide between two locations to build a new factory.

    Creative thinking is about looking for new solutions. In the factory example, a solution might be to build a factory on a boat so it travels between where the raw materials are collected to the market where they're to be sold.

    Communication is about explaining ideas, listening to others' views, and using persuasive speaking and writing to structure arguments. Is the factory boat the best idea? It might be, but without clear communication and debate, it will be tossed aside.

    In terms of employability, the app series Step Up outlines the varied needs faced by adult learners: "to improve their employability skills to get their first job, secure a promotion, find a different job, re-enter the workforce after an absence or change fields."

    Meeting these needs requires new teaching and assessment approaches.

    Be collaborative

    Teaching has to become more collaborative. This reflects the nature of modern work, where most people work in teams, rather than in the factory model where workers were interchangeable parts of a machine. Workers today need to identify problems, share ideas about how to solve them and negotiate, using critical and creative thinking.

    Assess positively

    Similarly, assessment needs to change to a model that allows students opportunities to show what they know in open-ended ways with multiple opportunities to achieve success. Tests with closed-ended questions aimed at tricking students are a thing of the past. Assessment today needs to present students with chances to learn and try again and again until they and their teachers are confident of their abilities.

    Learning a language and related abilities, like employability skills, is not a narrow classroom-bound experience. Students continue to learn and improve throughout their lives. More than anything else, the role of today's teachers is to set their students on a path of lifelong learning.

  • A group of business people chatting over some papers sat at a table in a office

    The ultimate guide to language assessment in business

    Por Samantha Ball
    Reading time: 7 minutes

    Effective international communication is crucial in the world of global commerce. In today's borderless marketplace, companies need to be linguistically flexible to be successful. For HR managers and business practitioners, comprehensive language evaluations are not just an add-on but a vital duty that can enhance your organization's success in the international market.

    This guide will take you through the crucial role of language assessment in business. It will discuss why language assessment is important, how to effectively implement it, its benefits for students and business professionals, and the current tools and technologies available. By the end, you will have the knowledge to identify the significance of language assessment in your business and make informed decisions that reflect the essential role of linguistic ability in the modern global economy.

    Why language assessment matters

    Enhancing communication effectiveness

    Effective communication is the lifeblood of any successful business. When diverse teams come together, be it in an office, on a shop floor, or at a virtual meeting, the ability to exchange information clearly and concisely can mean the difference between streamlined operations and costly misunderstandings. Language assessment ensures that employees possess the necessary language skills needed to communicate across teams and with clients worldwide.

    Global market competitiveness

    Accessing international markets hinges largely on the language capabilities of your workforce. Assessing language skills not only amplifies your organization's ability to engage with prospective clients and partners but also ensures that the customer experience is seamlessly delivered, regardless of the customer's primary language.

    Talent acquisition and retention

    Proper language assessments play a pivotal role in the recruitment process. They aid in identifying candidates with the language proficiencies required for specific roles, such as those involving international correspondence or multilingual customer support. By enabling a streamlined and objective measurement of language ability, your business can attract and retain talent, securing a competitive edge in a multilingual talent pool.

    Types of language assessments

    Proficiency tests

    Proficiency tests are the benchmarks for evaluating a person’s overall language capability. This category includes tests that cover all language skills, such as reading, writing, listening and speaking, like the Versant tests offered by app. These tests provide a comprehensive analysis that helps gauge an individual's ability to function effectively in a language, both in academic settings, the workplace and everyday contexts.

    Interviews

    Conversational skills and the application and comprehension of language in real-world scenarios play a significant role in many job roles. Structured interviews that focus on language are a vital component of the language assessment suite. These interviews typically evaluate oral and sometimes written language skills, providing context-specific insight into how these skills might translate in a professional setting.

    Role-playing scenarios

    Simulations that mimic workplace interactions provide an immersive method of assessing language skills beyond language proficiency tests. Candidates engage in practical scenarios that mirror the challenges and dynamics of a business environment, which is particularly valuable for assessing their ability to manage real-time communication under pressure.

    Implementing language assessment

    Best practices for HR managers

    When starting your language assessment program, begin with a thorough needs analysis. This involves identifying which roles require language proficiency, to what level and in which languages. Then, select or develop an assessment that aligns with these requirements. It's also important to provide clear instructions and support for candidates, ensuring that the assessment process is transparent and equitable.

    Training and development strategies

    Personalized learning plans can be a game-changer for employees who require language improvements. Identify the areas for development, set achievable goals, and incorporate regular assessments to measure progress. This approach ensures that training is targeted and effective, helping employees advance their language skills in a way that's both strategic and supportive.

    Benefits for business professionals

    Career advancement opportunities

    Proficient multilingualism can open doors to international career opportunities and is a key differentiator in many sectors. A dedication to mastering a foreign language can lead to personal and professional growth, including access to leadership positions and broader responsibilities.

    Cross-cultural competence

    Language proficiency is closely tied to cross-cultural understanding and competence. Business professionals who possess these abilities can traverse cultural boundaries with sensitivity and insight, essential for global leadership and successful collaborations in multicultural teams.

    Enhanced collaboration

    Clear and expressive communication fosters an environment of trust and collaboration. By investing in the linguistic abilities and listening skills of your teams, you're not only improving the performance of individual employees but also elevating the collective capacity for innovation and problem-solving within your organization.

  • Teacher speaking to a student with an open book on a desk

    6 tips to enjoy teaching an exam course

    Por John Wolf

    Who gets excited about English exam classes?

    You likely love teaching English, it’s fun, dynamic – almost chaotic at times – and provides the kind of autonomy necessary to be creative and driven.

    Some lessons are far less enjoyable than others - case in point: exam courses. Who gets excited about taking, not to mention preparing for, an English exam?

    Nevertheless, it’s all part of the job and requires finding ways to make exam courses effective for students and enjoyable for teachers.

    Job satisfaction is heavily dependent on our effectiveness as teachers and feeling like we actually accomplish something in the classroom - and is a key factor in avoiding burnout (Maslach & Leiter, 2016).

    While passing an exam is a concrete, achievable goal - a task we are able to accomplish - it also serves as a stressor, a general source of tension and anxiety. So how can we turn this around and enjoy teaching exam courses?