Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a persons mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test takers appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results C no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academics automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apples Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution C a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between apps system and a human rater is less than that between one human rater and another C in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system acts like a human rater when assessing test takers language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with apps Ordinate technology

The spoken portion of PTE Academic is automatically scored using apps Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor? (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor? (IEA), an automated scoring tool powered by apps state-of-the-art Knowledge Analysis Technologies? (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A teacher holding a tablet to a young student in a classroom sat at a table

    Talking technology: Teaching 21st century communication strategies

    By
    Reading time: 4 minutes

    When my son created a web consulting business as a summer job, I offered to have business cards made for him. Oh Dad, he said, Business cards are so 20th century!

    It was an embarrassing reminder that communication norms are constantly changing, as are the technologies we use. Younger generations share contact information on their phones social media apps, not with business cards. A similar shift has been the move away from business cards featuring fax numbers. Whats a fax? my son might ask.

    Fax machines have had a surprisingly long lifeCthe first fax machine was invented in 1843Cbut they have been largely retired because its easier to send images of documents via email attachments.

    More recent technologies, such as the 1992 invention of text messages, seem here to stay, but continue to evolve with innovations like emojis, a 1998 innovation whose name combines the Japanese words e (picture) and moji (character).?

    The 55/38/7 rule and the three Cs

    Changing technologies challenge language teachers who struggle to prepare students with the formats and the strategies they need to be effective in academic, business, and social settings. These challenges start with questions about why we have particular norms around communication. These norms form a culture of communication.

    The artist/musician Brian Eno defines culture as what we Dzԡ have to do. We may have to walk, but we Dzԡ have to dance. Dancing, therefore, is culture. Communication is full of cultural practices that we Dzԡ strictly need to do, but which make communication more successful. These include practices based on the 55/38/7 Rule and The Three Cs.

    The 55/38/7 rule is often misinterpreted as being about what someone hears when we speak. It actually refers to the insights of University of California professor, , who looked at how our attitudes, feelings, and beliefs influence our trust in what someone says.

    Mehrabian suggests words only account for seven percent of a messages impact; tone of voice makes up 38 percent, and body languageCincluding facial expressionsCaccount for the other 55 percent. The consequence of this for our students is that its sometimes not so important what they are saying as how they are saying it.

    Another way of looking at this nonverbal communication is in terms of The Three Cs: context, clusters, and congruence.?

    Context is about the environment in which communication takes place, any existing relationship between the speakers, and the roles they have. Imagine how each of these factors change if, for example, you met a surgeon at a party compared to meeting the same surgeon in an operating theater where you are about to have your head sawn open.

    Clusters are the sets of body language expressions that together make up a message; smiling while walking toward someone is far different than smiling while carefully backing away.

    Congruence refers to how body language matchesCor doesnt matchCa speakers words. People saying, Of course! Its possible! while unconsciously shaking their heads from side to side are perhaps being less than truthful.

    How does a culture of communication practices translate to new technologies? Mobile phone texts, just like 19th-century telegraph messages before them, need to be precise in conveying their meaning.

    In virtual meetings (on Teams and Google Hangouts, for example), students need to understand that tone of voice, facial expressions, and body language may be more important than the words they share.

    Politeness as one constant

    An additional key concern in virtual meetings is politeness. Once, in preparation for a new textbook, I was involved in soliciting topics of interest to university teachers. I was surprised that several teachers identified the need to teach politeness. The teachers pointed out that the brevity of social media meant that students were often unwittingly rude in their requests (typical email: Wheres my grade!). Moreover, such abruptness was crossing over to their in-person interactions.

    Politeness includes civility, getting along with others, as well as deference, showing respect to those who may have earned it through age, education, and achievement. But politeness is also related to strategies around persuasion and how to listen actively, engage with other speakers by clarifying and elaborating points and ask a range of question types. Online or in person, if students cannot interrupt politely or know when it is better to listen, whatever they have to say will be lost in the court of bad opinion.

    This is particularly important in preparation for academic and business contexts where students need to interact in groups, such as seminar settings and business meetings. Within these, its necessary for students to be able to take on a variety of roles, including leadership, taking notes, and playing devils advocate to challenge what a group thinks.

    Engaging students with project work

    Role-play can help raise awareness of these strategies among students, but its not enough to just take on a variety of roles found in common academic and business exchanges; students need to be able to reflect after each role-play session and infer what strategies are successful.

    Technology-based projects can also help students engage in a range of communication strategies. For example, a app series, StartUp, embraces technology in each unit by sprinkling various text messages and web-based research tasks. There are also multimedia projects where students use their phones to collect images or video and share the results in presentations that develop their critical thinking.

    For example:

    Make your own video

    Step 1 Choose a favorite restaurant or meal.

    Step 2 Make a 30-second video. Talk about the meal. Describe what you eat and drink. Explain why you like it.

    Step 3 Share your video. Answer questions and get feedback.

    This simple project subconsciously reinforces the units vocabulary and grammar. It also allows students to personalize the project based on things that they need to talk about in daily lifeCtheir local foods in this case. This means that each students presentation is unique. Unlike with essay assignments, students tend to work hard to craft several versions until they are satisfied because they know their work will be seen by other students and that they will be asked questions that only they can answer.

    All this forces students to consider speaking strategies, as well as strategies for appropriate facial expressions and body language. Similarly, they have to use active listening strategies when listening to others presentations while asking questions. As technology continues to evolve, teachers need to integrate new applications into their teaching so students learn how to communicate with the tools they have at their disposal.

  • A man sat in a living room with books and plants in the background, he is reading a book

    Words that can't be translated into English

    By Charlotte Guest
    Reading time: 4 minutes

    While English is a rich language, there are some words from other languages that Dzԡ have a direct translation. These words often describe special feelings, situations, or ideas that are deeply connected to their cultures. For example, just as some languages have specific words for different types of weather, other languages have unique words for particular moments or emotions that are hard to explain in English. Here are some interesting examples of untranslatable words that show us the different ways people see the world.

  • Expressing love around the world: Interesting facts and how to say "I love you" in different languages

    By Sam Colley
    Reading time: 5 minutes

    Love is a universal language that transcends borders, cultures and languages. For those learning another language, understanding how to express love in various tongues can be both fascinating and useful. Whether you're planning to travel, connect with friends from different backgrounds, or simply expand your linguistic repertoire, knowing how to say "I love you" in different languages as well as the cultural context, can be a beautiful way to show appreciation and affection. Let's take a look at some of the many ways to express this timeless sentiment and some interesting facts.

    1. Cultural nuances:

    In many cultures, the way you express love can carry different weights and meanings. For example, in Japanese, "ۤƤ" (aishiteru) is a very strong expression of love, often reserved for serious relationships, whereas "ä" (daisuki) is more commonly used among friends and family.
    However, French, often called the language of love, uses "je t'aime" to express love as a romantic phrase that can be used for both partners and close family members.

    2. Gender differences:

    Some languages have gender-specific ways of saying "I love you." For instance, in Arabic, "????" (a?ibbuka) is used when a woman says "I love you" to a man, and "????" (a?ibbuki) is used when a man says it to a woman. Similarly, in Hindi, men say "??? ????? ????? ???? ???" (main tumse pyaar karta hoon) and women say "??? ????? ????? ???? ???" (main tumse pyaar karti hoon).

    3. Formal vs. informal:

    Some languages have formal and informal ways of expressing love. In Spanish, "te amo" is more formal and used for romantic love, while "te quiero" is more casual and can be used for friends and family.
    ?