Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A overhead shot of a  person sat on a chair with a laptop to their left is icons of envelopes

    6 things to consider when writing English emails

    By Steffanie Zazulak

    Every day, an astonishing number of emails are sent and received worldwide. While a considerable amount of these are informal messages between friends, the majority are for business purposes. Whether you’re emailing someone you work with, applying for a new job or making new connections, here are some general rules to follow when writing English emails…

    1. Know your tone

    Always consider who you are writing to and adapt your language accordingly. Emails are less formal than letters, so it’s fine to start your email with “Hi” or “Good morning” – but it may be better to write “Dear…” if you are emailing someone for the first time or if they are senior to you. Similarly, ending emails with “Best/kind regards” rather than “Yours sincerely/faithfully” works well, with the latter being more appropriate for a formal email. Whatever the relationship, though, don’t feel tempted to use laid-back, colloquial expressions like “Hey, you guys”, “Yo!”, or “Hi folks”.

    2. It’s all about the titles

    It’s increasingly common to use first names in international business communications, so don’t be afraid to do so. Another title to consider is your email subject header: a short, clear text is important as busy people often decide whether to even open an email depending on the subject header. Examples of a good subject header include “Meeting date changed”, “Quick question about your presentation”, or “Suggestions for the proposal”.

    3. Use a professional email address

    If you work for a company you’ll be using your company email address. But if you’re using a personal email account because you’re self-employed or looking for a new job, you should be careful when choosing that address.

    You should always have an email address that includes your name so that the recipient knows exactly who is sending the email. Email addresses that you created while you were in school or college (IloveJohn@… or “Beerlover@…) are not appropriate for the workplace!

    4. Limit the small talk

    Small talk can help to build relationships but it doesn’t need to be overly personal. A simple “I hope you are well” or “How are things?” will usually suffice. Also, be cautious with humor as it can easily get lost in translation without the correct tone or facial expressions that accompany face-to-face meetings. It’s safer to leave out humor from emails unless you know the recipient well.

    5. Keep it simple

    Emails are intended to be written, read and understood quickly, so only include the important details – and avoid saturating your message with unnecessary information.

    6. Proofread every message

    Always check your emails before pressing Send. Read and re-read your email a few times, preferably aloud, to ensure there are no spelling or grammatical mistakes. And never just rely on the auto spell-check; spell-checking software doesn't always understand the context of your writing and can throw you off with incorrect suggestions.

    Happy emailing.

  • A overhead shot of a group of people working at a desk with papers and notebooks

    5 ways to make studying in a group work for you

    By Steffanie Zazulak

    There are many benefits to studying in a group, ranging from reducing procrastination and boosting your confidence to gaining new perspectives and learning faster. Many English language learners enjoy working in a group, and many English language teachers recommend it. Here’s how to make studying in a group work for you or your students…

    1. Create an effective group

    Even though you love spending time with your friends, don’t base study partners on friendship. Instead, look for people who stay alert in class, take notes, ask questions, and respond to the teacher’s questions – and don’t make the group too big. An ideal size for a study group is three or four students.

    It’s a great idea to try to meet on the same day and time each week because treating the study session as you would a class helps you to keep to a schedule and ensure that everyone attends.

    Finally, hold study group sessions in a place free of distractions with room to spread out the materials. This will help to ensure that you don’t end up talking about the latest movies or songs instead of studying the future tense!

    2. Decide on the topics and set goals

    Before your study group, think about the topics you’d like to discuss – then agree on one. This will help you concentrate on that topic without straying away. Once you’ve decided on your subject, also consider what you want to achieve at each session – but don’t take on too much material for one session.

    For example, if you’re using a novel written in English to learn more about the language, just think about one aspect of it, such as a couple of the characters, rather than trying to discuss the entire book.

    3. Prepare effectively

    Before attending your study group, prepare by reading more about or researching the topic you’re all going to discuss.

    Also, make a list of anything you aren’t sure you understand so that you can discuss it with your fellow learners. There are sure to be different things you all need explaining in more detail, so you can help each other to understand. Which leads us on to…

    4. Learn from each other by communicating well

    Communicate openly – it doesn’t matter if one of you doesn’t understand something or needs more explanation. And don’t be shy about asking for your peers’ feedback: “Am I talking too much?” or “Did I present your point of view correctly?”. It is often said that it is best to teach other material you understand, and learn from others who understand the material better than you do.

    If your study session reveals points of disagreement or confusion that you cannot resolve as a group, make a note of it and ask your teacher.

    5. Make it enjoyable

    Last on the list, but very important: try to make studying enjoyable in whatever way you can by keeping it interesting. Pick novels about subjects you all really enjoy reading about. Go to see an English-speaking movie together and discuss it at your study group. Or try a quiz together (online or in person) to see how you've picked the subject matter up.

  • a hand reaching for a book from a bookshelf

    9 great novels to help improve your English

    By Steffanie Zazulak

    Reading is one of the most fun and effective ways to help improve your English language skills. It can help to expand your vocabulary and expose you to different sentence structures, all while you enjoy some wonderful stories.

    E-readers and tablets make learning English even easier because if you don’t know a word, you can click on it to read its definition. On the Kindle, you can even add new words you’ve learnt to its Vocabulary Builder feature, which is stored on the device. Others recommend listening to and reading text at the same time as an excellent way to enhance the learning process. Kindle’s Whispersync for Voice is designed for just this purpose and includes audio with selected books, so you can listen and follow the text as you read.

    9 great novels to help improve your English

    Below, we reveal nine novels, including William Golding’s Lord of the Flies and George Orwell’s Animal Farm, to help improve your English.

    1. The Wind in the Willows – Kenneth Grahame

    This classic tale takes place in the English countryside and shares the adventures of the animals that live by the river. Grahame’s simple use of language with imaginative stories makes this a pleasurable read for adults and children.

    2. Lord of the Flies – William Golding

    This book is a modern classic and a popular study text for schools all over the world. When a group of boys are isolated on a desert island, the society they create descends into ruthless behavior. Golding uses dramatic and descriptive language, almost like poetry, making you feel like you’re in the scenes yourself.

    3. The Old Man and the Sea – Ernest Hemingway

    Ernest Hemingway is well-known for his clear, straightforward writing style and short sentence structure, which is great for English language learners and many people have read it in school. It’s the courageous tale of a Cuban fisherman and his battle to land a giant marlin and it’s a perfect introduction to Hemingway as an author.