Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • children sat at a desk working together

    How to help students deal with conflict in group work

    Por Rachael Roberts

    Is avoiding conflict always a good thing?

    Why is it that some groups work smoothly together, whereas for others it always seems to end in arguments? It may seem logical to avoid group work with classes where it often leads to disagreement, but is there such a thing as ‘healthy disagreement’?

    Perhaps the groups that appear to be working well together are actually just letting one or two people do everything? This would certainly avoid conflict, but they might also be avoiding learning very much.

    Sometimes conflict is a necessary step that teachers (of all subjects, not just language teachers) have to acknowledge as an important part of a learner's development.

  • a woman with headphones around her neck writing notes at a table

    How to reduce anxiety using to-do lists

    Por Rachael Roberts

    Help reduce anxiety using to-do lists

    Many teachers have a to-do list a mile long or even several to-do lists in different places.

    Theoretically, a to-do list is a good idea. When we write something down, the brain can stop holding onto it quite as tightly, which can reduce anxiety and any feeling of overwhelm.

    This tendency to obsess about unfinished tasks is called the Ziergarnik effect, after the Russian psychologist Bluma Ziergarnik, who noticed that waiters only remembered orders before they were served. As soon as the meals were delivered, the memory vanished.

    So, if your brain is constantly nagging you about tasks that remain undone, write them down or make a rough plan and the anxiety will often disappear.

    The downside of ‘to-do’ lists

    I once had a list of jobs that needed doing about the house which had some items dating back several years. Simply writing down a task doesn’t guarantee that it will get done. A lengthy to-do list can enable you to procrastinate and avoid doing important tasks as they are hidden amongst any number of others.

    Long lists can also be overwhelming and off-putting in themselves, and as you cross items off the list becomes cluttered and disorganized.

    Making better use of your to-do list(s)

    1. Learn to prioritize

    It is vital to distinguish between tasks as this will help you decide how to prioritize, delegate and ignore. Categorize your tasks in the following way:

    • Important and urgent (prioritize these).
    • Important but not urgent (book in a time to do these, and stick to it).
    • Urgent but not important (see if you can delegate these, or consider if you need to do them at all).
    • Not urgent and not important (you could almost certainly take these off your list altogether).

    2. Keep separate lists for separate areas of your life

    It’s much easier to see what needs to be done and to prioritize things if you keep separate lists. If you are comfortable with tech, there are plenty of apps to help you with this. You can flag things as important and set reminders and deadline alerts while keeping separate lists. and are just a couple of the options available. If you’re more old school, you can have separate pages in a notebook.

    3. Break tasks down

    Understandably, you might avoid getting started on a big task that will take a long time. There never seems to be a suitable time slot to get it done. Instead, break it down into smaller tasks and tackle them one at a time. For example, break down marking 30 books into three slots of marking ten books.

    4. Know your next actions

    Before you finish work on part of a bigger task, make sure you know your next step, so that when you come back to it, you can get started straight away. This also works well when you decide at the end of a working day what the first task you’ll do tomorrow is.

    5. Review weekly

    A regular review is essential. Look through what you’ve achieved and feel good about it - and remove it from your lists. Analyse where you tried to do too much (try to stick to 3-5 tasks a day) and consider this when setting yourself tasks for the following week. Take note of any tasks you should have done but didn’t get around to and, assuming a certain task is a priority, decide exactly when you will do it next week.

    Good luck!

    Teaching requires you to juggle 101 different things, and that’s before you take your home life into account as well. Get on top of your to-do list, and not only will you feel less overwhelmed, but you may also even find yourself a bit more free time.

  • People of various backgrounds and ages standing together holding paper speech bubbles

    4 poor communication skills (and what to do about them)

    Por Rachael Roberts

    How to help your students improve their general listening, speaking and understanding

    Do your students ever display poor communication skills?

    Most teachers will answer with a resounding ‘yes’. In fact, communication skills do not always come naturally to many people. Let’s look at some of the most common and egregious errors people make when speaking and listening to each other. I’ll also give you some valuable ways to help your students improve.

    1. They don’t even stop to breathe

    If you find one student doing a LOT of talking, it’s probably because no one else can get a word in edgeways. It can be tempting to assume that this is because the chatty student thinks their ideas are better than anyone else’s, but, in fact, it is often a sign of nerves.

    Look more carefully and see if they appear breathless or anxious. Whatever the reason, this kind of student may benefit from a more structured approach where students are given time to prepare what they are going to say, and everyone is expected to contribute equally. Or make it into a game where students have to make sure that they speak for 50% of the time each, as would be expected in an exam situation.

    2. They aren’t really paying attention to each other

    Whether your students are looking at their phones, staring out of the window or can’t wait to interrupt each other, poor listeners make poor communicators. Deal with this by always requiring the listener in any pair to do something specific. For example, tell them they will need to summarise what their partner said, or they have to think of three questions to ask their partner at the end.

    We usually have a reason to listen in real-life communication, so make sure you give them one. Otherwise, they may assume that only the teacher needs to pay attention when another student is talking.

    3. They ramble and it’s difficult to follow what they’re saying

    Students may ramble because they are unconfident about the target language. It’s fine for students to struggle a bit to communicate, but it’s sensible to set achievable tasks, or they may give up.

    It could also be that they would benefit from more rehearsal and practice. Being able to speak fluently ‘off the cuff’ is very challenging, particularly in front of an audience. Try giving students time to plan what they are going to say first. They shouldn’t be reading off the page, but writing it first can help with confidence and fluency.

    Repetition is also invaluable in building fluency. Try doing speaking tasks more than once, and see how the quality and quantity improves each time. You can ring the changes by swapping partners or changing the format from pairs to small groups, to the whole class.

    4. They don’t treat others’ feelings with respect

    This can show itself in different ways. Maybe your students just completely ignore the feelings the other person is demonstrating or telling them about. Or maybe they dismiss them in other ways, ‘Oh, exactly the same thing happened to me! I was just walking along…’

    You can model better ways to respond yourself. For example, “It sounds as if you feel quite angry about that?”, “That must have been really difficult.’

    This teaches students the kind of phrases they can use to validate, empathize and talk about emotions.

    If you can deal with these problems, your students will be well on the way to becoming master communicators.