Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring
  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A class of students sat at desks in a classroom looking at their teacher stood at the front

    5 ways to reinspire your students after the summer holidays

    Por Joanna Wiseman

    The new academic year is here and we're getting ready to head back to the English classroom. Yet, after a long and relaxing summer holiday, some students may feel unmotivated to return to the same class routine, especially if they have been learning English for several years. So, how can we reinspire students to keep learning and reconnect with English? By bringing in new resources, learning approaches and targets, we are sure you'll be able to rekindle their love of learning.

    So let's look at five ways to reinspire your English students in the coming academic year.

    1. Set new goals

    Students may lose interest in classes or feel discouraged when they don't have a clear target to work towards. If this is the case with your class, have them write up a list of five new goals they'd like to achieve.

    These goals must be SMART: Specific, Measurable, Achievable, Relevant and Timely. So rather than just saying "I'd like to learn more vocabulary", have students make it SMART.

    For example:

    Specific: "I'd like to learn new advanced vocabulary to use in my writing."

    Measurable: "I'll test myself to see if I can define and use 20 new words in sentences."

    Achievable: "I will dedicate 2 hours a week to studying the definitions and writing example sentences in context."

    Relevant: "This will help me get a good score in myas I struggle with formal academic language."

    Timely: "I will learn 20 new words by the end of September."

    If learners find it difficult to think of goals, ask them to write one for each language skill: listening, reading, writing and speaking. You can also refer to the GSE Teacher Toolkit, which has hundreds of learning objectives organized by age, level, skill type and more.

    The idea is to encourage them to set clear objectives, giving them an exciting new challenge to work towards for the year ahead.

    2. Encourage students to find conversation partners

    Students may lose interest in improving their English if they've only been studying in a classroom. They may see it as something boring and unrelated to their real lives.

    A great way to tackle this is by encouraging them to talk with English speakers outside of class. By doing this, they'll pick up new vocabulary and expressions, giving them more confidencein their language abilities.

    Suggest that they attend a language exchange.andaregreat platforms to find regular language exchange events in their local area. While this is suitable for intermediate learners and above, it may be a bit daunting for beginners.

    In this case, the appmay be a suitable alternative. Similar to a language exchange, learners can connect with people from around the world. They can choose people with a similar level as them and either write messages, send short audios, or do video calls, depending on their ability and confidence.

    Communicating with real people is a fun and encouraging reason for your learners to want to improve.

    3. Introduce interesting new vocabulary

    Students may become disheartened if they've been learning for years but aren't seeing much progress. A simple and effective way to help them improve their level is by encouraging them toexpand their vocabulary.

    They already have to study a lot of vocabulary from their textbooks, so why not give it a more personal twist and ask for suggestions of topics that interest them?

    Maybe they are gamers and want to learn how to communicate better with other players around the world. Select vocabulary about styles of games, turn-taking, and strategizing that they could use – they can practice in class and be thrilled to be given homework.

    Perhaps some of your students want to study or work abroad. This may be a common topic, but one thing that is not frequently discussed is how to deal with the paperwork of living in another country. For example, getting into more specific language about banking, housing rentals, or setting up wifi will help them feel more confident about their move. Though these things differ between countries, there is a lot of overlapping vocabulary and roleplaying will do wonders to reassure and excite them about their upcoming adventures.

    By allowing your students to take control of their learning, their motivation is naturally higher and you too will enjoy finding out specific language about their interests.

    4. Work on specific problem areas

    Language learners may become frustrated and lose motivation if they continue to make the same mistakes. It may cause them to feel disheartened in their abilities and want to give up, especially for those who aim to sit exams. You can help them level up by identifying specific problem areas and tailoring your classes to work on these.

    Tests can help your learners discover their weaknesses and avoid the frustration of sitting and not passing an exam. They'll be able to pinpoint what they need to work on, and you can dedicate your classes to exactly what they need, rather than cover areas they may not have problems with.

    For example, if students are experiencing difficulties with reading comprehension, you could try introducing more varied reading materials. Ask them to bring in blog posts, magazines and news articles on topics that they find interesting. Highlight keywords in the text to enhance their understanding of the piece and create comprehension questions similar to the test format they'll take.

    By giving a little extra attention to fixing problem areas, learners will soon start to see their progress, encouraging and inspiring them to keep going.

    5. Change your class format

    Sometimes learners become demotivated simply because they have become too used to the format of the classes. If this is the case, you might want to take a break from the textbook and try more creative language learning methods. For example:

    Use interactive games

    Suitable for all levels, you can use platforms such asor to test your learners. They offer a new dimension to the class, encouraging students to have fun with the language. Divide them into teams to add an element of competition – there's nothing like a friendly game to excite students!

    Set project work

    Put your class into small groups and have them work on a project to present to the rest of the group. Choose topics they might cover in their textbooks, such as occupations, travel or cultural traditions. Or even better – let students come up with their own! This activity can be modified to suit all levels and offers a challenge as learners will need to push their language limits.

    Hold class debates

    More suitable for intermediate learners and above, class debates get everyone talking. You can ask students to brainstorm topics they're interested in. You can offer prompts such as climate change, the advertising of junk food or the impacts of social media. They'll be happy to talk about things that concern them.

    Throw in some unexpected activities to bring students' attention back to class and spark their interest in learning again.

  • A group of children sat in a park high fiving each other

    Back to school: 5 team building activities to help break down boundaries

    Por Anna Roslaniec

    At the beginning of term adolescent learners are often shy, embarrassed and awkward. They are reluctant to speak English in front of their peers or show enthusiasm in class, often suffering from social pressure and lack of self-confidence. It can take weeks or months for students to get to know each other and form bonds. However, don’t despair if your teen learners are like this. There are plenty of team-building games and activities you can do to help students build relationships that will allow them to feel comfortable and relaxed in the classroom.

  • Two school girls getting onto. a schoolbus

    4 steps to make sure the transition back to school goes well

    Por Donatella Fitzgerald MBE

    As we start thinking about returning to school, the big question for teachers and parents is: How can we help our children get off to a smooth start?

    After the long break, students might be keen to see their friends again. But it's not always easy to get back into the class routine. It's especially difficult when students are moving into a newclass orare facing important exams.

    So what can you do to support children in transitioning back to school at the beginning of the academic year?

    Here are 4 steps to help them get off to a great start.

    1. Be organized and create routines

    Organizing is what you do before you do something, so that when you do it, it's not all mixed up. – A. A. Milne

    Have a "family meeting" to brainstorm the routines for the school year,e.g., tidying room, getting school clothes/uniform/bag ready the evening before, homework routines, familymeal times and exercise. Ensure each member of the family has input into the routine too. Create a chart with the routines so everyone can see what has been agreed upon and how they are being adhered to.

    2. Start bedtime routines early

    Early to bed and early to rise, makes a man healthy, wealthy, and wise. – Benjamin Franklin

    Agreeing on firm bedtimes during the school week is very important for everyone in the family.

    If children and teenagers do not get enough sleep, it can negatively impact their healthand academic achievement. It's important that they are aware of this too.

    In an ideal world, we should ensure that adults and children are not exposed to any form of screen time for at least 30 minutes to one hour before bedtime. This will help everyone sleep better.

    Start the back-to-school sleep transition gradually. To help your child adjust, move bedtime up by 30 to 60 minutes at a time over the course of a few days or a week before the start of school so the transition from a later bedtime to an earlier one is progressive.

    3. Talk about homework and teach organizational skills

    When it's obvious that the goals cannot be reached, don't adjust the goals, adjust the action steps. – Confucius

    Organization needs to be taught and practiced. As children get older, homework usually increases. Take an active interest in your child's homework.

    Parents can be supportive by demonstrating organizational skills and helping children with time management. Assist them with creating a plan for their homework and encourage your child (whatever age) to have a study plan of some sort and to set goals for their homework which are SMART:

    • Specific – Describe in detail what activities they are going to do.
    • Measurable – How will they know when they are progressing/finished?
    • Achievable – Do they have the skills and resources to get to their goal?
    • Relevant – How does this goal connect to short and long-term targets?
    • Time-bound – Set a concrete deadline.

    Teach them how to approach homework with a "strategy". How much homework do they have? How long will it take to do it? Prioritize urgent homework and do more difficult things when they are less tired.

    Make sure there's adequate space in your house and set up a homework-friendly area that is welllit, andhas a table with enough room to put their pens and books. Of course, this is preferably somewhere quiet.

    Also, it's important that parents are motivators and monitors and that they try to make themselves available for advice. Always praise children for their work and efforts. If you spot any problems, try and address them. Keep distractions to a minimum at home while they are doing their homework.

    4. Make mealtimes quality family time: listen and share one good thing about each day

    Kids who grow up having family dinners, when they're on their own, tend to eat more healthily. – Anne Fishel 

    A meal around the table can bring to the entire family and be an important opportunity for daily interaction. Sitting down to eat as a family provides the opportunity to have an influence over both short and long-term familyhealth, andcan help children establish resilience and the ability to cope with the demands of life as we know it now and in the future. It can also be an opportunity to introduce mindful eating too – being more aware of what and how we are eating.

    Additionally, this time together allows for members to talk and share things about their day and also offers an opportunity to establish a strong and powerful bonding experience. How can we make sure family mealtimes are quality time?

    • Be attentive and offer undivided attention during this time.
    • Turn all modern technology off during the meal so everyone is focused on each other.
    • Talk to each other about topics such as: What lessons do you have today (at breakfast)? What did you enjoy about today? What did you have for lunch (while sharing the evening meal). Tell me one thing you learned today. What made you laugh today? What made you happy today?
    • Listen mindfully to your child's thoughts and worries (if any).
    • Encourage each member to talk about one good thing that has happened to them that day. This lets them know you are there for them.
    • Assign mealtime jobs to involve everyone, e.g.: setting and clearing the table and putting away the dishes.

    However, sometimes it's not possible to share meal times during the week so plan at least one on the weekend if possible.

    The benefits of any small moment of time can have long-lasting positive influences on your child's mental and physical health. Children model adultbehaviorand if they see you eating and engaging positively with them and others, they will carry this into their own lives.

    With a bit of preparation, the leadup and transition back to school can be smooth and enjoyable not only for children but also for the rest of the family.