Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A pile of open books on top of each other

    Using authentic material from the real world to teach English

    Por

    There are lots of resources available to English teachers today: from textbooks to online teaching tools, they can all aid and enrich English lessons. Many ESL teachers also introduce authentic English material into their lessons to expose learners to the language as it is spoken in the real world.

    Authentic material is any material written in English that was not created for intentional use in the English classroom. Using this content for teaching English can make the teaching experience even more engaging, imaginative and motivating for students. It can also be useful to elicit genuine responses from English language learners.

    The great thing about using authentic material is that it is everywhere, which makes it easy to find, and simple for learners to practice English in their own time. Remember that it isn’t limited to articles from newspapers and magazines. Songs, TV programs and films, radio and podcasts, leaflets, menus – anything written in English constitutes authentic material.

    Selecting authentic material

    The best content to select depends on the learners, their level of English and the course content the English teacher wishes to focus on. It’s also a good idea to find out the learners’ interests – after all, there’s no point trying to get students fascinated by a text on the latest sci-fi movie if they’re all fans of action films.

    The materials should reflect a situation that a student may face in an English-speaking environment – this will help them transition into a world where English is the norm. In this world, people use abbreviations, body language is important and they’ll use 'filler' sounds – such as 'ummm' – when they are speaking English – and learners will encounter these in authentic material.

    It’s important not to overwhelm learners with the first piece of authentic material. So, choose articles, songs or sections of TV programs or movies that aren’t too difficult to understand or take too long to get through.

  • a young boy and a young girl sat at desks in a classroom, smiling and looking at another child in front of them

    Five ways to promote friendship in your English language classroom

    Por Joanna Wiseman

    There is a strong link between well-being and friendship, which is just as accurate for children as it is for adults. that children with stable friendships are happier, more able to cope with stress, and have higher self-esteem. Moreover, too, and children who experience friendship adjust more easily to school and perform better academically. School friendships are also a valuable way of learning social skills like sharing, resolving conflict, and engaging with peers positively.

    Having friends is an important part of school life, and teachers can play a significant role in creating a positive classroom culture and helping children and young people to form friendships. How can you promote friendships between your students? Here are some ideas:

    1. Make friendship a central theme in your classes

    If there is an example of a good friendship in a book you are reading with students or in your lesson materials, draw your students’ attention to it. For instance, The Jungle Book is an excellent example of a story about friendship. Encourage your students to think critically about the friendships that they read about. You can ask questions like:

    • What are some things a good friend does?
    • What are the qualities of a good friend?
    • What words do you associate with friendship?

    By discussing friendship regularly in the classroom, your students will learn about the behavior and characteristics of being a good friend.

    2. Create opportunities for interaction

    You can create lots of opportunities for students to build friendships with one another during class time. Design activities that call for pair or group work, and include at least one or two in every lesson. Focusing on a common goal or problem is a great way to encourage your students to bond with one another.As a bonus, this collaborative way of working promotes important 21st century skills like communication and problem-solving.

    Another way of promoting friendships between different groups of students is moving them around. Adjust your seating plan regularly to ensure that all your students interact with and work alongside different classmates, to promote positive, friendly classroom vibes.

    3. Discuss and model good behavior

    Friendship is a social skill that children need to learn – and you can help by teaching them. Be explicit about exactly what you expect from them. Saying, “Be kind” is an abstract, vague concept that children might struggle to put into practice. Instead, give them concrete examples of behavior, such as:

    • take turns when playing
    • speak respectfully
    • share pens or other resources

    When you see a student engaging in these behaviors, draw attention to them and praise them.

    But to really convey the importance of school friendships and kind behavior, it’s not enough to tell them. As the teacher, you must model the behavior you’d like to see in your students. Be consistently positive and upbeat in your interactions with students, thank your learners when they offer help with something, and try to refrain from sounding impatient (even when you are!). All these steps will help to build a positive classroom culture where students feel secure, supported and happy.

    4. Help to navigate difficult situations

    Conflict is an inevitable part of school life, but you can help students to navigate arguments and other difficult situations in the classroom. If two students argue, you can help them to resolve it with the following steps:

    • Separate the students to give them time to calm down.
    • Have a conversation with them. Ask for their perspective on the conflict, and find out how their day has been in general. Sometimes, you can better understand a student’s reaction when you look at the bigger picture. Ask them how they feel, and how they could resolve the situation.
    • Bring the students together again to discuss things and find a resolution together. Reconciliation is the goal.

    Following these simple steps teaches children a framework for resolving conflict and gives them the tools to process strong emotions.

    5. Do some friendship-focused activities

    An excellent way to promote a positive classroom culture is to do friendship-focused activities with your students:

    You could do a Venn diagram activity, where students work in pairs to discuss and write down things they have in common and areas of difference. It’s good to pair students who don’t spend much time together for this activity.

    Another positive activity is a 'Honey Roast'. Give students enough squares of paper to have one for each of their classmates. Get them to write their classmates’ names on the paper, and then, on the other side, write down a compliment or something they like about that person, for example, “I like your handwriting”, or “You always use kind words”. Collect the pieces of paper and read out some nice examples to the class. Alternatively, give them directly to the students to read to themselves.

  • Two young women on a sofa talking to eachother

    9 English conversation mistakes to avoid

    Por Mohamed Shaik

    As humans, we learn and grow through our interactions with other people. Often these encounters are centered around great conversations – rich, meaningful exchanges among a small group of people where each person actively listens and shares. Great discussions are invaluable – they enrich our understanding of people and the world around us. Socially, being someone who can engage effortlessly with others allows us to create deep friendships and gain incredible personal growth and satisfaction. In our careers, we are more effective when we work well with others – the ability to collaborate and solve problems together makes us more effective professionals and makes our businesses more successful.

    Common English conversation mistakes

    But what if you must hold a conversation in English and it is not your native language? One of the many obstacles to learning something new, like English conversation, is that it can be difficult, time-consuming and even a little scary! That’s why we put together a list of nine English conversation mistakes to avoid that apply whether you are speaking with one or several people at once. Keep these tips in mind to help you improve your interactions with people all over the world…

    1) Faking interest in the person

    One of the things that separates a conversation from a 'transaction' (such as ordering something in a restaurant) is the genuine mutual interest of each person in the other. If one isn’t interested in knowing more about the other person, neither will engage meaningfully, and the interaction will become transactional or just 'small talk'. Most people are fascinating – take the time to learn what you can about them.

    2) Discussing negative and sensitive topics

    People are more engaged and willing to share when they are relaxed and happy. Especially when you don’t know someone well, it is always better to focus on the positives – avoid both sharing your biggest troubles and bringing up topics that could be negative from a cultural, religious, political, or even personal perspective. There is always something positive to share!

    3) Trying to 'win' an argument

    Particularly when speaking with someone you don’t know well and/or someone from another country or background, it is precarious for a conversation (especially one where you are practicing your English conversation skills) to evolve into a debate or argument. It is likely that for any two people, there will be many points of disagreement, and if such differences emerge, it’s better to attempt to understand the other person’s point of view rather than to 'win' an argument. It is perfectly acceptable to agree to disagree about specific issues and move on.

    4) Disrespecting others beliefs

    If you want a great conversation, others must feel you aren’t judgmental. When someone feels their ideas and beliefs are questioned or belittled, any meaningful exchange will often shut down. Instead, try listening for understanding, and you may learn something!

    5) 'Hogging' the stage

    It is said that great actors make their fellow performers look great. It is the essence of teamwork, and the same principle applies to great conversationalists. Ask questions that allow others to be positive, confident, and maybe even a little boastful, but certainly remember to do it in a genuine way. The positive energy will be contagious!

    6) Fearing learning something unknown

    There are over seven billion people worldwide, and none are exactly like you! The greatest learning experiences are often from interactions with those who are very different from ourselves. Embrace and celebrate those differences. Allow others to share their unique perspective and journey, always keeping in mind we all share so much in common. We all want to be happy, love others, and have meaning in our lives.

    7) Trying to be someone you are not

    There’s only one person you can be, so don’t try to be someone else or something you are not. A great conversation is based on authenticity; most people can easily sense when another is not truthful or authentic. While keeping in mind all of the other rules, it’s both acceptable and expected for you to share your own journey!

    8) Monopolizing the conversation

    We’ve all been in those conversations where the other person dominates by talking incessantly. At some point, we shut down, just waiting for it to end. Active listening and learning ceases. Engagement requires participation from both parties – don’t hold back from sharing, but at the same time, don’t be that person who dominates and effectively shuts down that engagement.

    9) Focusing on superficial topics

    What separates a great conversation from 'small talk' is the meaningful nature of the dialog. Talking about the weather doesn’t elicit much other than maybe politeness. A great conversationalist elicits meaningful thoughts from others, and those come from purposeful questions. People love to think; asking them something that requires thoughtfulness deepens the conversation's value and strengthens the relationship between the parties.