Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • A teacher sat in a classroom pointing and smiling,  surrounded by children sat on the floor

    5 ways to keep students attention in class

    Por Joanna Wiseman

    Do you ever find it hard to keep students focused and on task? Young learners get easily distracted and it can be hard to find ways to keep them engaged.

    So what can we do to get, and more importantly, keep our students’ attention? Here are our five top tips.

    1. Plan a range of activities

    Young learners have relatively short attention spans. In the classroom, it is rare to have the whole class fully engaged in something for a long time, since the children will have different interests and levels, so it is essential to plan a number of activities for each lesson.

    The more variety you can include in the activities and tasks you plan, the easier it is to provide something enjoyable and relevant for each child. Choose short tasks and try to have a couple of extra activities up your sleeve if something you planned doesn’t work well. However, don’t worry if you don’t have time to do them all – you can always save them for a future lesson.

    2. Vary the dynamics and pay attention to the mood

    Another way of keeping students engaged is to mix up the classroom dynamics, having a combination of individual heads-down work, pair work, group work, and whole class discussions or games. When planning your lesson, consider how your students might feel at each stage. After doing some reading or quiet work, students may start to become restless, and this is the ideal time to get them up and moving about.

    While you are in class, pay close attention to the mood of the class. When you sense that students are becoming distracted or bored, change the dynamics of the activity.

    3. Use brain breaks

    Ever notice that students become lethargic and show a lack of interest? Why not try introducing brain breaks at strategic points in your lessons? Brain breaks are short physical activities or games designed to get the blood flowing and to re-energize students to help them get ready for learning. They range from short activities that last a couple of minutes, to longer breaks that may be suitable if your lessons last more than an hour.

    4. Peer teaching

    We can vary different aspects of the lesson using the previous strategies, but one thing that rarely changes is the role of the teacher! One way of keeping students involved is by giving them more responsibility and allowing them to take a more active role in their learning.

    Peer teaching completely changes the classroom dynamic and has students teach their peers while you take a step back. For primary classes, ask one or two students to take charge of a ready-made activity, e.g. one from your course book. They should give instructions, demonstrate, monitor as necessary, and check answers.

    When students are used to doing this, you can start to have them work in pairs or small groups to plan their own activities to use in class.

    5. Useful classroom management strategies

    Of course, nobody is perfect and there will be times when you lose students’ attention and they are not on task. For these occasions, you can use a wealth of classroom management strategies to regain the class's attention. Here are a few techniques:

    • Walk around the classroom as students are working. They are less likely to go off-task if you are available and watching.
    • Stand next to or behind individuals who are not paying attention, or move your position to a strategic point in the classroom where everyone, particularly those who are not listening, can see and hear you clearly.
    • Have a code word. Choose a word before the lesson and display it on the board. Tell students that you will sometimes call out this word during the lesson and they need to pay special attention. You could ask students to do an action e.g. stand up and turn around, and give points to the first student who does so.
    • Silence. An old but effective trick is to stand in silence at the front of the class and wait for everyone to stop talking.

    Your enthusiasm is key

    Finally, if we want our students to be motivated and engaged in our lessons, we must show enthusiasm for what we are teaching. The more lively and animated you are about the lesson, the more the students will want to join you and learn.

  • Two Adult students studying in a library, one with a laptop, the other writing

    How can we motivate adult language learners?

    Por

    The problem of motivation

    Have you ever had a problem like Jaime?

    One of my TESOL graduate students in Colombia does. She has a dream job, teaching English at a country club near Bogota. The club attracts international visitors who come to stay, swim, play tennis and golf. Her job is to improve the English skills of everyone from the club’s hotel staff to the tennis coaches, lifeguards, and golf caddies. Most are highly motivated.

    Except for Jaime.

    Jaime, a golf caddy, would slump into class each day, throw down his bag, sit at the back, and automatically take out his phone.

    “Jaime, could you take out your book, please?”

    “Forgot it, teacher.”

    “Jaime, could you answer the question?”

    “No idea, teacher.”

    She tried everything to motivate him, but nothing seemed to work. So, imagine her surprise one day when she walked into class to find him sitting at the front, book open, sharpened pencils ready, and no phone in sight. Although she found it surprising, she didn’t want to embarrass him and instead taught the lesson as normal. Jaime’s hand shot up constantly, either asking her to repeat a point while he took careful notes, or attempting to answer each of her questions.

    What changed? By the end of the week, she had to find out and asked him to stay behind.

    “Because I only speak Spanish,” he sighed, “I only work for local golfers. But I just found out that the caddies who speak English and carry clubs for the international golfers get tipped ten times as much.”

    Jaime had discovered one of two major reasons for adults to learn a second language: career progression. The other, if you can’t guess, is love...removing language as a barrier to intimacy.

    Transferring motivation

    Everyone is motivated about something and that motivation can be translated to the classroom. For example, without mentioning language learning, ask your students to list a few things they have been motivated about in the past, and identify the principles involved.

    Let’s say a student is motivated to improve her soccer skills. Beyond “It’s fun!”, dig deeper and you’ll uncover things such as understanding personal potential, doing something social, and not letting down teammates. If you substitute classmates for teammates, you can see that these are all motivations for learning a language. There are countless more, but focus on what is personal for each student.

    Enemies of motivation

    Beyond a lack of awareness about the advantages of learning English, here are three enemies of motivation.

    Enemy 1: I’m shy.

    It’s a simple truth that more outgoing people have an easier time learning a language; they’re more willing to make an effort and continue trying until they succeed.

    Naturally shy people will still learn – sometimes focusing more on reading and writing – but there are ways to help them be more outgoing. For example, language games can help shy students by increasing a sense of competitiveness and lowering their affective filters, the emotions that interfere with language acquisition such as anxiety and a lack of self-confidence (Krashen, 1992). The connection is confidence; build up students’ confidence and they will be more motivated.

    Enemy 2: If I don’t speak, I won’t make mistakes.

    This issue is often a byproduct of constant teacher interruptions and comments on students’ language use. Be patient, give students time to think before they answer, and don’t always take the answer from the first person to raise a hand. Consider asking everyone to raise their hands before asking one student.

    When you give feedback, focus on errors and ignore mistakes. We all make mistakes in our first and second languages when we speak and generally know we are making them. It’s just that we may be temporarily distracted. Errors, on the other hand, are mistakes that are repeated and the speaker is unaware of them. This leads to fossilization and challenges remediating them. Focus on systematic errors instead of mistakes.

    Also, as a general rule, when students are speaking, observe the same politeness you would with anyone else. Avoid interrupting, and offer corrections later, perhaps privately or in a note.

    Enemy 3: I don’t have time.

    Many adult language learners are heavily committed to other activities; families and work obligations leave them little time to study. However, others may assume they are busy. As a language task, ask students to write a schedule of their typical day and explain what they do hour by hour.

    While studying cello in my 30s, I was fortunate to come across a book by educator John Holt (1991) who also learned cello as an adult. In the book, he said that if you do anything new in a major way, make room in your life by giving up something else. For students, that something else may be part of their social life.

    But even the busiest students have a few minutes between tasks… time waiting in lines, or on a bus. Using a phone app can provide practice.

    Motivation over time

    Students tend to be motivated to different degrees as they progress through a language. At first, students will make rapid progress and have high expectations about learning English. However, as they advance, their progress becomes less noticeable. This can be a difficult time for students.

    Try using 'imagining sessions' with students, where they roleplay themselves 20 years from now as fluent English speakers. What will they be doing? Maybe they’ll be English teachers themselves! Share your journey to becoming someone who now teaches English. Perhaps you were once a golf caddy named Jaime.

  • A young woman taking notes in a lecture theatre, she is sat by other young people.

    Preparing your learners for university study abroad

    Por Richard Cleeve

    Whether your learners are going for a single semester, academic year or an entire university course, studying abroad is an excellent opportunity for them. They’ll have the chance to discover a new culture, develop new skills and make new friends.

    University study in another country also poses several challenges. But as a teacher, you can equip them for this experience and prepare them for future academic success.

    Why study abroad?

    Most people think that studying at university is hard enough, without the added difficulty of doing it overseas. But that doesn’t stop hundreds of thousands of university students from leaving the support of family and friends and relocating to a foreign country.

    People apply to study in another country for a range of reasons. A university program abroad might offer the student better tuition and a greater promise of future employment or simply represent better value for money. And in the case of very specialist university courses, studying abroad may be the only option.

    Whatever the reason, the decision to study in a foreign country is likely to involve a high level of proficiency in another language – and more often than not, that language is English.

    A move towards English language in higher education

    There has been a significant shift in higher education in the last ten years, as many European institutions look to internationalize their programs. As a result, across Europe, we have seen a sharp growth in the number of university courses taught in English. English-taught bachelor’s programs offered by universities in the have multiplied dramatically over the last decade.

    What challenges do learners face?

    Academic skills

    There are a whole range of academic skills that students are expected to know when they start university. From research and evaluation, to note-making and referencing, many learners will enter higher education lacking many of the essential skills they require.

    Studying in a foreign language

    Not only will they have to master new skills, but they may need to do them in a second language. What’s more, even everyday things that fluent speakers may take for granted, such as understanding lectures, reading academic papers, writing essays and even socializing with new friends, will take a lot more effort if English isn’t your first language.

    Administrative issues

    There are many potential pitfalls for a student in a new academic setting. From the administrative process and campus regulations to the types of lessons and assessments, there may be a lot of differences to deal with. Even understanding the etiquette of addressing and interacting with professors can be daunting.

    Problems integrating

    Another challenge is integrating into another culture. Even if the host country is culturally similar, adapting to new surroundings is not always straightforward. There can also be a certain amount of ghettoization, where international students might stick together and remain isolated from the local student population.

    Homesickness

    for international students to deal with. Depending on how far they travel to study, your learners may be unable to return home easily, visit their families and alleviate their homesickness.

    Mental health

    Moving abroad and living in a completely new place can be very stressful and overwhelming, and many factors can exacerbate/cause . Making it harder to do day-to-day tasks, socialize and study.

    Money worries

    Without a grant or a scholarship, studying abroad can be very expensive. If your learners currently live at home with their parents, the cost of accommodation may be formidable. The higher cost of living could mean they have to look for a part-time job to supplement their income. Understanding a country's can also be confusing and hard to calculate into their budget.

    What can you do to get your students ready?

    All of the challenges mentioned above have one thing in common. If a student cannot communicate effectively, these situations can be exacerbated. Language is key, whether it’s accessing support, communicating with professors or getting to grips with a new culture.

    Here are some things you can do to help your learners prepare for university life:

    1) Put them in touch with past students

    It’s important that your learners have a clear idea of what university study abroad entails. Creating a chance for them to speak to other students who have already gone through that experience can be extremely valuable.

    Students who have returned from studying abroad can help with your learners' doubts and put their minds at rest. They might be able to provide essential advice about a specific country or university or simply tell their story. Either way, it’s a great way to reassure and encourage your learners.

    2) Use appropriate authentic content

    In preparation for your learner’s time abroad, the language course that you teach should align with their future linguistic needs. One of the main aims should be to develop the language skills required to perform successfully and confidently in their new context.

    3) Teach them academic study skills

    Think back to when you were at university and what you struggled with. Group work, presentations, critical thinking and exam skills are all things which your learners will need to be proficient in, so the more you practice them in class the better.

    4) Promote autonomous learning

    Success at university is deeply rooted in a student’s ability to work independently and develop practical self-study skills. Giving your learners more choice in the language learning process is one way to encourage autonomy.