Explaining computerized English testing in plain English

app Languages
a pair of hands typing at a laptop

Research has shown that automated scoring can give more reliable and objective results than human examiners when evaluating a person’s mastery of English. This is because an automated scoring system is impartial, unlike humans, who can be influenced by irrelevant factors such as a test taker’s appearance or body language. Additionally, automated scoring treats regional accents equally, unlike human examiners who may favor accents they are more familiar with. Automated scoring also allows individual features of a spoken or written test question response to be analyzed independent of one another, so that a weakness in one area of language does not affect the scoring of other areas.

was created in response to the demand for a more accurate, objective, secure and relevant test of English. Our automated scoring system is a central feature of the test, and vital to ensuring the delivery of accurate, objective and relevant results – no matter who the test-taker is or where the test is taken.

Development and validation of the scoring system to ensure accuracy

PTE Academic’s automated scoring system was developed after extensive research and field testing. A prototype test was developed and administered to a sample of more than 10,000 test takers from 158 different countries, speaking 126 different native languages. This data was collected and used to train the automated scoring engines for both the written and spoken PTE Academic items.

To do this, multiple trained human markers assess each answer. Those results are used as the training material for machine learning algorithms, similar to those used by systems like Google Search or Apple’s Siri. The model makes initial guesses as to the scores each response should get, then consults the actual scores to see well how it did, adjusts itself in a few directions, then goes through the training set over and over again, adjusting and improving until it arrives at a maximally correct solution – a solution that ideally gets very close to predicting the set of human ratings.

Once trained up and performing at a high level, this model is used as a marking algorithm, able to score new responses just like human markers would. Correlations between scores given by this system and trained human markers are quite high. The standard error of measurement between app’s system and a human rater is less than that between one human rater and another – in other words, the machine scores are more accurate than those given by a pair of human raters, because much of the bias and unreliability has been squeezed out of them. In general, you can think of a machine scoring system as one that takes the best stuff out of human ratings, then acts like an idealized human marker.

app conducts scoring validation studies to ensure that the machine scores are consistently comparable to ratings given by skilled human raters. Here, a new set of test-taker responses (never seen by the machine) are scored by both human raters and by the automated scoring system. Research has demonstrated that the automated scoring technology underlying PTE Academic produces scores comparable to those obtained from careful human experts. This means that the automated system “acts” like a human rater when assessing test takers’ language skills, but does so with a machine's precision, consistency and objectivity.

Scoring speaking responses with app’s Ordinate technology

The spoken portion of PTE Academic is automatically scored using app’s Ordinate technology. Ordinate technology results from years of research in speech recognition, statistical modeling, linguistics and testing theory. The technology uses a proprietary speech processing system that is specifically designed to analyze and automatically score speech from fluent and second-language English speakers. The Ordinate scoring system collects hundreds of pieces of information from the test takers’ spoken responses in addition to just the words, such as pace, timing and rhythm, as well as the power of their voice, emphasis, intonation and accuracy of pronunciation. It is trained to recognize even somewhat mispronounced words, and quickly evaluates the content, relevance and coherence of the response. In particular, the meaning of the spoken response is evaluated, making it possible for these models to assess whether or not what was said deserves a high score.

Scoring writing responses with Intelligent Essay Assessor™ (IEA)

The written portion of PTE Academic is scored using the Intelligent Essay Assessor™ (IEA), an automated scoring tool powered by app’s state-of-the-art Knowledge Analysis Technologies™ (KAT) engine. Based on more than 20 years of research and development, the KAT engine automatically evaluates the meaning of text, such as an essay written by a student in response to a particular prompt. The KAT engine evaluates writing as accurately as skilled human raters using a proprietary application of the mathematical approach known as Latent Semantic Analysis (LSA). LSA evaluates the meaning of language by analyzing large bodies of relevant text and their meanings. Therefore, using LSA, the KAT engine can understand the meaning of text much like a human.

What aspects of English does PTE Academic assess?

Written scoring

Spoken scoring

  • Word choice
  • Grammar and mechanics
  • Progression of ideas
  • Organization
  • Style, tone
  • Paragraph structure
  • Development, coherence
  • Point of view
  • Task completion
  • Sentence mastery
  • Content
  • Vocabulary
  • Accuracy
  • Pronunciation
  • Intonation
  • Fluency
  • Expressiveness
  • Pragmatics

More blogs from app

  • Three young people sat outside eating pizza and smiling

    English food names explained: A culinary journey through language

    Por

    Food is not just a means of nutrition; it can be a vibrant part of a culture's identity. English cuisine, influenced by its history, boasts a range of interesting and sometimes puzzling food names (some even puzzling fluent English speakers).Let's explore the stories behind some of the most well-known English food names.

  • A man smiling in a coffee shop, with a coffee in one hand and paying for it with his card in the other

    Studying in the UK: A students' guide to budgeting

    Por app Languages

    Studying at a UK university can be an amazing experience, but it comes with financial obligations. If you're thinking about studying abroad, it's essential to thoroughly research the cost of living in the UK and create a budget that works for you.

    We’ve created a simple guide to help you get an idea what to think about when it comes to researching the living expenses in the UK for international students. To make sure you're prepared before you head off to the UK.

    Consider all costs

    As you prepare for your trip abroad, there are various expenses you'll have to pay for before leaving your country. These one-time expenses include fees for a UK student visa, university fees for international students, and course materials.

    Although these costs can be significant, it's important to include them in your budgeting.

    Recurring expenses

    After moving to the UK, you will have regular expenses to take care of such as rent, transportation, household and phone bills. These expenses remain consistent throughout the year and can be easily planned for.

    Unplanned expenses

    As an international student in the UK, you may face various living expenses that can vary based on your individual circumstances.

    For example, If you often travel on weekends or spend a lot of time going out with your friends, it's best to overestimate those expenses.

    It's important to consider your expenses for socializing, shopping for clothes, and pursuing hobbies. If you have a limited budget, it's crucial to monitor these expenses closely.

    Researching the cost of living in your area

    If you're planning to study in the UK, keep in mind that the cost of living will depend on your location. Living in a big city versus a small town can result in significantly different accommodation costs.

    For instance, the average one-bedroom rental can range from £1100 in a smaller city like Oxford to £1,695 in metropolitan London.

    If walking isn't an option for you, then you should consider budgeting for transportation. In Glasgow, a monthly pass for public transportation costs £63, while in Manchester, it will cost you £74.

    If you're someone who likes to stay active and exercise, you may also need to account for workout costs. Even with student discounts, joining the university gym in Cambridge will cost students about £35 per month, compared to £22 at the on-campus facilities in Leeds.

    Where to find local information

    Before deciding where to study, it's important to research the cost of living in different cities and regions to ensure it fits within your budget due to the variation in costs.

    Do you know any international students who have studied at the university you plan to attend? Or is there a student organization you can contact for information?

    Reaching out to current students can provide insight into their daily expenses and they might even offer great tips on where to find good deals or places to avoid.

    The cost of student housing varies based on the type of accommodation you prefer. Are you interested in living on campus, renting your own place, or sharing with fellow students off campus? Most schools provide comprehensive details about housing options, including contact information for potential roommates, landlords, or homestay families.

    Handy tip:

    To help with budgeting as a student in the UK, you can use a like the one provided by the Universities and Colleges Admission Service (UCAS).

    The calculator also offers the convenience of selecting your school from a list for accurate local costs.

    Part-time UK jobs for international students

    With all these expenses, a bit of extra income would be helpful for any student. One possible way to supplement your finances during your studies is through tutoring. You can consider tutoring children or your fellow students at the university. It's important to note that international students are allowed to work up to 20 hours per week during the school year while on a student visa.

    Your university's career center can assist you in finding part-time work opportunities, especially if you are looking to gain experience in your field of study. Maybe you're looking to work in a pub or restaurant to improve your English skills and experience the local culture.

    Scholarships are available for both undergraduate and postgraduate studies, either directly from universities or through government initiatives. Non-EU nationals seeking postgraduate studies can apply for .

    An example of a student budget

    Although it may seem overwhelming, estimating your cost of living only requires a few calculations. Simply add up your fixed and variable expenses, as you would if you were living in your home country.

    As a helpful reference, we have provided an example of a monthly budget for a student living in Nottingham. The budget is calculated using average spending data obtained from a UK student budget calculator.

    Item

    Cost

    Rental apartment

    £385

    Transportation (monthly pass with student discount)

    £49

    Utility bills (electricity, gas, water, etc)

    £46

    Gym membership (university gym)

    £23

    Phone and internet

    £27

    Grocery shopping

    £73

    Bank fees

    £30

    Clothes and personal items

    £50

    Takeaway and eating out

    £70

    Total

    £753

    It's common to come across discounts for students, like deals on food at campus eateries, special phone plans, or lower prices at certain shops and restaurants.

    Don't miss out on these opportunities and make sure to ask about them. Keep in mind that this budget doesn't cover expenses for shopping or hobbies, so take those into account based on your individual situation.

    If you're able to work while studying abroad, it can help ease some of the financial strain. UK residents who require additional support may be eligible for .

    There are lots of regarding financing and budgeting whilst abroad, so make sure to do your research and check them out.

  • A teacher stood in front of a classroom of students sat at their desks

    5 ways to deal with mixed ability students in secondary classes

    Por Anna Roslaniec

    No two teenagers are the same. Within all of our classes there tends to be not only a range of English proficiency levels, but also general learning styles, maturity, motivation, and personalities. This diversity can bring some challenges, but also opportunities to vary your classroom activities and teaching methodology.

    Here are some ways to help deal with mixed-ability classes and ensure all your students experience success in their language learning journeys.