This study conducted a comparison of assessment behavior between nonnative raters with overseas educational backgrounds (NNSWOEB) and nonnative speakers without overseas educational backgrounds (NNSW/OOEB) in an oral English assessment.
The 8 participants, all nonnative speakers, are divided by their overseas educational background into two categories. Each category, consisting of 4 raters, are required to rate 28 English Education major students' oral English samples. Data analysis was conducted using Minifac (Facets) Rasch software. Results of the analysis revealed that a: there is significant difference in severity between NNSWOEB raters and NNSW/OOEB raters. b: NNSW/OOEB raters are more severe than NNSWOEB raters in every rating criteria except criteria of pronunciation. c: NNSWOEB raters have different rating concept to rating scales with NNSW/OOEB raters.
Study findings add support to previous research that has demonstrated nonnative speakers or speakers without overseas educational background are more severe than native speakers or speakers with overseas educational backgrounds , particularly in grammatical criteria.
Table of Contents
1. Introduction
2. Literature Review
3. Methodology
4. Results and Discussion
5. Conclusion
6. Suggestions and Further Research
Research Objectives and Themes
This study aims to investigate and compare the assessment behaviors of nonnative English raters with and without overseas educational backgrounds. By utilizing the Minifac (Facets) Rasch software, the research explores how these two groups differ in terms of rater severity, consistency, and the interpretation of rating criteria and scales during oral English assessments.
- Comparison of rater severity between NNSWOEB and NNSW/OOEB groups.
- Analysis of consistency and potential rater bias or misfit.
- Evaluation of rating behavior differences across specific analytic criteria.
- Assessment of variations in the application of rating scales.
Excerpt from the Book
Raters' variability
McNAMARA (1996) pointed out some of the important ways in which raters may differ from one another.
1. Two raters may simply differ in their overall leniency
2. Raters may display particular patterns of severity or leniency in relation to only one group of candidates, not others, or in relation to particular tasks, not others, or in relation to particular analytic scores, not others. In general, a rater may be consistently lenient on one item, consistently severe on another; this is a kind of rater-item interaction.
3. Raters may differ from each other in the way they interpret the rating scale they are using.
4. Finally, and rather more obviously, rater may differ in terms of their consistency; that is, the extent of the random error associated with their ratings.
Summary of Chapters
Introduction: Provides the context for the study, highlighting the reliance on nonnative speakers for oral English assessments in Korea and China, and establishes the research questions regarding rater severity, criteria, and scales.
Literature Review: Examines the concept of performance assessment and the established factors of rater variability, including the impact of language background on rating behavior.
Methodology: Details the study design, including the participant pool of 8 raters, the use of IELTS speaking tasks, and the application of Minifac (Facets) Rasch software for data analysis.
Results and Discussion: Presents the statistical analysis of rater severity, consistency, and the application of rating criteria, revealing significant differences between the two groups.
Conclusion: Summarizes the findings, confirming that raters without overseas educational backgrounds tend to be more severe, and identifies specific needs for rater retraining.
Suggestions and Further Research: Discusses the limitations of the current study and proposes future directions, such as increasing sample sizes and investigating the role of rater experience.
Keywords
NNSWOEB, NNSW/OOEB, rater severity, rater consistency, performance assessment, oral English, IELTS, Rasch model, rating behavior, nonnative speakers, overseas education, rater variability, assessment, fluency, grammar
Frequently Asked Questions
What is the core focus of this research?
The study focuses on comparing the assessment behavior of nonnative English raters who have overseas educational backgrounds with those who do not, specifically within the context of oral English proficiency testing.
What are the primary themes investigated in the study?
The study investigates rater severity, the consistency of raters (infit/outfit), the application of specific analytic rating criteria, and the interpretation of rating scales.
What is the main research goal?
The goal is to determine if overseas educational exposure results in significant differences in how nonnative raters judge students' oral English performance.
Which methodology was employed to analyze the data?
The researcher used the Minifac (Facets) Rasch measurement software to quantitatively analyze the assessment data provided by the eight raters.
What topics are covered in the main body of the paper?
The main body covers the theoretical background of rater variability, the specific methodology regarding rater classification and testing instruments, and an extensive statistical discussion of rating patterns, severity, and scale usage.
Which keywords best characterize this work?
Key terms include NNSWOEB, NNSW/OOEB, rater severity, rater consistency, Rasch model, and oral English assessment.
Why are raters without overseas educational backgrounds generally more severe?
The study suggests that nonnative raters without overseas experience tend to judge grammatical errors more strictly compared to their counterparts with overseas educational backgrounds.
How does overseas educational experience affect the usage of rating scales?
The study finds that the two groups adopt different rating scales and have different conceptual understandings of the intervals or "gaps" between score points.
What were the findings regarding rater consistency?
The study found that while most raters fell within an acceptable range, some exhibited significant misfit or overfit, indicating a potential need for targeted retraining to ensure more accurate and consistent assessment.
- Quote paper
- Doctor Yin Xiaoteng (Author), 2016, Assessment behavior between Raters with and without Overseas Education Background in Oral English Assessment. A comparison, Munich, GRIN Verlag, https://www.hausarbeiten.de/document/341436
