Index-Dependent Detection of Insufficient Effort Responding Under Repeated Administration in a Large-Scale Assessment System

Resumen de los autores

Screening for insufficient effort responding (IER) using response matrix data rests on a small family of indices—longstring, intra-individual response variability, even–odd consistency, and multivariate outlyingness—which are known to identify different respondents and are therefore combined into a composite flag. Whether they also disagree about conditions, rather than about cases, has not been examined. It matters more, because a composite can be robust to disagreement over which respondents are flagged while remaining unable to recover a direction of association on which its components differ. The test case is a burden condition created by assessment design: large-scale assessments may require teachers to complete an identical questionnaire once per sampled class, so that burden is the number of questionnaires rather than the length of one. In the SAEB 2023 teacher file the indices associate with burden in opposite directions, and no aggregation of them—union, intersection, or count—recovers an association: an analyst screening with response variability concludes that quality improves with burden, one screening with multivariate outlyingness concludes that it deteriorates, and one screening with the composite concludes that burden is irrelevant. The divergence is sharper on a homogeneous same-valence block, where the multivariate flag rate roughly doubles across the burden range while run-length flagging falls. Decomposition shows where the divergence lives: the outlyingness gradient is carried by between-school composition and is absorbed or reversed under school fixed effects, so the contradictory verdicts are a property of the file as it would operationally be screened, not of a within-teacher degradation process. A separate consequence of repeated administration is that a teacher's questionnaires are near-duplicates of one another, agreeing on about 98% of responses against roughly 67% for two teachers in the same school, so records in such files are far from independent. The screening verdict therefore depends on the index and the cutoff, and composition alone can manufacture a disagreement that no aggregation rule survives.

Áreas y palabras clave

Cómo citar (APA 7)

de Souza, V. V. (2026). Index-Dependent Detection of Insufficient Effort Responding Under Repeated Administration in a Large-Scale Assessment System. https://doi.org/10.31234/osf.io/e895f_v4