What Is Criterion Contamination in Personality Assessment?

What is criterion contamination in personality assessment evidence?

Criterion contamination occurs when an outcome used to evaluate a personality score is systematically influenced by factors outside the outcome the study intends to measure. A criterion is the measured outcome used to examine an interpretation of a score. If a supervisor knows an employee’s personality result before rating their work, expectations may shape the rating. Differences in equipment, work location, shift, or opportunity can also affect recorded outcomes. The Society for Industrial and Organizational Psychology’s Principles for the Validation and Use of Personnel Selection Procedures defines contamination as extraneous, systematic variance. This guidance concerns personnel selection. The concern is that the observed score–outcome relationship may partly reflect outside influences. It need not be inflated: the direction and importance depend on how an influence relates to the score and how the outcome is produced. This is about the criterion side of evidence, separate from whether the personality score itself is consistent or useful. A possible source of contamination also does not prove the reported relationship false. Before accepting a broad claim, find out what counted as the outcome, who measured it, and under what conditions. The conclusion should be no broader than the evidence about that outcome.

Sources: Principles for the Validation and Use of Personnel Selection Procedures; In the Army Now: The United States Army and the All-Volunteer Force

What exactly is contaminated: the score, the criterion, or both?

In a study of a personality assessment, the score is the predictor being examined; the criterion is the measured outcome against which the score is compared. That outcome might be a behavior count, a supervisor rating, performance on a training task, or another recorded result. Criterion contamination means that the outcome measure includes systematic influences that do not belong to the intended outcome. The concern is therefore about how the criterion was defined and produced. It does not, by itself, mean the personality score is contaminated, the entire study is worthless, or the reported association must be false. The Society for Industrial and Organizational Psychology’s Principles for the Validation and Use of Personnel Selection Procedures defines contamination as extraneous, systematic variance in a criterion measure. “Systematic” matters: an influence is not just a one-off accident, but a factor that tends to shape recorded outcomes in a patterned way. The Principles list differences in machinery quality, unequal sales territories, job tenure, work shift, job location, and raters’ knowledge of predictor scores or attitudes. These are workplace examples from selection-validation guidance; the underlying measurement point is narrower and portable: a measure called “performance” may also reflect conditions under which performance was observed. Consider a rating intended to represent how effectively people complete a task. If one group has better equipment or more time, those conditions can affect the rating alongside task behavior. A different route arises when a rater knows a person’s personality result before scoring the criterion. The rater may then interpret ambiguous behavior through an expectation linked to that result. This does not establish that the rater was biased in a particular study; it identifies a possible pathway that the study’s design or reporting should address. The Society’s Principles specifically name both predictor-score knowledge and rater attitudes among possible contaminating factors. In the chapter “Evaluating Psychological Tests,” the authors likewise describe a supervisor who receives test information and may develop expectations that influence later performance. The distinction between the score and criterion is useful because either side can raise questions, and both could do so in the same study. Here, however, criterion contamination names an outcome-side problem: systematic influences enter what is treated as the result. A personality inventory could be administered consistently while the outcome it is compared with still reflects shift conditions, access to resources, or expectations. Do not collapse those into one vague verdict that “the test is biased.” Identify which measure and which process the evidence actually implicates. A broad label is not enough to tell. “Teamwork,” “adjustment,” or “job performance” names an idea, not the observations used to represent it. The Society’s Principles says criteria should have a stated rationale connecting them to the proposed use and gives examples such as work behaviors, outputs, training performance, and ratings. A report reader can therefore ask what counted: which records, behaviors, or ratings; who supplied or scored them; and under what conditions. When the criterion definition or collection process is missing, keep the interpretation narrow and seek the technical account before treating the association as evidence about a broader personal tendency.

Sources: Principles for the Validation and Use of Personnel Selection Procedures; In the Army Now: The United States Army and the All-Volunteer Force; Psychological Testing

What does contamination have to do with reliability and deficiency?

Reliability, criterion deficiency, and criterion contamination answer different questions about an outcome measure. Reliability asks whether the measure produces sufficiently consistent results under specified conditions. Deficiency asks whether the measure leaves out important parts of the outcome it is meant to represent. Contamination asks whether it includes systematic influences that do not belong to that intended outcome. A measure can be consistent yet incomplete, or affected by both problems. Evidence about one property does not settle the others. | Question | What to inspect | Possible consequence | |---|---|---| | Reliability | Would the measure give reasonably consistent results under the stated conditions? | Inconsistency makes the observed outcome less dependable. | | Criterion deficiency | Which relevant parts of the intended outcome are absent? | The measure may represent only a narrow portion of the outcome. | | Criterion contamination | Which systematic, irrelevant influences enter the recorded outcome? | The measure may reflect the intended outcome and other influences together. | The *Principles for the Validation and Use of Personnel Selection Procedures* distinguishes contamination, which adds extraneous systematic variance, from deficiency, which excludes relevant variance. The *National Research Council* chapter on personnel selection likewise treats criteria as imperfect approximations of a broader outcome: a measure can omit important elements or incorporate factors unrelated to the target. These definitions concern the outcome used to evaluate a score interpretation. They show why readers need to know what it captures. The target should be stated before judging whether a measure is complete or contaminated; the same recorded behavior may be relevant to one claim and peripheral to another. Consider an explicitly illustrative case. A supervisor’s overall performance rating consistently counts how quickly tasks are completed, but does not include safety or quality. If the intended outcome is broad work performance, the rating is deficient because important components are missing. Suppose the supervisor’s judgments are also systematically influenced by visible behaviors associated with a personality profile, rather than only by the defined work outcome. That would raise a contamination concern. Stability would not restore the omissions or remove the influence. A broader measure could still be unreliable if ratings fluctuate under similar conditions. A reliability coefficient, when a report provides one, addresses consistency for a particular measure and set of conditions. It cannot by itself establish that the outcome covers the intended domain or is insulated from irrelevant influences. The *CASRAI Criterion Validity Guide* separates reliability from criterion relevance and freedom from extraneous variance, while cautioning that real-world criteria are imperfect. It is methods guidance, not a professional standard; the definitions above rest chiefly on the SIOP principles and National Research Council discussion. For a reader, the practical check is to ask three questions separately: Is the outcome recorded consistently? What relevant parts of the intended outcome are included or missing? What systematic influences could shape what gets recorded? A report that answers only the first question has not answered the other two. A possible contaminating influence does not prove that it changed a particular result; it is a reason to inspect the criterion and narrow the inference if details are unavailable.

Sources: Principles for the Validation and Use of Personnel Selection Procedures; In the Army Now: The United States Army and the All-Volunteer Force; Criterion Validity: Concurrent vs Predictive Evidence Explained

How is contamination different from a shared method or ordinary error?

Criterion contamination concerns systematic influences on the outcome used to evaluate a personality score. Shared-method variance concerns a feature the predictor and outcome have in common, such as the same respondent, response format, or occasion. Ordinary measurement error concerns inconsistency: the recorded result varies because measurement is imperfect. These ideas can overlap, but they answer different questions. A common method is not automatically contamination, and inconsistency is not evidence that an outside influence has entered the outcome. A study can produce an association for more than one reason. Imagine people report both a personality tendency and how often they display a work behavior, using similarly worded questionnaires at the same sitting. Response style, current mood, or a wish to present oneself favorably could affect both sets of answers. Their shared source or procedure may contribute to the association, even if no external outcome, such as a supervisor rating, is involved. That is a shared-method concern: the measurement route may help explain why scores move together. It does not establish that the association is entirely an artifact. The design and evidence would have to show how much the method contributed. The National Research Council’s chapter on criterion measures discusses dependencies that can arise when predictor and criterion information share paper-and-pencil methods. It supports treating shared procedure as a possible source of ambiguity, not assuming every same-method association is false. The *Psychological Testing* methods chapter also discusses measurement sources and how test results or expectations may affect later performance. Its examples offer foundational distinctions; they do not test a particular contemporary personality report. Readers need the study’s actual design before deciding whether shared method is a material limitation. Now change the example. Suppose the personality score comes from a questionnaire, while the outcome is an independently recorded work result. The sources differ, reducing direct same-respondent overlap. Yet the result might still reflect unequal access to equipment, different assignments, or a supervisor’s expectations informed by the personality score. Those are possible criterion-side influences despite separate methods. Conversely, an outcome can be clearly defined and collected without an obvious systematic influence, yet fluctuate from one observation to another. That is a reliability or error question, not by itself contamination. The *Criterion Validity* guide distinguishes reliability from relevance and freedom from extraneous influence; the professional principles and National Research Council chapter anchor these as separate checks. A report should make the measurement path visible. Ask who supplied the predictor and outcome, who scored them, whether the same person or procedure contributed to both, and whether raters knew the personality result. Then ask what conditions could systematically shape the outcome and whether it is consistent enough to interpret. These are separate questions: source overlap, outside influence, and consistency. A second source or independent record may help test whether a finding depends on one method, but multiple sources do not automatically make an outcome relevant or complete. Keep the conclusion narrow: shared method can cloud interpretation, ordinary error can reduce consistency, and criterion contamination can introduce systematic influence. Evidence about one does not settle the others.

Sources: Psychological Testing; In the Army Now: The United States Army and the All-Volunteer Force; Criterion Validity: Concurrent vs Predictive Evidence Explained

How can contamination change the relationship?

Contamination matters most when the outside influence is connected to the personality score or otherwise changes how the outcome is produced. It can make a relationship look stronger, weaker, or different in meaning; the label alone does not tell us which direction to expect. If the outcome also reflects a systematic influence, the association may partly describe that influence rather than the interpretation a report invites. The National Research Council’s *In the Army Now: The United States Army and the All-Volunteer Force* explains that contamination is especially consequential when it is correlated with a predictor, and that it can raise one predictor’s apparent validity while lowering another’s. That is a measurement warning, not a finding that any particular personality result has been distorted. Consider an explicitly illustrative case, not a study or a claim about a real assessment. A researcher wants to know whether a hypothetical personality score is associated with careful checking of completed work. One criterion is a structured observation of checking behavior, collected by observers who do not know the score and under comparable opportunities to complete the same tasks. Another is a broad supervisor rating collected after supervisors have seen the personality profiles. The second rating may reflect observed checking, but it may also reflect expectations formed from the profile. If those expectations tend to favor people with certain scores, part of the measured relationship could follow the expectation rather than the behavior. The Principles for the Validation and Use of Personnel Selection Procedures identifies predictor knowledge and rater attitudes among possible sources of extraneous systematic variance; it also gives differences in equipment, shifts, locations, and tenure as examples of conditions that can affect criteria. These examples come from personnel-selection guidance. They clarify a general measurement mechanism, but do not establish that every supervisor rating or personality report has this problem. Now change one condition in the illustration. Suppose people with similar scores receive different work queues or tools, and the outcome is the number of completed checks. A person assigned more complex tasks may have less time to check each one, while another person’s queue allows repeated review. The count then reflects both checking behavior and opportunity. If opportunity differences happen to align with score differences, the association can appear stronger or take on a meaning broader than the intended behavior. If they run against the score–behavior pattern, they can obscure an association that would otherwise be visible. If the conditions vary without a systematic connection to scores or behavior, they may add noise without consistently pushing the relationship in one direction. These are conditional possibilities, not results from an invented dataset. A plausible source of contamination is not proof that it changed a reported coefficient; that requires evidence about the actual study’s measurement and conditions. Ask whether an alternative route could connect the score to what was recorded: Did the rater know the profile? Did people have comparable chances to display the behavior? Did the outcome include conditions beyond the construct named in the report? If the study addresses these questions through consistent procedures, independent observations, or careful description, the interpretation may be better bounded. Such safeguards cannot guarantee a perfect outcome measure. Without that information, treat the result as evidence about the observed score–criterion relationship under the reported conditions. It is not, on its own, a direct measure of an immutable personal quality or a universal forecast.

Sources: In the Army Now: The United States Army and the All-Volunteer Force; Principles for the Validation and Use of Personnel Selection Procedures

What does a personality-specific example show—and not show?

A 2014 study comparing two versions of a five-factor personality inventory offers a useful, bounded example of how evaluative wording can affect interpretation. The original version was compared with an evaluatively neutralized version, whose items were rephrased to reduce popularity. In the publisher abstract for [Criterion Validity is Maintained When Items Are Evaluatively Neutralized: Evidence from a Full-Scale Five-Factor Model Inventory](https://journals.sagepub.com/doi/abs/10.1002/per.1960), the authors report similar criterion validity across three sets of self-rated behaviors. They also report stronger relations between the original version and the rated social desirability of criteria, and less correlation between factors for the neutralized version. The authors interpret the results as indicating that neutralization can reduce social desirability in self-ratings.

That pattern is relevant because personality items can carry an evaluative tone: an answer may sound more popular or approved, apart from how well it describes the respondent. Rephrasing items to reduce that popularity tests whether the inventory’s relations with behavior ratings persist under a change in wording. Here, the abstract reports that they were similar across the three behavior sets. It also reports a difference in relation to criteria’s rated desirability. Those findings make a narrow point: wording and evaluation can matter to how an inventory’s evidence is understood, while a wording change need not erase its reported relation to the outcomes examined.

The distinction from criterion contamination matters. The study changed the predictor inventory’s item wording and examined self-rated behaviors and ratings of criterion social desirability. The abstract does not describe a direct test in which an outcome rater knew a personality score, or in which unequal opportunities systematically shaped an external performance rating. It is therefore an adjacent example about measurement method and evaluative content, not a demonstration of every route by which a criterion can be contaminated. Nor does the abstract establish that the self-rated behaviors were contaminated. A concern worth checking is not proof that the concern occurred.

The evidence available on the publisher page is an abstract; the full article is restricted. The abstract does not give the sample, the exact inventory, the three behavior sets, numerical effect estimates, or enough procedural detail to judge how the comparison was conducted. Those omissions limit how far a reader can take the result. The study concerns one inventory comparison and the outcomes described in its abstract. Similar criterion validity in those comparisons does not show that all self-report criteria are sound, that all personality inventories behave similarly, or that neutralizing wording will work across instruments and settings. The abstract’s conclusion about a viable technique is the authors’ interpretation of this particular result, not a universal remedy.

For a report that relies on self-rated behaviors, the practical question is whether the predictor and criterion share an evaluative tone, response source, or procedure that could shape their observed relation. Ask how each measure was worded, who supplied each rating, and whether the interpretation depends on the behavior itself or on how desirable it sounds. If the report provides no such detail, keep the inference narrow and seek the underlying validation account when the decision matters. Do not discard a result merely because self-report was used; examine the design and the specific claim. This study makes wording a sensible part of that examination, while leaving criterion contamination to be assessed on its own evidence.

Sources: European Journal of Personality, Volume 28, Issue 6; Criterion Validity is Maintained When Items Are Evaluatively Neutralized: Evidence from a Full-Scale Five-Factor Model Inventory

What should you inspect before trusting a criterion claim?

A useful criterion audit moves from the claim to the observation, then checks the conditions that connect them. Start by translating the report's sentence into a precise proposition. “The score is related to effective collaboration” is not yet specific enough to assess. Ask what counted as collaboration: peer ratings, observed exchanges, project records, or a self-report? The Society for Industrial and Organizational Psychology's *Principles for the Validation and Use of Personnel Selection Procedures* says a criterion should have a clear rationale linking it to the proposed use. Although its guidance concerns personnel selection, the basic question travels well: does the selected outcome represent the claim the report makes, or merely an available proxy? Next, identify who produced the criterion, when, and with what information. A rating collected after a manager has seen the personality score has a different risk profile from a rating made by someone unaware of it. The SIOP *Principles* names raters' knowledge of predictor scores and raters' attitudes among possible sources of systematic influence; it recommends minimizing such effects and notes that standardizing administration can help. Ask whether the same person completed the personality measure and the outcome measure, whether the outcome was gathered at the same sitting or later, and whether the scorer knew the study's hypothesis. These details do not prove distortion. They tell you which alternative explanations remain open. Then inspect the setting and opportunity to show the outcome. A record of completed projects may reflect access to equipment, assignment difficulty, staffing, shift, or location as well as the behavior the report purports to capture. The SIOP *Principles* gives differences in machinery quality, sales territories, tenure, shifts, and job locations as examples. The practical question is not whether context exists; it is whether the study describes conditions that could systematically affect the criterion, and whether those conditions could be related to personality scores. If people had unequal chances to demonstrate the target behavior, the recorded outcome may combine behavior with opportunity. Keep four checks separate. Relevance asks whether the chosen outcome fits the interpretation. Coverage asks what important parts it leaves out. Reliability asks how consistently it is measured. Contamination asks whether systematic, irrelevant influences enter it. The CASRAI guide *Criterion Validity: Concurrent vs Predictive Evidence Explained* notes that work records can reflect factors beyond the intended trait and recommends blinding raters where feasible, using independently generated records, and separating administration from scoring. Those are safeguards, not guarantees. A record can be independent of the assessment and still be incomplete; a carefully designed rating can be relevant but inconsistent. Multiple indicators can reveal whether a conclusion depends on one source, but combining measures does not automatically resolve their limitations. Ask what the results show beyond one summary association. If a report gives only a correlation, ask whether the technical evidence also describes the criterion's distribution, its measurement consistency, missing observations, and subgroups or conditions. A single coefficient compresses the evidence into one number; it does not tell you which observations produced it or whether the relation holds across the range studied. This asks for context, not a research paper in every consumer report. The SIOP *Principles* treats the research sample and data analysis as part of validation reporting, while CASRAI emphasizes that criterion evidence is about a specified external outcome. A report can state the outcome and point to fuller documentation. Finally, match the evidence to the claim being made now. Note the participants, setting, measure, timing, and intended use in the study, then compare each with the report's current interpretation. Evidence from one group or context does not, by itself, settle what the same score means elsewhere. The SIOP *Principles* says samples should be chosen with the intended generalization in mind and treats generalization as an empirical question. If the report does not disclose the criterion definition, source, timing, or relevant conditions, narrow your confidence in that relationship and ask for the technical report or primary validation evidence. Do not reject it solely because the outcome uses ratings or self-report. The decision is whether the disclosed evidence supports this particular interpretation for this particular purpose.

Sources: Principles for the Validation and Use of Personnel Selection Procedures; In the Army Now: The United States Army and the All-Volunteer Force; Criterion Validity: Concurrent vs Predictive Evidence Explained

What conclusion is safe to draw from a personality report?

A personality report’s criterion evidence supports a bounded conclusion: this score was associated with this defined outcome, for this population, under the study’s conditions. It does not establish that the score captures a fixed quality, predicts every setting, or explains the outcome. The National Research Council chapter on criteria explains that an outcome approximates the intended domain, and systematic influences can change how a predictor appears to relate to it. The SIOP Principles and Standards note that such influences may be difficult to identify or eliminate. Their possibility warrants scrutiny; it does not prove an association false. Before relying on a relationship, ask the publisher to define the criterion in observable terms; identify who supplied or scored it and when; ask whether that person knew the personality result; and request information about consistency, omitted outcome components, conditions, and the population and use the evidence supports. These are separate questions: a consistent rating can still reflect an incomplete outcome or systematic influence. A consequential decision calls for evidence matched to that decision, not a broad label attached to a correlation. Keep the interpretation provisional when details are missing. Confidence could increase if the criterion were clearly defined, relevant to the claim, gathered under defensible conditions, and supported by further evidence aligned with the same use. Until then, retain the narrow association, ask how it was measured, and avoid turning it into a verdict about the person.

Sources: Principles for the Validation and Use of Personnel Selection Procedures; In the Army Now: The United States Army and the All-Volunteer Force