What Validity Evidence Must Support in a Personality Report

What does the Standards’ central rule require a report to show?

The 2014 Standards for Educational and Psychological Testing require evidence and theory to support each intended interpretation of test scores for a specified use. For a personality report, readers should be told what a score represents and what conclusion or action it is meant to support. Calling a test “validated” in general does not answer either question. Ask instead: what does this score mean for this purpose, and what evidence supports that reading? The Standards were developed jointly by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. Their validity framework concerns score interpretation and use, rather than treating validity as a permanent quality a test possesses in the abstract. In plain terms, validity is the degree to which evidence and theory support a particular interpretation of scores for a stated use. Evidence supporting a modest claim may leave a broader conclusion unestablished. Report language can move from description to recommendation. A score might summarize answers about a work tendency; a reader may then infer how someone will act in a role or what decision should follow. Those are different claims. The Standards require support for the intended interpretation, including the meaning assigned to the score and the proposed use. Their guidance does not establish that any particular personality report meets that requirement. The official American Psychological Association page “Standards for Educational and Psychological Testing” identifies the 2014 edition and says a joint committee has been appointed to lead a revision. That announcement does not establish that a replacement edition has been published. This article addresses the 2014 text, not validation of a named personality instrument. Start by translating a report sentence into a claim about a score, its meaning, the people and conditions involved, and the purpose at hand. Then ask whether the evidence supports that chain.

Sources: STANDARDS for Educational and Psychological Testing; Standards for Educational and Psychological Testing

Why is ‘this test is valid’ too broad?

Because validity applies to an interpretation of scores for a specified use, not as a blanket property attached to a test. Reading a score as a self-reported tendency is narrower than predicting performance or making a selection decision. A report sentence can sound like one conclusion while joining several claims: that answers were recorded as intended, that the score represents a construct, and that this meaning supports a further conclusion or action. The Standards for Educational and Psychological Testing require evidence and theory to support the intended interpretation and use. The reader’s task is to see where those links are stated and where an added inference begins. Consider this clearly illustrative sentence: ‘This response pattern suggests a preference for planning.’ It interprets answers as a score and describes the score as a tendency. A reader may ask what “preference” means, whether the items represent it, and whether the description has appropriate limits. But the sentence does not claim that the person plans consistently in every setting, completes projects on time, or succeeds in a particular occupation. Those are different propositions. The example is a reasoning aid, not a finding about an actual personality measure or its respondents. Now extend the illustration: ‘Therefore, this person will perform well as a project manager.’ That conclusion treats a self-reported tendency as evidence about behavior beyond the assessment, connects that behavior to a work outcome, and assumes the score can support a prediction in a particular role and setting. It may invite an action, such as recommending a role or comparing applicants. Each step calls for relevant support; evidence describing a response pattern does not automatically show that it predicts performance or warrants a decision. The larger the claim, the more the reader needs to know about how the evidence bears on it. Kane’s argument-based account of validation helps make this distinction concrete. In the publisher abstract for “Validating the Interpretations and Uses of Test Scores,” Kane distinguishes proposed interpretations from proposed uses and says more ambitious claims require more support. The abstract also states that validating an interpretation does not by itself validate a use based on it. This is a conceptual framework, not a study of a personality report; it clarifies how to reason about a claim without supplying evidence for any particular instrument. The National Council on Measurement in Education glossary makes a related distinction: reliability or precision concerns score consistency and random measurement error, while validity concerns evidence and theory supporting a specific score interpretation for a given use. A consistent score can still support an unwarranted inference. So the practical question is not simply whether a report says “valid,” but what its evidence supports the reader in saying or doing. When the report moves from a stated tendency to a prediction, recommendation, or selection decision, pause at that transition and ask what evidence supports the added step.

Sources: STANDARDS for Educational and Psychological Testing; Validating the Interpretations and Uses of Test Scores; Glossary

Which evidence sources fit the claim—and why are they not a checklist?

The Standards describe five possible sources of validity evidence: test content, response processes, internal structure, relationships with other variables, and consequences. For a personality report, these are ways to investigate different parts of a proposed interpretation. The question is which proposition needs support. Evidence based on test content concerns the relationship between what items ask and the tendency the report says they measure. If a scale is described as a preference for planning, a reader might ask whether its items represent that preference or mix in other demands, such as comfort with detailed instructions. This is illustrative, not a finding about a named scale. Content evidence can help show what an instrument samples; by itself, it cannot establish that respondents interpret items as intended or that a score predicts behavior outside the questionnaire. Evidence based on response processes asks whether people answer in a way that fits the report’s interpretation. A respondent might read an item about “taking charge” as asking about formal authority, while the report interprets agreement as comfort initiating action among peers. That mismatch may undermine the report’s interpretation. Studying response processes could help investigate that gap, but would not alone show that the tendency predicts a work outcome. Evidence based on internal structure examines how items and scales relate to one another. Analyses might ask whether items intended to reflect separate tendencies behave as distinguishable dimensions, or whether a proposed structure fits observed responses. That can support a claim about how scores are organized. It does not alone establish that labels capture real-world constructs or forecast a result. Evidence based on relationships with other variables asks whether scores relate to outside measures as the interpretation predicts. A proposed planning tendency might be expected to relate to a measure of planning behavior. The comparison matters: an association with a related self-report is not the same proposition as prediction of observed work performance. Support is needed for each proposition; one relationship cannot carry a broader claim by itself. Evidence concerning consequences considers effects relevant to the interpretation or use. Questions might include whether using a score for a stated purpose produces effects that challenge the report’s assumptions or responsible use. The Standards include consequences among possible sources, while scholarship has debated how consequences fit within validity itself. Consequences may matter to evaluating a use, but do not settle every question about fairness or acceptability. The Standards say these sources are not distinct “types” of validity and each is not required in every setting. Support is needed for propositions underlying a specified interpretation; complex claims may need multiple links supported. Professional judgment guides which sources fit. “Multiple” does not mean one study from every category. A few strong pieces addressing the relevant proposition may be more informative than many weak or unrelated results. The Standards explicitly caution that acceptability cannot be determined by a checklist. Ask instead: what does each piece support, and where does the report continue beyond it? “Validity: on the meaningful interpretation of assessment data” likewise describes the five sources as fruitful areas for evidence rather than a fixed recipe. Its medical education examples orient the framework, not findings about personality measures. Ask which evidence bears on the report’s sentence, what alternative remains possible, and which further claim needs separate support. This traces the report’s reasoning.

Sources: STANDARDS for Educational and Psychological Testing; Validity: on the meaningful interpretation of assessment data

How do the evidence sources test a report’s separate claims?

Each evidence source tests a different part of the reasoning, so one strong result cannot carry every conclusion. For example, coherent scale structure may support treating dimensions as distinguishable, but it does not alone show that a score predicts a work outcome. The Standards for Educational and Psychological Testing organize evidence around the propositions behind a proposed score interpretation. The relevant evidence depends on what the score means and what conclusion follows.

Evidence based on test content asks whether the items adequately represent the tendency the report says it measures. A mismatch could leave a score accurately calculated but poorly matched to its stated label. Evidence about response processes asks whether respondents understand and answer items in ways consistent with the interpretation. For a self-report, this can matter if the interpretation assumes a particular reading of an item or a particular time frame. These questions concern how answers are produced; they do not show that the scale predicts an external outcome.

Evidence based on internal structure examines whether relations among items and scales fit the proposed meaning. The Standards note that when a report interprets separate scores or a profile, evidence should support those specific scores and their interpretation. A pattern showing that two scales are distinguishable could support treating them as separate dimensions; it would not, by itself, establish that either predicts performance. Evidence based on relationships with other variables asks whether scores relate to other measures or outcomes in ways the interpretation predicts. Similarity to a relevant measure may support a claim about shared meaning, while a claim about a work outcome calls for evidence bearing on that outcome. These are different questions, even when both studies report correlations.

Evidence about consequences considers effects relevant to the proposed interpretation or use. Its relevance depends on the claim and proposed use. The Standards and “Validity: on the Meaningful Interpretation of Assessment Data” list these five sources as fruitful ways to investigate validity evidence, not five universal boxes every report must tick. A source matters when it tests a proposition; no label substitutes for examining method and result. In practical terms, a report that cites evidence about item wording has addressed a different concern from one that cites an association with an outside measure. The reader should match the source to the sentence, not treat any research reference as support for every paragraph in the report. This comparison is a reading aid, not a demand that every brief self-reflection report publish every kind of study.

Reliability/precision means the degree to which scores are consistent across repeated applications and free of random measurement error for a group, as defined in the NCME Glossary. It matters because an unstable score can weaken an interpretation, but consistency does not establish that the score means what the report claims. A high reliability coefficient is therefore not proof of validity. Separate “Can this score be measured consistently?” from “Does this evidence support this interpretation for this use?”

Sources: STANDARDS for Educational and Psychological Testing; Glossary; Validity: on the meaningful interpretation of assessment data

Does evidence travel to a different population or setting?

Not automatically. Evidence from one group or setting may inform another, but a reader needs a defensible reason to carry the score’s meaning and relevant relationships across. The 2014 Standards for Educational and Psychological Testing ask developers to define the population, context, and administration conditions for intended interpretations. They also make users responsible for judging whether evidence fits the situation where they plan to use a score. Research is a starting point for comparison, not universal permission.

The transfer question is concrete: how much does the current situation differ from the one represented in the evidence, and could that difference matter to the claim? The Standards say validity-study samples should be described in practical detail, including relevant demographic and developmental characteristics. Possible influences include self-selection, attrition, language ability, disability, and exclusion rules. For workplace evidence, they also identify whether participants were applicants or current employees, plus experience and education. Such differences do not prove score meaning changes; they show what research covers and where questions remain.

The test version and administration matter too. A translation, changed instructions, or revised scoring can affect how responses are produced or interpreted. The Standards say that if an adaptation changes the construct underlying a test, the modified version needs evidence for its intended interpretation. The issue is whether the same meaning remains warranted. Without information about version, language, and administration, readers cannot readily compare a report with its supporting research.

The Standards do not demand a new local study every time a tool moves to another setting. Their discussion of validity generalization says a strong, relevant, consistent meta-analytic database can sometimes be more informative than a small local study. When evidence is limited or inconsistent, or the new situation differs in construct, role, criterion, or test-taker group, situation-specific evidence may matter more. One carefully conducted local study with an adequate sample can sometimes support or challenge a use, though a single imprecise study may not settle it. Similarity warrants examination; it does not replace it.

The American Psychological Association’s Guidelines for Psychological Assessment and Evaluation similarly say developer evidence commonly concerns stated purposes and cohorts like original standardization samples. A different purpose or population calls for checking evidence for that use. A reader might ask whether research on a reported work tendency involved people and conditions relevant to the report’s claim. Applying that claim to another role or group adds a transfer question. This is an application of the guidance, not evidence that any particular group differs.

Responsibility is shared, but not interchangeable. The Standards expect developers to support recommendations and put final responsibility for evaluating local relevance on test users. Under Standard 1.4, a user introducing a new interpretation or use must justify it and gather new evidence if needed. APA guidance also asks practitioners to weigh purpose, population, setting, and context when choosing a measure. A reader can ask who was studied, with which version and conditions, and for what interpretation or outcome. Compare those details with the present case. If differences remain unexplained, keep the conclusion narrower: evidence may support an interpretation in the studied context, while its reach here remains uncertain. The amount of new evidence depends on its quality, relevance, and the size of the inference.

Sources: STANDARDS for Educational and Psychological Testing; Guidelines for Psychological Assessment and Evaluation

What changes when a report moves from reflection to prediction?

A change in intended use changes the claim that needs support. A score that helps someone reflect on a self-described tendency does not, without additional evidence, establish future performance, job fit, or a hiring recommendation. The Standards for Educational and Psychological Testing frame validity around score interpretations for specified uses. Their employment chapter says each inference drawn from a score must stay within the evidence available for it. The question is whether evidence supports the particular step being taken.

Consider an explicitly illustrative tendency: a report says someone prefers to plan before acting. At first, responses are interpreted as a self-described preference. That says what answers mean under the instrument’s scoring model; it does not establish how consistently the tendency appears in different circumstances. Claiming the person usually plans across settings adds a proposition about behavior beyond the response occasion. Evidence must support generalization to relevant contexts. The Standards and the APA Guidelines for Psychological Assessment and Evaluation treat transfer to a different population, purpose, or setting as something to evaluate. This example illustrates reasoning only, not a finding about an assessment or people with a particular score.

A third step predicts a defined work outcome, such as meeting deadlines in a specified role. That is stronger than describing a preference. The Standards’ employment chapter says validation commonly begins by analyzing the job to define performance, then asks whether scores support prediction of that criterion. If empirical predictor–criterion evidence is used, Standard 11.6 names relevant conditions: a reliable criterion measure, a representative sample, and adequate sample size. Standard 11.7 says criteria should reflect important work behaviors or outputs identified from job information. A generic link between a trait and broad “success” would not show that this score predicts the specific outcome. Accumulated research can sometimes support a relationship without a new local study, but its fit to predictor, criterion, and use still matters.

A fourth step uses that prediction to choose among applicants. This adds a decision rule and justification. A relationship between a score and an outcome is not automatically a warranted rule for ranking or exclusion. “Validating the Interpretations and Uses of Test Scores” explains that more ambitious claims require more support and that supporting an interpretation does not itself validate a use. The Standards likewise caution against treating a narrow score as overall job suitability. Selection evidence must address the intended job, applicant population, and conditions; the APA guidelines emphasize evaluating purpose and context when these differ from those studied. Fairness and consequences also affect whether use is appropriate.

These stages do not establish that personality measures can never contribute to selection. They show why evidence for bounded self-reflection cannot simply carry forward to prediction and hiring. Each transition adds a proposition: from answers to score meaning, to behavior across situations, to an outcome, then to a consequential choice. A user should identify evidence for each added proposition and whether its sample and setting fit. No instrument-specific selection evidence is assessed here, so this comparison cannot establish that a particular report predicts performance or supports hiring. Ask what new claim the proposed action adds and which evidence supports it for people and conditions like these. If evidence stops at a self-described preference, keep the conclusion there.

Sources: STANDARDS for Educational and Psychological Testing; Validating the Interpretations and Uses of Test Scores; Guidelines for Psychological Assessment and Evaluation

Does a supported interpretation automatically justify the decision?

No. Evidence supporting what a personality score means does not, by itself, settle whether a decision based on it is justified. The Standards for Educational and Psychological Testing discuss interpretations, uses, and consequences, while distinguishing score meaning from broader effects that may matter to policy. The interpretation and the proposed action can have different levels of support. As an illustration, a report describes someone as tending to prefer planning. Evidence might support interpreting self-report answers as a stated preference. Using that description for reflection is one action. Predicting performance in a role adds claims about behavior and an outcome. Using that prediction to choose among applicants adds a decision rule and consequences . Each step needs justification; score meaning cannot carry later steps automatically. In “Validating the Interpretations and Uses of Test Scores,” Kane argues that more ambitious claims require more support, evaluating a proposed use includes consequences, and validating an interpretation does not validate its use. A score can have a defensible, bounded meaning while a decision remains unsupported or unacceptable. Rejecting one use does not necessarily invalidate the interpretation: concern about applicant screening does not by itself show that a tendency score cannot be discussed in voluntary reflection. The 2014 Standards include consequences in evaluation, but leave room for disagreement about their place in validity theory. Consequences can inform decisions about use, yet do not alone necessarily undermine an interpretation. The Standards call for examining their sources before judging relevance to validity. This guards against “a harmful result proves the score is invalid.” Effects may arise from interpretation, decision procedure, or conditions around testing. “Validity and Validation,” a chapter in the fifth edition of *Educational Measurement*, takes a stronger position: its authors argue consequences and test use are integral to validity and describe the Standards’ treatment as narrow or ambiguous. This scholarly counterposition is not a replacement standard or personality-specific research. Whether a score statement is supported and whether acting on it is acceptable remain questions to examine, even when a framework places both within one validity argument. The Standards’ discussion of fairness shows why a supported sentence does not settle whether applying it to a person or group is appropriate. The APA’s *Guidelines for Psychological Assessment and Evaluation* emphasize purpose, setting, user competence, and scrutiny when a measure is applied in a different setting from the publisher’s description. This is professional guidance, not evidence about a particular report. Privacy and unequal treatment deserve consideration when a report may affect a consequential decision. Keep the questions separate: what supports the description, for whom, and under what conditions? What further claim does the action assume, what consequences may follow, and who must defend that use? If evidence supports only a modest self-description, keep the conclusion there. A stronger prediction or employment decision needs relevant support and a reasoned account of consequences and fairness. The Standards provide the 2014 framework, while Kane and the newer chapter clarify the debate. Evidence for a score meaning is necessary, but it does not grant general permission for every decision attached to it.

Sources: STANDARDS for Educational and Psychological Testing; Validating the Interpretations and Uses of Test Scores; Validity and Validation; Guidelines for Psychological Assessment and Evaluation

What should you ask when the report leaves a link unsupported?

A missing explanation is a reason to ask a narrower question, not to accept the broadest conclusion or dismiss every result. Ask the provider or responsible test user to name the exact interpretation, the people and setting it covers, the proposed use, and the evidence connecting those pieces. If support reaches only a limited claim, keep your interpretation within that boundary until the extra step is explained. Start with the score: “What does this score represent, and what does it not represent?” A report may describe a response pattern, but that description does not automatically establish a stable tendency across situations or an outcome at work. If the report applies evidence to people, languages, or conditions different from those studied, ask what supports that transfer. Then ask which evidence bears on the specific interpretation: “Which findings support this sentence in the report?” Ask what alternative explanation or counterevidence was considered, too. Evidence consistent with one interpretation may leave plausible alternatives; the issue is whether it distinguishes among them well enough for the conclusion. Do not treat a list of evidence labels as proof. The Standards for Educational and Psychological Testing frame validity as an argument about interpretations and uses, not a count of studies or a certification badge. Next, check whether the proposed action adds a claim. “Would this evidence also support using the score to predict a defined work outcome or make a decision?” shifts attention from what a report says to what someone plans to do. Self-reflection and a consequential decision ask different things of evidence, even when they begin with the same score. Ask who is responsible for supporting the proposed use, especially if it differs from the developer’s stated purpose. Standard 9.4 of the Standards applies when a test is used for a purpose with little or no validity evidence. It places responsibility on the user to document the rationale for selecting the test and obtain evidence of score reliability or precision and of the validity of interpretations supporting that purpose. A user cannot simply point to a report written for another purpose. The Standard allows professional judgment, but unsupported decisions should not be presented as well documented. Its comment says hypotheses generated from such data should be labeled tentative and interested parties told about potential limitations. Reliability or precision does not replace validity. The NCME Glossary defines reliability or precision in terms of score consistency and freedom from random measurement error. This describes score consistency under specified conditions; it does not show by itself that the score means what the report says, applies to this population, or supports the action under consideration. When Standard 9.4 applies, ask for both kinds of evidence and how they relate to the intended interpretation, rather than accepting a reliability figure as a complete answer. A practical request can be brief: “Please show what this score is intended to mean, for whom and under what conditions that interpretation is supported, and what evidence supports the action proposed here.” A bounded description may support low-stakes reflection while leaving prediction or selection unanswered. The missing link should lead to a specific follow-up with the developer or user; keep the conclusion no broader than the evidence they can provide.

Sources: STANDARDS for Educational and Psychological Testing; Glossary

What is the next responsible step?

Keep a report’s conclusion at the narrowest level its evidence supports, and match the next action to the confidence and consequences involved. The Standards for Educational and Psychological Testing make this a question of scope: what interpretation is supported, for which people and conditions, and for what use? If the material supports a description of a self-reported work tendency, preserve that description. Do not turn it silently into a forecast about behavior in every setting or an employment recommendation.

Ask the report provider or the person proposing to use it: “Which evidence supports this exact interpretation for people and conditions like these, and what further claim does the action add?” Listen for an account of the score’s meaning, the population and setting addressed, and the step from interpretation to action. If the action adds a prediction or decision, ask what supports that claim. The Standards’ caution against blanket claims of compliance makes documentation and a specific answer more useful than a general assurance that a test is validated.

If your remaining question is a low-stakes reflection on work or collaboration, the live Work Pattern Report can help organize observations across ten decision-and-collaboration continuums. Treat it as a private self-reflection aid: it supplies no norm, cutoff, type, or selection score, and it is not validated for hiring or employment decisions. It cannot resolve a missing validity argument in another assessment. Its narrower role is to give you specific patterns to consider and discuss.

Sources: STANDARDS for Educational and Psychological Testing