How to Tell Whether a Personality Report’s Advice Follows Its Scores

What is the recommendation actually asking you to accept?

A recommendation follows from a personality score only when the report shows how responses became that score, why it supports the interpretation, what evidence connects it to the relevant situation, and why that situation warrants the action. If a link is missing, the advice may prompt reflection, but remains a hypothesis rather than a personal verdict. Begin with the recommendation sentence. Is it describing a tendency, predicting an outcome, or prescribing a choice? Each move adds a claim needing support. A description characterizes a scale or score. A prediction says something about behavior beyond the assessment. A prescription tells the reader what to do, avoid, or choose. For example, an illustrative description might say, “Your responses produced a score interpreted as a preference for planning.” A prediction might say, “You will struggle in work that changes quickly.” A prescription might say, “Avoid roles with frequent change.” These claims do not follow automatically from the first: they add assumptions about circumstances, future behavior, and the value of a decision. When reading your own report, ask what you are being asked to accept: that the score was calculated as described; that it represents a tendency; that the tendency applies in this setting; or that acting on it would help. A clearly explained score may support a description while leaving a forecast or instruction untested. The NCME Assessment Glossary defines validity as evidence supporting a particular score interpretation for a given use. Thus “Is this test valid?” is too broad to settle a recommendation. Ask whether evidence supports this interpretation and use. “A Contemporary Approach to Validity Arguments: A Practical Guide to Kane’s Framework” describes four inferences: scoring, generalization from the test occasion, extrapolation to real-world performance, and implications for an action. Evidence should address assumptions along this chain; needs depend on the intended decision. This framework comes from medical education. Applying it here offers an audit structure, not proof that a personality report has been validated. Isolate the advice sentence, then mark where it moves from description to prediction and prescription. If support reaches only the first step, keep that part and pause at the rest. You need not accept or dismiss the whole report. The gap identifies what to ask: what evidence links this score to this situation, and why does it justify the action?

Sources: NCME Assessment Glossary; A Contemporary Approach to Validity Arguments: A Practical Guide to Kane’s Framework

Can you trace the score back to what the test measured?

Start with the number behind the recommendation. Identify the construct, scoring rule, score type, report version, and comparison group. A construct is the characteristic a test is designed to measure. The NCME Assessment Glossary describes personality inventories as measuring psychological attributes or interpersonal tendencies. This does not prove a test captures its construct well. Ask whether advice describes what items and scoring represent, or adds qualities the score did not measure. A raw score is the sum or other combination of item scores. A report may transform it before display, for example by comparing it with a reference distribution. Ask which responses contribute and how they become the displayed number or label. You need to know whether cited evidence concerns the score this version produced. Next identify the comparison. A norm-referenced interpretation compares a score with a specified reference population. A criterion-referenced interpretation compares it with a defined standard, such as a threshold. They answer different questions: “Where does this score sit relative to this group?” and “Does it meet this criterion?” Neither alone tells you what action to take. A percentile is easy to overread. The NCME Assessment Glossary defines percentile rank as the percentage of scores in a specified distribution below a given score. A 70th-percentile result means the score is above 70 percent of scores in that distribution. It does not mean the person displays a behavior 70 percent of the time or has a 70 percent chance of succeeding. Those are different claims requiring different evidence. The phrase “specified distribution” matters. The report should identify who is represented and when norm data were collected. The NCME glossary defines norms as statistics summarizing score distributions for specified groups, usually intended to represent a larger reference population. A local group can support a limited comparison, but cannot stand for everyone. If the report calls a score “high” without naming the comparison, ask: high relative to whom? A mismatched norm can change the advice. Version matters too. A manual or study supports the report only if it concerns the instrument and scoring version used, or explains why evidence transfers. Changes to items, scoring, language, or norms may change the displayed number’s meaning. The Standards for Educational and Psychological Testing frames interpretation around intended use and population; evidence about another version or group is not automatically interchangeable. Ask the provider to explain this gap. Finally, distinguish a score description from an action rule. “Above the reference-group average” describes relative position; “avoid roles involving rapid change” is advice. The first does not establish the second. The NCME glossary defines a cut score as a point where interpretation or action changes. If advice rests on a band, ask how it was set and whether evidence supports using it for this purpose. For a quick audit, copy the recommendation, underline the score phrase it relies on, then write the score type and comparison group beside it. If documentation does not let you do that, you do not yet know what the advice means. Ask the provider for the scoring explanation, reference group, and manual for this version. Once the number is traceable, assess whether evidence supports carrying its interpretation into the situation named by the recommendation.

Sources: NCME Assessment Glossary; Standards for Educational and Psychological Testing

Does the score have enough precision for this recommendation?

A score can be consistent and still be too uncertain to support a recommendation that turns on a fine distinction. Reliability asks how consistently scores behave under specified conditions; it does not establish that the report measured the right tendency or that its advice is appropriate. The relevant question is whether, for this scale, population, and recommendation, measurement uncertainty could change the interpretation that triggers the advice. The ETS guide Test Reliability—Basic Concepts defines reliability through consistency across occasions, test editions, or raters. Internal consistency concerns how responses within a test relate to one another; it does not show whether a person would answer similarly months later. Retest stability addresses consistency across occasions, while interrater reliability concerns whether different raters score the same response similarly. A self-report questionnaire may not involve a human rater, making interrater evidence beside the point. A retest estimate needs its time interval, sample, and conditions. “What is the reliability?” leaves out which variation the evidence addresses. A standard error of measurement estimates uncertainty around a reported score under a specified measurement model. It describes how much a score might vary because measurement is imperfect; it does not reveal a person’s exact underlying personality. The ETS guide explains that this quantity is connected to reliability and score variability, and that a reliability coefficient alone is not enough to interpret precision. Neither is a universal property detached from scale and population. A coefficient reported for another version, language, or group may not show whether this reader’s score is sufficiently precise. The 2014 Standards for Educational and Psychological Testing frame score interpretation around intended use and population; they set no magic reliability threshold for every recommendation. Precision matters especially when advice depends on a cutoff or narrow band. Consider an explicitly illustrative case: a report labels a score “high” above a boundary and recommends avoiding work requiring frequent planning when the score falls below it. If plausible measurement variation could place the same person on either side, the label and advice may change although the underlying tendency has not meaningfully changed. No numeric interval can be supplied here because the scale documentation and applicable error estimate are unknown. A categorical recommendation inherits uncertainty from the score used to assign the category; a small boundary crossing should not be mistaken for a sharp difference in real-world behavior. Ask the provider which reliability estimate applies to the exact scale and version, how it was obtained, and for which population and conditions. Then ask whether the report gives a standard error or other uncertainty information and how it affects the band or threshold behind the recommendation. If given only a coefficient, ask what kind of consistency it describes and whether advice depends on a boundary. A score comfortably away from a cutoff may reduce one concern, but does not establish that the interpretation transfers to the situation or that the action will help. Precision is one link in the argument, not the whole argument. A stable score is not automatically accurate, useful, or action-guiding. If a borderline label drives a major choice, treat the advice as uncertain until its precision and downstream evidence are clear. A reversible reflection prompt can be tested against experience while remaining provisional. The question is whether the score’s precision is adequate for the particular claim.

Sources: Test Reliability—Basic Concepts; Test Reliability: Basic Concepts; Standards for Educational and Psychological Testing

Which evidence supports each step from score to advice?

Look for evidence at each step: that responses are scored as described, that the score represents a tendency beyond this one test-taking occasion, that the tendency applies in the situation named, and that the proposed action is justified for the stated use. If one step has no support, the recommendation may still prompt reflection, but the report has not shown that its advice follows from the score. A useful way to trace the argument comes from “A contemporary approach to validity arguments: a practical guide to Kane's framework.” The framework was developed for medical education; here it organizes questions, not evidence that a personality measure is valid. Kane’s four inferences are scoring, generalization, extrapolation, and implications. They ask: How did responses become this score? How well does the score represent performance across occasions or forms of the test? Does that interpretation carry into relevant behavior outside the test? And what decision or action is justified by it? Each question calls for a different kind of support. For scoring, documentation should describe the rule that turns item responses into the reported number or scale. For generalization, evidence might examine whether scores are sufficiently consistent across occasions, forms, or raters. That consistency does not prove that the interpretation is right; it addresses whether the score is stable enough for the claim being made. For extrapolation, the report needs evidence that the measured tendency relates to the real-world behavior it names, in the relevant setting. Finally, the implications inference asks whether that evidence supports the proposed use or action. A study connecting a score with a related construct may inform interpretation, but it is not automatically evidence that following a particular recommendation improves an outcome. This distinction helps separate plausible language from a supported chain. A report might say that a scale reflects a preference for planning. That could be a description of what its items were designed to capture. If it then says the reader should avoid work involving uncertainty, the report has moved from describing a scale to making a claim about behavior and prescribing a choice. Face-valid wording can make that move sound natural: the items and narrative appear to fit. But a good-sounding explanation does not, by itself, establish that the score predicts behavior in the relevant context. Likewise, a correlation with another measure of a related tendency supports a relationship between measures; it does not settle whether the broader real-world claim or instruction is warranted. The “Standards for Educational and Psychological Testing” and the National Council on Measurement in Education’s Assessment Glossary reinforce why intended use matters. The glossary describes validity evidence as supporting a particular score interpretation for a given use; the Standards frame validation around interpretations and uses for intended populations. In practice, evidence for a low-stakes prompt to notice planning habits could be less demanding than evidence for advice that would narrow a person’s work options. That stakes-sensitive distinction is an application of the frameworks, not a universal formal threshold. When a link is unclear, ask the provider a narrow question: “Which evidence supports this interpretation for people like me and this proposed use?” If the answer is a scoring explanation, it may clarify the number without supporting a real-world recommendation. Naming the missing inference turns vague skepticism into a testable request, while keeping the advice at hypothesis level until supported.

Sources: A Contemporary Approach to Validity Arguments: A Practical Guide to Kane’s Framework; Standards for Educational and Psychological Testing; NCME Assessment Glossary

Who and what does the evidence actually represent?

Evidence is relevant only when it resembles the claim a report makes. To judge whether advice follows from a score, compare the evidence’s instrument, population, setting, outcome, and intended use with the recommendation in front of you. A study about one sample or result may support a cautious interpretation while leaving a different person or decision untested. Begin with the outcome. A study may show that a scale relates to another questionnaire measuring a similar tendency. That can support a claim about how the scale behaves alongside that measure; it does not, by itself, establish a claim about what someone will do at work, how they will respond in a particular team, or which action will help them. The *Standards for Educational and Psychological Testing* treats validity as support for specified interpretations and uses, and calls for attention to the samples and settings used in validation. If its study measured self-described preferences but its recommendation predicts later behavior, ask what evidence bridges those two outcomes. Next, check who supplied the data and under what conditions. Evidence from volunteers, students, or people already working in a role may not answer the same question as evidence from the population named in a report’s recommendation. The report should explain why findings from that group carry over. A relationship found among people who have remained in a job, for example, may not tell a reader choosing whether to enter that occupation what will happen to them. A quiet reflection exercise and a pressured formal assessment are different uses, even if their questions look alike. Language and access also affect what a score can mean. A translated form, an unfamiliar phrase, inaccessible presentation, or a response format that creates an avoidable barrier may change how a person can answer. Those influences can introduce variation unrelated to the tendency the test intends to measure. The 2014 *Standards for Educational and Psychological Testing* says test design, administration, and scoring should minimize construct-irrelevant barriers and support valid interpretations for intended uses and examinees. This is a professional standard, not proof that any particular report meets it. When a recommendation is applied to you, relevant documentation should make clear which language, administration conditions, and examinee groups were studied, and whether the interpretation is supported for the population in question. This check concerns the recommendation, not whether a study represents everyone perfectly. A report may have evidence that supports a limited pattern, while leaving uncertainty about a different language group, setting, or outcome. The *Contemporary Approach to Validity Arguments: A Practical Guide to Kane’s Framework* likewise treats validity as a chain of inferences whose assumptions need evidence; the evidence needed depends on the use. Transfer is an assumption to inspect, not an automatic verdict on the whole report. A useful provider question is: “Which study supports this recommendation for people like the population it addresses, in a setting like mine, and on the outcome it names?” Compare the answer with the recommendation’s wording. If the report moves from a related measure to a real-world prediction, from a different population to you, or from one administration context to another without explaining the bridge, that is an open assumption. The score may still prompt a question worth exploring. The evidence has not yet shown that the recommendation applies to your circumstances as stated.

Sources: Standards for Educational and Psychological Testing; NCME Assessment Glossary; A Contemporary Approach to Validity Arguments: A Practical Guide to Kane’s Framework

What can group-level work evidence tell one reader?

A group association can show that a tendency is relevant enough to investigate; it does not establish that a particular person will perform a particular job well or should choose it. The distinction matters when a report turns a score into career advice.

Consider conscientiousness, often described in reports through order, persistence, or follow-through. In “The criterion-related validity of conscientiousness in personnel selection: A meta-analytic reality check,” Watrin and colleagues combined field studies relating self-reported conscientiousness to overall job performance rated by supervisors. Their uncorrected mean correlation was .17 across 102 correlations and 23,305 participants. A correlation summarizes how two measures varied together across the studied observations; it is not the percentage of a person’s success caused by a trait, nor a probability that one reader will succeed. The positive average is still informative: it suggests the measured tendency has some relationship to performance in the assembled studies, so the evidence is not equivalent to no relationship at all.

But that average does not settle what a report may say to an individual. The review found substantial variation among study results. Its evidence also came mainly from incumbent employees and concurrent designs, where scores and performance were measured in existing workers, rather than from realistic prediction with applicants before a job outcome. Only about 12% of the reviewed studies used actual applicants in predictive designs. The authors called for more research under realistic selection conditions.

The U.S. Office of Personnel Management’s “Designing an Assessment Strategy” makes the intended-use issue concrete: if a personality test is meant to forecast later performance in a particular position, predictive evidence should connect its scores with subsequent performance in that job. It does illustrate why the label “personality predicts performance” is too broad to carry every recommendation. Which scale? Which role? What outcome? What population and conditions? A result for overall performance across varied jobs does not automatically answer whether one person should enter, leave, or avoid a particular field.

An individual role decision also involves information that a conscientiousness score alone does not measure: interests, abilities, learned skills, values, working conditions, available opportunities, and constraints. For example, an unnamed reader might use a report’s description of planning as a prompt to compare how they handle deadlines in different settings. That is an illustration, not a study result. The observation could help identify a useful question about role structure; it would not show that the reader must choose a highly structured occupation.

The fair conclusion is neither that group research dictates an individual’s path nor that group research has nothing to offer. It can make a proposed connection plausible enough to examine. The report still has to show why its own score represents the tendency, why the tendency matters in the reader’s actual context, and how that supports the suggested action. Keep the useful question raised by the association; withhold the personal career verdict unless evidence and reasoning cross those additional steps.

Sources: The criterion-related validity of conscientiousness in personnel selection: A meta-analytic reality check; Designing an Assessment Strategy

Does the proposed action match the strength of the evidence?

The more an action narrows options or imposes consequences, the more directly relevant the evidence and alternative explanations need to be. A tentative coaching prompt can be useful with a weaker claim than a hiring, promotion, or career verdict can support. This is a practical decision rule inferred from intended-use validation, not a formal cutoff stated by a standards body. “A contemporary approach to validity arguments: a practical guide to Kane’s framework” describes the final step in an assessment argument as implications: applying scores to inform a decision or action. Its authors say evidence needs depend on the intended use and associated decision. The framework concerns assessment in medical education, so applying it here is a reasoned extension, not personality-test-specific validation. It suggests a useful question: what evidence supports this action, given what it may change? Compare “you may want to ask for clearer deadlines” with “you should avoid project leadership.” The first is a reversible prompt to examine a work pattern. The second could close off opportunities and treats a score as a career verdict. These are illustrative statements, not findings about anyone. The latter needs a stronger bridge from the measured tendency to the role’s demands and outcomes, plus a reason the action is likely to help. An association with an outcome across a group does not establish that a score caused the outcome, predict one individual’s future with certainty, or show that acting on advice improves a decision. Prediction asks whether evidence helps anticipate an outcome in a defined setting. Utility asks whether using the evidence makes a decision better after weighing costs, benefits, and alternatives. “Designing an Assessment Strategy,” from the U.S. Office of Personnel Management, says predictive evidence may be needed when a personality test is intended to forecast later performance in a customer-service role. That example does not validate another test, job, or personal career recommendation. Before accepting a work instruction, consider whether the difficulty named by the report could also involve limited training, excessive workload, unclear expectations, role design, or temporary disruption. These are possibilities to examine, not explanations to assign to the reader. If the report treats a tendency as the only cause, it may have skipped alternatives relevant to the action. The Standards for Educational and Psychological Testing says assessment processes should minimize irrelevant barriers and support valid interpretations for intended uses and examinees. Ask whether the interpretation fits this person and context, or depends on conditions the report has not considered. Some instruments may have credible, directly relevant evidence for a bounded recommendation in the population and setting for which they were designed. If the score is precise enough, studies support its interpretation in a comparable context, and the report explains why the action follows, a reader need not dismiss the advice simply because it is personal. The conclusion remains bounded by that evidence. The verdict is conditional: act in proportion to evidence for the consequences. Ask what independent evidence supports the interpretation, whether uncertainty could change it, whether outcome data match the situation, and how the action follows. If links are missing, downgrade the advice, not necessarily the whole report. Keep it as a question to test against experience, rather than a diagnosis, hiring score, or destiny claim.

Sources: A Contemporary Approach to Validity Arguments: A Practical Guide to Kane’s Framework; Designing an Assessment Strategy; Standards for Educational and Psychological Testing

What should you do with an unsupported recommendation?

When a recommendation outruns its evidence, mark the first unsupported step and pause. Rewrite it in plain language, then ask the provider: Which score supports this statement? What study supports this interpretation for this population and use? What uncertainty or context could change it? Look for evidence relevant to the claim, not only a reliability coefficient or general scale description. “A contemporary approach to validity arguments: a practical guide to Kane's framework” treats the move from score to action as an inference whose assumptions need evidence; evidence depends on the decision.

Then compare the proposed action with that evidence. If the link remains unclear, keep the advice as a limited reflection question and test it against specific experience before a consequential choice. “I may need more time to plan” could prompt a reversible conversation about a current task; it does not establish that someone should avoid a role. unsupported advice is a hypothesis to examine, not a personal instruction. For a way to organize observations about planning, ambiguity, feedback, or collaboration, the live Work Pattern Report at /assessment covers ten work continuums. It has no norms, cutoff, type, or selection score, so it cannot validate another report or match someone to a career. Use it to make a work-pattern question more specific before deciding what evidence or conversation would help.

Sources: Work Pattern Report; Personality Report topics

Questions readers ask