Why One Personality Score Should Not Rank Applicants

Why should one workplace personality score not decide who ranks first?

A manager may see one personality score and assume the higher number identifies the stronger applicant. That goes beyond what the number says. A report may summarize answers about tendencies; a ranking turns that description into a prediction about who will do better at a particular job, then uses the prediction to decide. Each step needs support. The Office of Personnel Management’s “Designing an Assessment Strategy” distinguishes score consistency from evidence that an assessment relates to job performance. It says that when a personality test is intended to forecast success in a role, evidence is needed that scores relate to later performance. The guidance also explains that different job-related methods can contribute distinct information. A consistent score, then, does not by itself show that a higher result means better performance or adds useful evidence here. The Society for Industrial and Organizational Psychology’s “Principles for the Validation and Use of Personnel Selection Procedures” treats rank order as one selection strategy. That leaves room for a narrow, supported use; it does not make every report’s leaderboard defensible. Ask what the score represents, what job outcome it predicts, whether it can distinguish people precisely, and what rule connects it to the decision. Without those links, treat the number as a prompt to examine work behavior, not a measure of overall value.

Sources: Designing an Assessment Strategy; Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition

What does the number actually summarize?

A workplace report’s number can represent several different things: a raw total from item responses, a standardized score on a transformed scale, a percentile relative to a reference group, a band grouping results, or a composite formed from several dimensions. These formats answer different questions. A standardized score expresses a result on a scale whose meaning depends on the transformation and reference used. A percentile locates a result relative to a specified comparison group; it does not state the proportion of work tasks someone will perform successfully. A band groups values; a composite combines components by a rule. The Standards for Educational and Psychological Testing treats scales, norms, and interpretation as matters requiring clear documentation.

That distinction matters because a neat dashboard can make unlike quantities look self-explanatory. A percentile is not a percentage correct, a probability of succeeding, or a measure of a person’s overall value. It describes relative standing against that group and method. If the reference group is not named, a reader cannot tell whom the comparison represents. If a report changes its norms or scoring version, the same displayed number may not retain the same comparison meaning. Standards therefore emphasize documenting norms and interpretation.

A composite adds another layer. It can combine measured dimensions, but the final total is an index created by a particular scoring rule. The report should identify components, weights, and intended meaning. Hypothetically, two dimensions could be added with equal weight, or one could receive twice the weight of the other. The totals could order the same profiles differently. That change reflects the rule, not a new fact about either person. Scoring documentation is therefore needed to understand the total.

The Standards for Educational and Psychological Testing supports a practical reading rule: identify the score type, scale and scoring procedure, the norm or reference group when one is used, and the interpretation attached to it. These details distinguish a defined construct score from a decision-specific index. A defined score’s format alone does not establish job relevance or prediction. Before comparing people, request the score definition, scoring rule, version, and reference group. Without them, a rank has no clear meaning beyond the stated scale.

Sources: The Standards for Educational and Psychological Testing

How does combining traits create a second claim?

Combining several trait scores into one total can be defensible only when the components and their weights fit the intended job decision. The total is a new composite, not automatically a natural measure of personality or job ability; its weighting rule needs its own rationale and supporting evidence. The Canadian Psychological Association’s “Psychology Works” Fact Sheet describes why an employer might construct one: an overall score can make the selection rule more explicit. It describes both an average of relevant trait scores and a weighted average based on job-relevance ratings. Neither choice is self-justifying. An unweighted average gives each included trait the same influence. A weighted composite gives some traits more influence according to a stated rationale. A third approach estimates weights from collected data about scores and a defined outcome. Job experts may judge which traits appear relevant; that judgment can guide what goes into a composite. Empirical estimation asks how scores relate to an outcome in observed data. Neither step alone shows that the resulting score works for a different employer, applicant population, or assessment version. The fact sheet says data-derived weights require large samples and cross-validation to be useful. Cross-validation checks a model on data not used to fit it, The fact sheet offers a counterexample to a blanket objection to ranking: it says an overall score may rank applicants as one part of a selection process, alongside activities such as a structured interview. The sum is not itself a particular trait or characteristic. A composite can therefore be a decision index whose meaning comes from its documented construction and purpose, rather than a discovery of one hidden quality called “overall personality.” The Society for Industrial and Organizational Psychology’s “Principles for the Validation and Use of Personnel Selection Procedures” treats scoring as part of a selection system. It notes that changing components or their combination can affect reliability and subgroup passing rates. A defensible recipe does not prove every close rank is meaningful or that the rule transfers to another setting. Ask who selected the components and weights, what work analysis or outcome evidence supports them, and whether the composite as used was evaluated. If those answers are unavailable, the total may prompt reflection, but it cannot support the added claim that its ordering distinguishes likely work outcomes.

Sources: Psychology Works Fact Sheet: Pre-employment Personality Assessment in Personnel Selection; Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition

Why can two nearby scores fail to support a meaningful order?

Two applicants can receive different numbers without evidence showing that one has a meaningfully higher level of the measured tendency. Reliability describes how consistently scores are produced across specified conditions, such as occasions, test editions, or raters. Measurement error is variation that limits confidence in an observed score. The ETS memorandum “Test Reliability—Basic Concepts” defines reliability through consistency across replications; consistency alone does not establish that a score predicts work performance. That distinction matters when a report sorts people into first, second, and third place. The AERA, APA, and NCME “Standards for Educational and Psychological Testing” explain that a standard error of measurement estimates uncertainty and can support a confidence interval. Conditional standard errors, estimated at particular score levels, add information beyond a single average because precision may differ across the scale. The Standards also note that reliability coefficients can differ across subgroups when score variability differs. A broad-sample coefficient may therefore not answer how much confidence to place in a close comparison for a particular population or score range. Imagine, as an illustration rather than a real result, that two scores appear close and the report gives no score-specific precision information. Their displayed order is exact arithmetic, but that does not show the difference is stable. A standard error or another suitable estimate could help if it applies to the measure, population, and score range. Do not infer a margin from a generic coefficient: needed information depends on the instrument, scale, conditions, and intended interpretation. If precision varies across score levels, an overall average may conceal it. The Standards distinguish random from systematic error. A procedure can produce consistent scores while a systematic influence distorts what they mean; this may weaken validity without appearing as low reliability. Consistency is one question about a measure, not a complete check on interpretation. Overlapping uncertainty does not prove two people are identical. It means the report may not support a confident fine-grained ordering on that score alone. For employment decisions, the “Principles for the Validation and Use of Personnel Selection Procedures” advises considering conditional standard errors near a cutoff. That guidance concerns thresholds, but shows why uncertainty near a consequential boundary deserves attention. A full rank order creates apparent separation between adjacent pairs, even where documentation supports only broader distinctions. If the report supplies no relevant precision evidence, ask for it or avoid treating neighboring ranks as decisive. A score difference should carry only the weight its documented precision can bear.

Sources: The Standards for Educational and Psychological Testing; Test Reliability—Basic Concepts; Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition

What does a personality score have to predict before it can rank applicants?

To justify a workplace ranking, evidence must support the inference from a score to a defined job outcome for the relevant population and decision. A report may measure a tendency consistently; that alone does not show that higher scores predict better performance in this job or that sorting applicants by small differences improves hiring. Validity is evidence supporting a particular interpretation and use, not a permanent quality stamped onto a test. Prediction requires a criterion: a clear account of effective performance and a way to assess it. “Leadership potential” can conceal several outcomes. Does it mean coordinating a project or meeting a service goal? A report should connect its construct to work that can be described and evaluated, rather than rely on a trait name that feels desirable. The U.S. Office of Personnel Management’s *Designing an Assessment Strategy* says that if a personality test is meant to forecast customer service success, predictive evidence would show that scores relate to later job performance. Its validity-coefficient examples are general guidance, not evidence for an unnamed report or employer’s role. Findings from another instrument, sample, or job cannot simply be transferred; evidence should match the measure and proposed decision. OPM’s *Assessment Decision Guide* counters a blanket rejection of personality assessment. It says such measures have predicted job performance in various settings, while noting relevant factors depend on the job. It describes interpersonal or team settings as common contexts for use and highly scripted jobs as less useful contexts. This supports matching evidence to work, not treating a personality number as a universal measure of applicant quality. A score may add information alongside other methods, but more evidence is not automatically better. *Designing an Assessment Strategy* calls the added predictive contribution of one tool beyond another incremental validity. To claim a particular report adds value in a specific process, evidence should show it contributes job-relevant information beyond existing methods, with a rationale for how that contribution affects the decision. Ask what outcome the report is meant to predict, what evidence links this measure to it, and which jobs and populations are represented. A consistent score does not establish that the measure predicts this work. *Principles for the Validation and Use of Personnel Selection Procedures* notes mixed findings about linearity between personality measures and job performance. Without evidence for the specific link and ranking rule, a general association may not justify a fine-grained order; the number may prompt reflection but cannot carry an employment conclusion on its own.

Sources: Designing an Assessment Strategy; Assessment Decision Guide; Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition

What job question is hidden inside a personality ranking?

A workplace ranking assumes the measured tendency matters to the work and the definition of performance captures what the employer values. Without those definitions, a trait label can stand in for an unexamined idea of the “right” employee. The question is whether this score relates to a specified outcome in this role.

Job analysis connects a measure to a job decision by identifying important tasks and outcomes before choosing a predictor. The Canadian Psychological Association’s “Psychology Works” Fact Sheet: Pre-employment Personality Assessment in Personnel Selection describes personality-oriented work analysis. It says reviewers should consider trait definitions, not just names, and define effective performance for the job. This is professional guidance, not validation of any report. Its sequence is to define the work, then ask whether a measured tendency is relevant.

That sequence guards against a mismatch between what a scale measures and what an employer wants to infer. “Good collaborator” could mean sharing information, resolving disagreement, or coordinating a handoff. These are distinct behaviors; a score associated with one cannot automatically stand for all of them or overall performance. A report summarizes answers about tendencies. It does not observe every action across people, tasks, and conditions, or determine what someone will do in each situation.

Consider an illustrative decision, not a research finding: a role requires careful review of records and timely responses when a customer’s situation changes. A preference for established procedures might help with consistency in the first task, while flexibility could matter more in the second. Neither establishes that one preference makes a person better overall. Employers must specify behavior and importance; one rank can hide tradeoffs between duties.

The U.S. Equal Employment Opportunity Commission’s Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures says ranking is permitted when validity evidence supports that use. This is U.S. federal guidance; it does not validate an unnamed report or define a job. Alongside the CPA fact sheet, it suggests a test: can the employer name an observable, job-relevant outcome, explain how the score relates to it, and show why that relationship supports ranking? If not, the number may carry an unestablished assumption.

Translate the report’s claim into a checkable question: what behavior should this score help anticipate, and how would effective performance be recognized? Ask whether evidence concerns that behavior and role, rather than a broad impression of personality. A tendency is information about work, not a verdict about a person’s capacity or worth.

Sources: Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures; Psychology Works Fact Sheet: Pre-employment Personality Assessment in Personnel Selection

What is the fair comparison: one leaderboard or a job-related evidence picture?

The leaderboard begins after several choices. Someone has selected traits, decided how component scores should be combined, chosen a reference group, and set a rule for sorting the numbers. The ranked list can make these choices look like a direct reading of who will do best. It is instead a chain of claims: the score represents a tendency; that tendency matters for this job; differences on the score correspond to differences in an outcome; and the ordering should change a decision. A report that shows only the final number leaves the reader unable to inspect that chain. A job-related evidence picture starts with the work rather than the available score. First define an outcome that matters for the role in observable terms. Then ask whether this personality measure captures a relevant tendency and whether evidence connects it to that outcome for the intended use. Inspect score precision and the fit between the people and conditions studied and the decision now being made. Finally, ask what additional evidence contributes and how each part affects the decision. The U.S. Office of Personnel Management’s “Designing an Assessment Strategy” calls the added predictive contribution of one assessment beyond another incremental validity. Its point is conditional: methods can add distinct information when they measure different job-related factors. It does not mean every extra test improves a decision, or that a personality measure earns a place simply because it produces a score. Other methods can examine different parts of the work. A work sample makes sense when it represents important role tasks; OPM explains that a sample intended to mimic actual job tasks needs a convincing match to the job’s content. A structured interview can use consistent job-related questions and scoring to make answers more comparable than an informal conversation. These methods still need suitable design and evidence. The question is not whether observable methods are automatically fairer or more accurate, but whether each addresses a defined work requirement and adds information the others do not provide. The comparison also has a fairness dimension. The U.S. Equal Employment Opportunity Commission’s “Employment Tests and Selection Procedures” says selection procedures should be properly validated for the positions and purposes in which they are used, and notes that disproportionate exclusion can raise questions under U.S. federal law. That guidance does not decide whether an unnamed personality report is appropriate; it makes job- and use-specific evidence part of responsible evaluation. A reader can ask: what distinct, job-relevant information does this personality result add, and how does it change the decision? If no one can answer, sorting people may add a precise-looking order without adding useful evidence.

Sources: Designing an Assessment Strategy; Assessment Method Considerations; Employment Tests and Selection Procedures

What can make a score ranking unfair or misleading?

A score can look precise while poorly matched to those being ranked. A percentile or band has meaning relative to the reference group and scoring method that produced it. If the norm sample differs materially from applicants, or the report does not describe who was included, the reader cannot assume the same position means the same thing in this decision. The Standards for Educational and Psychological Testing say validation samples and the populations they represent should be described, including relevant characteristics and factors that may limit representativeness. This calls for inspection, not an assumption that a particular norm is unsuitable. The comparison can also be distorted by how the assessment was presented or completed. The Standards explain that differences in directions, testing conditions, scoring, language, or response mode can reduce comparability when not appropriately handled. They describe accommodations as a way to remove barriers unrelated to the construct, while considering whether score interpretations remain supported. For a workplace personality report, ask whether respondents understood the items in the language used, had comparable conditions, and could access the assessment as intended. A changed condition does not automatically invalidate a score; the question is whether evidence supports interpreting scores comparably for this use. A single rank conceals these details, so the report should make its administration and interpretation limits available. Fairness in hiring raises a related but distinct question: what happens to groups when the procedure is used? The U.S. Equal Employment Opportunity Commission’s “Employment Tests and Selection Procedures” guidance says selection procedures may raise federal antidiscrimination concerns when they disproportionately exclude people on a covered basis, unless the employer can justify the procedure under applicable law. This concerns selection outcomes and a legal framework; it does not prove a particular personality score is biased or unlawful. Nor does an individual report reveal whether the process has disparate effects. Analysis needs appropriate group and decision data, plus evidence that the procedure relates to the particular job. The guidance is U.S.-specific and does not settle requirements elsewhere. A defensible fairness review should connect the report’s norm and validation populations, language and administration conditions, and score meaning to the actual people and job decision. It should examine selection outcomes rather than infer fairness from a polished report or the absence of an obvious problem in one person’s result. Ask which populations and roles the evidence covers, whether conditions fit, and how group outcomes are monitored. If those answers are unavailable, a ranking is not established as fair merely because everyone received a number; it should not carry more weight than the evidence supports.

Sources: The Standards for Educational and Psychological Testing; Employment Tests and Selection Procedures; Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition

Is a single score ever defensible?

Yes, when the score and its ordering are supported for a defined selection decision. The Society for Industrial and Organizational Psychology’s Principles for the Validation and Use of Personnel Selection Procedures treats top-down ranking as one strategy alongside cutoffs and score bands, not as a default for any personality report. The guidance says evidence on whether personality measures relate linearly to job performance across the score range is mixed.

A cutoff asks whether a score meets a threshold; a band groups scores whose candidates are treated alike. Bands can account for imprecision, though the Principles notes they may reduce predicted outcomes compared with top-down ordering. Neither rule repairs a measure without relevant evidence. A cutoff needs a defensible link to a performance threshold; a band needs a documented basis and decision rule.

The Canadian Psychological Association’s “Psychology Works” Fact Sheet: Pre-employment Personality Assessment in Personnel Selection describes combining job-relevant trait scores using weights from a work analysis, then using the total with other selection activities. It frames the composite as a score for ranking applicants, not as one personality trait. This counters a blanket ban on composites, but is practice guidance, not validation of a particular instrument, job, weighting scheme, or close score difference.

A defensible ranking needs documentation for the exact use: which work requirements determined its components, what job outcome it is meant to relate to, whether evidence supports that relationship for the relevant population, how precision affects consequential distinctions, and why rank order is the chosen rule. Without those links, avoid treating a leaderboard as meaningful. With them, read it only within the defined decision; it does not establish a person’s overall worth or future.

Sources: Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition; Psychology Works Fact Sheet: Pre-employment Personality Assessment in Personnel Selection

What should you ask before acting on the report?

Before acting, request the report’s technical or interpretive documentation. Find the construct measured, norm group, population and purpose covered by the evidence, and the rule that converts scores into a rank. Ask what work outcome the score is meant to predict and whether evidence links this version of the measure to that outcome for a comparable job. The U.S. Office of Personnel Management’s “Designing an Assessment Strategy” says a personality test intended to forecast job success needs evidence relating scores to later job performance; it also explains that different job-related methods can contribute distinct information. Ask how close scores are handled. Without relevant precision information, a narrow gap should not become a firm distinction. Before accepting an employment inference, request an observable example or complementary evidence tied to the work. For private reflection or coaching, choose one tendency to discuss and identify a behavior to notice at work. The live Work Pattern Report at /assessment is a low-stakes self-report across decision and collaboration patterns, with no norms, cutoff, type, or selection score. It can support reflection, but cannot rank applicants or recommend a job. For help interpreting reports, continue to /topics. A number means only what its evidence supports. Ask for that evidence, or turn the result into one observable question.

Sources: Designing an Assessment Strategy; Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition

Questions readers ask