Why Hiring Validity for a Personality Assessment Is Job-Specific

Why can’t a validity result travel by itself?

A hiring validity result does not automatically apply to every job because validity supports a particular interpretation of scores for a particular use. A correlation reported for one measure, outcome, group, and work setting is evidence about that arrangement; it is not a permanent property attached to the test name. Before carrying it elsewhere, ask whether the score means the same thing, whether the target work calls for the relevant behavior, whether the outcome is comparable, and whether the people and conditions resemble those represented in the evidence.

The Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition describes this as a question of transportability: evidence from one setting may support another when the source evidence is sound and the relevant similarities are established. That leaves room for accumulated research and reasoned comparison; a new local study is not automatically required for every role. But a broad average or shared trait label cannot do the comparison on its own. The evidence must connect the measure and score interpretation to the work and decision now proposed. The article later checks one disputed moderator claim against the full study, illustrating why a headline summary alone is not enough to establish that connection.

This distinction changes how to read a vendor or research claim. “The assessment is valid” leaves out the score interpretation, criterion, population, and decision that make the evidence meaningful. A more useful claim names those boundaries and explains why they fit the proposed use. If that explanation is absent, the sensible conclusion is that transfer has not yet been shown for this use—not that the measure can never be useful elsewhere.

Sources: Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition

What makes two jobs similar enough to compare?

To use evidence from a studied job when considering a target job, compare the work people actually perform and the conditions under which performance is judged. Similar titles are weak evidence by themselves: a title can cover different responsibilities across employers, and different titles can describe work built around much the same behavior. The Uniform Guidelines on Employee Selection Procedures make job analysis and the relationship between the studied work and the target work central to this comparison. Their guidance is U.S.-specific; it is a framework for examining selection evidence in that context, not a universal rule for every country.

Begin with tasks and observable behaviors rather than adjectives in a job description. What decisions recur? Which actions must be performed, how often, and under what constraints? What products or outcomes count as successful work? A role described as “detail-oriented,” for example, might require checking records against explicit rules, catching exceptions in incomplete information, or maintaining accuracy while switching between requests. Those are different behavior demands. A personality score associated with a broad performance rating does not reveal, by itself, which of these demands it may relate to or whether the association holds in the target setting.

Then compare context. A task performed with a stable procedure, close supervision, and immediate correction is not necessarily equivalent to a task requiring independent judgment, shifting priorities, or coordination across teams. Note the tools, discretion, pace, consequences of error, support available, and opportunity to learn. These details matter because they can change what a behavior looks like and how much it contributes to the outcome. They also keep the comparison specific: “both jobs require conscientiousness” is a proposed interpretation, while a documented account of the recurring actions and conditions gives that interpretation something observable to answer to.

Finally, check how the evidence was produced and whom it represents. The studied sample, assessment conditions, scoring, and performance criterion should be sufficiently relevant to the target use. If the source evidence concerns people already doing a job and their supervisor ratings, that does not automatically describe applicants taking a selection test or a different outcome such as error rates. The Uniform Guidelines on Employee Selection Procedures: Questions and Answers recognizes that external studies can inform another setting under conditions; it does not make a job title a substitute for showing the relevant similarities. The comparison should explain which dimensions match and where the target differs, rather than implying that every contextual detail must be identical.

Consider this hypothetical comparison: one employer wants to use a personality score for an operations role, while the available study concerns a different role. The titles alone cannot settle the issue. The employer would need to compare recurring tasks, behavior demands, work conditions, the performance measure, and the people and assessment conditions represented in the study. If both roles involve checking complex records against changing requirements, that shared work may be relevant; if one role instead centers on rapid customer negotiation, the overlap may be smaller. These examples illustrate questions to investigate, not findings about either occupation or proof that a particular score predicts either outcome.

A sound comparison can therefore support cautious use of external evidence without claiming that jobs are interchangeable. Different titles may conceal close task and context overlap; identical titles may conceal changed technology, authority, or performance expectations. The decision rests on the documented work and the match between the evidence and proposed interpretation, not the label attached to the vacancy. If the match is partial, say which parts are supported and which remain uncertain. That is more informative than either declaring transfer impossible or treating a general personality association as proof for every job.

Sources: Uniform Guidelines on Employee Selection Procedures; Uniform Guidelines on Employee Selection Procedures: Questions and Answers

How much can a broad average say about one job?

An overall conscientiousness association establishes that, across the studies included in a synthesis, higher measured conscientiousness tended to accompany higher measured job performance. It does not establish the size of the association for a particular job, assessment, applicant pool, or hiring rule. A pooled correlation is a summary across included observations; role-specific prediction is a further inference that depends on what those observations represent and how consistently the relationship holds. The distinction matters because a positive average can coexist with variation across studies and with uncertainty about the use a reader has in mind.

The 2023 study *The criterion-related validity of conscientiousness in personnel selection: A meta-analytic reality check* reported a mean raw correlation of .17 across 102 effects and 23,305 participants. The estimate describes the literature it assembled on global conscientiousness and overall performance; it is not a coefficient for every occupation or every personality instrument. Its confidence interval for the pooled mean was .15 to .19, while its prediction interval for effects across settings was .03 to .31. That wider interval is a useful reminder: the uncertainty around the average and the spread expected across settings answer different questions. Even a well estimated mean does not tell a reader where a specific job lies within that distribution. [Study](https://onlinelibrary.wiley.com/doi/10.1111/ijsa.12413)

The same study found substantial between-study heterogeneity and tested whether applicant versus incumbent samples, predictive versus concurrent designs, and their combination explained it. It did not detect statistically significant differences for those moderators. That result is relevant counterevidence to a categorical claim that incumbent findings cannot inform applicant settings: the observed data did not show a clear difference on those coded comparisons. But absence of a detected difference is not proof that the settings are equivalent. In particular, only 10 effects represented applicants, and the applicant-predictive subset contained nine effects. The authors therefore concluded that the evidence under realistic applicant conditions remained limited. The appropriate reading holds both points together: broad evidence supports an association, and the available moderator test does not settle whether a given selection context behaves like the incumbent-heavy evidence base.

A separate challenge concerns whether the pooled literature itself gives an unbiased magnitude. *The Validity of Conscientiousness Is Overestimated in the Prediction of Job Performance* reanalyzed conscientiousness evidence with sensitivity analyses for influential samples and publication bias. Its authors reported that estimates were moderately overstated overall, by roughly 30 percent or about .06 in correlation, with small effects missing disproportionately from journal literature. Their analysis used several methods, including funnel-plot diagnostics, trim-and-fill, selection models, and cumulative analyses; the authors argued that convergence across methods strengthened their publication-bias interpretation. This is not a direct estimate of validity for a specific assessment or role, but it makes a single headline average less conclusive than it may first appear. [Full article](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0141468)

The two syntheses do not simply cancel each other out. The 2023 analysis asks how conscientiousness relates to performance and whether sample type and study design moderate that relation; the 2015 critique asks how sensitive accumulated estimates are to missing or selectively published effects. They cover different evidence and methods, so the critique does not show that the association is zero, and the later mean does not erase the earlier concern. The practical conclusion is narrower: the broad association is evidence worth considering, while its magnitude and transfer to a target use remain open to scrutiny. A claim about one vacancy needs more than a trait-level average; it needs a defensible link from the exact score and work criterion to the target context. That role-level question is the next step in the argument, not something the pooled coefficient answers by itself.

Sources: The criterion-related validity of conscientiousness in personnel selection: A meta-analytic reality check; The Validity of Conscientiousness Is Overestimated in the Prediction of Job Performance

What does the disputed routinization finding actually show?

No. The 2013 paper *The Validity of Conscientiousness for Predicting Job Performance: A Meta-Analytic Test of Two Hypotheses* does not establish that conscientiousness predicts performance more strongly in highly routinized jobs. Its abstract says that it does, but the full paper’s weighted least squares (WLS) moderator test reports no significant routinization moderation (b = −.11, ns). When an abstract summary and the article’s reported test conflict, the answer to a question about what the study statistically established should follow the detailed results and the authors’ discussion of those results—not the abstract alone. [Full paper](https://emilkirkegaard.dk/en/wp-content/uploads/The-Validity-of-Conscientiousness-for-Predicting-Job-Performance-A-meta-analytic-test-of-two-hypotheses.pdf)

The distinction is between descriptive estimates in categories and a test of whether the moderator explains differences across estimates. In the subgroup table, the reported mean correlations for high-, medium-, and low-routinization jobs were .11, .15, and .12. Those values can look like a pattern when read quickly: the high-routinization estimate is close to the low-routinization estimate, and the middle category is higher. Yet three subgroup means do not by themselves show a reliable upward or downward trend. Their ordering is not monotonic, and each estimate comes from a different collection of studies and jobs. Comparing selected values visually is not a substitute for the regression that tests the proposed relationship across the coded moderator values.

The authors conducted the moderator analyses on a dataset of 75 effect sizes from 12,331 participants. They coded job characteristics from O*NET, including a routinization index built from features such as automation, the importance of accuracy and repetition, and job structure. WLS regression related the coded job characteristic to each study’s conscientiousness-validity estimate. In that model, the routinization coefficient was −.11 and nonsignificant. The paper’s discussion consequently says the results did not support its prediction that conscientiousness validity would decrease as jobs became more structured; it also says validity was not lower for highly routinized jobs. Neither statement means the study proved identical validity at every level. It means this analysis did not provide statistically significant evidence for the proposed routinization trend.

Why, then, might the abstract imply the opposite? The abstract compresses the paper’s conclusion into a sentence that is inconsistent with the routinization regression and with the discussion’s explicit account of the test. Descriptive subgroup values may explain how a reader could be drawn toward a broad interpretation, but they cannot reconcile that contradiction or override the nonsignificant coefficient. The careful account is to identify the inconsistency and give the reader the actual test result. It would be unsafe to repeat the abstract as though the full paper confirmed it, and equally unsafe to infer that routinization never matters in any job or measure. A single meta-analysis cannot establish either universal claim.

The contrast with the paper’s second hypothesis shows why the coefficients should not be blurred together. For cognitive-ability requirements, the WLS table reports b = −.43 with p < .01, a statistically significant negative moderator: in the analyzed studies, conscientiousness validity tended to be lower as the coded cognitive-ability requirement increased. This is a separate finding, tested with a different job characteristic. It is not evidence for a routinization effect, and it does not license a stereotype that people in any named occupation are more or less conscientious. The paper uses occupation-linked job analysis codes to compare studies; its model is not an experiment that assigns workers to job conditions or isolates a causal mechanism.

The authors themselves note a limit on interpreting this second result: their meta-analysis tested differences between jobs using average job-level cognitive-ability requirements, not the interaction between individual workers’ measured ability and conscientiousness. They discuss cognitive buffering as a possible explanation, but the analysis does not directly demonstrate that mechanism. The coefficient therefore supports a moderator association within this dataset, not a causal account of why it appeared. This is a useful contrast with routinization: one proposed moderator was not significant in the regression, while another was. Naming the test and the level at which it operates keeps the evidence from turning into a stronger story than the design can sustain.

For a reader evaluating an employer’s claim, the corrected takeaway is specific. Shaffer and Postlethwaite’s full 2013 results do not support using their study to say that a personality measure will predict more strongly in a highly routinized role. The paper does report a separate association involving cognitive-ability requirements, but that result remains bounded by its job-level coding, included samples, and overall-performance criterion. Neither finding identifies the validity of a different test in a particular vacancy. When a claim depends on this paper, check whether it cites the abstract’s shorthand or the WLS result, and whether it keeps the two moderators distinct.

Sources: The Validity of Conscientiousness for Predicting Job Performance: A Meta-Analytic Test of Two Hypotheses

What performance outcome did the study actually measure?

A validity result speaks only to the outcome that was measured, so “job success” is too broad unless the study defines it and shows why it represents the work in question. A criterion can resemble the target work yet omit important parts of it, or include influences that the assessment should not be credited with predicting. This is a different question from whether two jobs have similar tasks: even a strong job match cannot repair a weak or mismatched outcome measure.

Criterion correspondence asks whether the recorded outcome captures the behavior or result the proposed decision cares about. If the role depends on accurate case resolution, a count of cases closed may correspond only partly: it records volume, not whether resolutions were correct or durable. A supervisor’s overall rating may include accuracy, cooperation, initiative, and dependability, but unless its content and use are clear, the reader cannot tell which of those dimensions drove the reported association. The Uniform Guidelines on Employee Selection Procedures treats criterion-related support as a relationship to a defined criterion and emphasizes documenting the work basis for its use. Its framework is U.S. guidance, not a universal legal rule.

Criterion deficiency is what the chosen outcome leaves out. A single task-output measure might capture speed while missing quality, safety, or coordination that is central to the job. A supervisor rating could cover a wider range of work but still omit infrequent high-consequence duties or contributions made outside the rater’s view. Deficiency does not mean every study must measure every imaginable task. It means the omitted parts matter when they are important to the target role and when the article or employer moves from a narrow outcome to a broad claim about success.

Criterion contamination occurs when the recorded outcome reflects something other than the work performance the assessment is meant to predict. For example, a rating may partly reflect whether a supervisor favors a particular communication style, or a retention outcome may reflect pay, schedule, management, or labor-market conditions alongside the employee’s work. These examples are possibilities to check, not claims about a specific study. Contamination can distort an association in either direction: an unrelated influence may make a score look more useful or mask a relationship that would appear with a better-targeted outcome. The reader needs to know how the outcome was gathered and what else could shape it.

Consider this hypothetical study of a service role. Its assessment score is compared with a supervisor’s single overall rating. If the claim is that the score predicts reliable case handling, the rating must meaningfully represent case accuracy and follow-through; if it mostly reflects visible enthusiasm, the criterion may not correspond well. If the rating excludes teamwork needed to hand off complex cases, it may be deficient for a claim about overall contribution. If it is affected by a supervisor’s preference for one style of interaction, that may contaminate the criterion. None of these concerns is solved just by showing that the study and target role have similar tasks.

A broad criterion is not automatically poor. A carefully defined composite can combine task output, quality, teamwork, and other outcomes when each component matters to the target work and the combination is justified. A supervisor rating may also be useful if its dimensions, raters, and collection process fit the intended interpretation. The practical test is whether the criterion matches the stated claim at the same level of breadth. Evidence about one rating, retention measure, or output cannot silently become evidence for every meaning of job success; conversely, a representative composite can support a broader conclusion than any single component alone.

Sources: Principles for the Validation and Use of Personnel Selection Procedures, Fifth Edition; Uniform Guidelines on Employee Selection Procedures

Does the measure match behavior at work?

Evidence for a personality measure does not automatically transfer to another questionnaire just because both use the same trait name. The relevant object is a specific score interpretation produced by particular items, instructions, scoring, and version. Work-contextualized wording may make a question more directly about behavior in a job setting, but its label alone does not establish that it measures the same thing, or predicts the same outcome, as a general self-report scale.

A Matter of Context: A Meta-Analytic Investigation of the Relative Validity of Contextualized and Noncontextualized Personality Measures reports higher average validity for work-contextualized measures than for general measures: r = .24 versus r = .11. These are pooled averages reported in the publisher abstract, not estimates for a named commercial test, vacancy, or hiring decision. The abstract also reports that checked moderators did not account for the difference. Because the full article was not available in the reviewed record, the result should be stated at this level and not expanded into unverified details about its samples or analyses.

The comparison is informative but does not show that adding work language caused a particular assessment to become more predictive. The meta-analysis compares bodies of studies; measures may differ in more than item framing, including their content, samples, criteria, administration, and research settings. A higher contextualized average could therefore reflect contextual wording, other differences among the studies, or some combination. The reported contrast supports asking whether an instrument’s items fit the behavior being discussed. It does not isolate wording as the explanation or provide a conversion rule from a general score to a work score.

For a transfer claim, inspect what respondents were asked to judge. A general item may ask how well a person usually follows through, while a contextualized item may ask about following through on work commitments. The second framing names a setting, but it still leaves questions: which commitments, over what period, and under what constraints? Then check whether response options and scoring preserve the meaning claimed for the score, and whether the tested version is the version proposed for use. Small changes in instructions, item set, scoring, or norm reference can change the interpretation even when a report keeps the same trait heading.

This is why a trait name cannot stand in for measurement equivalence. Two scales can both call a score conscientiousness while sampling different behaviors; two versions of one instrument can also differ in item wording or scoring. To move evidence across them, the argument needs documentation or research connecting the actual score interpretation and version to the relevant outcome. Without that bridge, the older finding may be useful background about a construct or family of measures, but it is not a demonstrated validity result for the new version. The .24 and .11 averages sharpen the question about context; they do not answer it for an individual product.

Nor does contextualization itself guarantee a better measure. Items can mention work while still failing to represent the target behavior, and a well-designed general measure may remain relevant when the work criterion and intended interpretation support it. The practical conclusion is to compare the item frame, instructions, scoring, version, and outcome rather than infer equivalence from a shared label. The 2012 averages make contextual fit worth investigating, while the between-study design and abstract-level evidence leave causation and instrument-specific transfer unresolved.

Sources: A Matter of Context: A Meta-Analytic Investigation of the Relative Validity of Contextualized and Noncontextualized Personality Measures

Does the proposed decision match the evidence?

A score may be associated with later performance without being suitable for every hiring decision. The key question is what the employer does with it. A forecast used as one input to a structured review has a different consequence from a score that screens applicants out, determines their rank, or outweighs other information. The more a score controls access to the job, the more important it is to show that the exact interpretation and decision rule are supported for the intended applicants and work. Predicting an outcome and justifying a use are related questions, but they are not interchangeable.

The distinction is visible in the 2023 meta-analysis *The criterion-related validity of conscientiousness in personnel selection: A meta-analytic reality check*. It reports an average association for global conscientiousness and overall performance, and no statistically detected difference across its coded incumbent/applicant or concurrent/predictive comparisons. That finding complicates a blanket claim that evidence from current employees can never inform selection. It does not establish that the settings are equivalent, or that the average predicts how a particular applicant pool will perform. The authors’ evidence base included relatively few real-applicant predictive studies, so conclusions under realistic applicant conditions remain less certain. A lack of detected moderation is a result about those comparisons and data, not proof that applicant conditions have no effect.

This boundary matters because a selection score is observed before the work outcome it is intended to forecast. Applicant responses may be shaped by the stakes and by what respondents think the employer wants, while incumbent studies observe people already in the job. Those are reasons to ask whether the evidence represents the proposed administration and population; they do not establish that applicants necessarily score higher, answer dishonestly, or produce a weaker validity coefficient. The prudent reading is conditional: the meta-analysis offers relevant evidence about conscientiousness, while sparse applicant evidence leaves the precision of transfer to a specific high-stakes use unresolved.

A second decision question is whether the score contributes useful information beyond the employer’s existing process. A variable can correlate with performance and still add little if the current method already captures much of the same information. Conversely, a measure with a modest association could contribute something distinct when combined with other well-supported predictors. The Office of Personnel Management’s *Designing an Assessment Strategy* explains this as incremental validity: evaluate the value of adding an assessment to the selection strategy, rather than reading a standalone correlation as proof that the addition improves decisions. Its displayed comparison table is adapted from Schmidt and Hunter (1998), so it illustrates a historical synthesis, not a universal forecast for a particular test, occupation, or current applicant pool.

The practical comparison is therefore between decision strategies: what information is already collected, what the personality score adds, how the inputs are combined, and what happens at the cutoff or ranking stage. An assessment that adds a distinct, job-relevant signal to a considered process may have a different rationale from the same score used alone to eliminate candidates. A composite does not become fair or useful merely because it combines numbers; its components, weights, threshold, and intended interpretation still need support. Nor does a small incremental gain automatically justify a consequential use: the decision maker must judge its value alongside error, burden, and the costs of excluding qualified people. OPM’s framework helps pose this question; it does not supply a ready-made answer for every employer.

So when someone says a personality test predicts hiring success, ask what the score is being asked to do. If the claim is only that it is associated with a broad performance criterion, that is one level of support. If it screens, ranks, or changes a composite decision, the evidence should also fit that specific use and show why the score improves the process in the target setting. The 2023 result supports neither automatic rejection of applicant use nor automatic transfer to it. The decisive issue is the match between evidence, applicant conditions, and the actual rule that turns scores into decisions.

Sources: The criterion-related validity of conscientiousness in personnel selection: A meta-analytic reality check; Designing an Assessment Strategy

What should you ask before relying on the claim?

Ask the employer or test provider to show the link between the result and the decision being made. Which assessment version, score, and scoring rule are involved? What target work was analyzed, and which performance outcome is the score said to predict? Who was studied, under what conditions, and was the evidence predictive or based on people already in the role? Finally, how will the score be used: as one input, a screen, a rank, or part of a composite? These questions clarify the claim; they do not mean every employer must run a new local study. Evidence from other settings can inform a decision when its relevance is explained. Ask, too, whether a threshold or ranking rule was evaluated and what alternatives were compared. A polished report or familiar trait label does not answer those questions by itself. Request documentation rather than relying only on a sales summary or a general statement that the test is validated. If the details are missing, the support for this particular use is unclear—not proof that every personality measure is useless. For a private, low-stakes way to reflect on how you decide, plan, collaborate, handle conflict, adapt, and learn, you can try the live [Work Pattern Report](/assessment). Its report describes patterns for reflection; it supplies no hiring score or job recommendation and is not validated for selection.

Questions readers ask