Should a Personality Report Claim Incremental Validity?

What does ‘incremental validity’ mean in this report?

Incremental validity means that a measure contributes information about a defined outcome beyond information already available. It is a comparison, not a general badge of test quality. Here, the claim should mean that adding a personality score improved a specified prediction or interpretation after a stated baseline was considered. Haynes and Lench’s review, “Incremental validity of new clinical assessment measures,” defines it by what a measure explains or predicts relative to other measures, and notes that findings depend on the criterion, comparison measures, and sample. The baseline is the existing information used for comparison; the criterion is the outcome or target being predicted. Researchers compare the baseline alone with that same baseline plus the personality score, for the same target and sample. The added score may provide distinct information or overlap with what the baseline already captures. Hunsley and Meyer’s review, “The incremental validity of psychological testing and assessment,” identifies predictor-entry order, increment size, and criterion choice as factors affecting interpretation. A report’s claim should be paraphrasable as: “This score adds X beyond Y for Z.” Without those particulars, “the report predicts success” leaves both the comparison and target unclear.

Sources: Incremental validity of new clinical assessment measures; The incremental validity of psychological testing and assessment: conceptual, methodological, and statistical issues

Why do the baseline and outcome change the answer?

The same personality score can add information beyond one baseline and little beyond another. It may relate to one outcome more clearly than another. Incremental validity describes a particular comparison, not a permanent property of a report. “Incremental Validity of New Clinical Assessment Measures” defines it as the degree to which a measure explains or predicts a phenomenon relative to other measures. The review says the index can vary with the criterion, comparison measure, and sample. Its clinical focus supports this logic, not a finding about a workplace report. A baseline is the information included in the comparison. A criterion is the outcome the analysis aims to explain or predict. Researchers compare the baseline alone with the baseline plus the personality score, keeping the criterion and evaluation procedure steady. The question is whether the score contributes after the baseline is considered, not simply whether it relates to the outcome. If a report leaves the baseline unnamed, “adds value” has no clear reference point. Consider an illustrative selection decision, not a research finding. A team wants to anticipate a defined behavior at work. It uses a structured interview, then asks whether a broad personality score improves prediction. The result addresses the score’s contribution after that interview for that outcome and studied group. Now imagine the baseline is a work sample requiring applicants to perform a relevant task. This asks whether the score contributes beyond evidence closer to the target behavior. The score and outcome have not changed, but the decision question has. The baseline should reflect available information and intended use. Predictor-entry order matters because a score entered later receives credit for what it explains after earlier predictors are included. “The Incremental Validity of Psychological Testing and Assessment” identifies entry order, interpreting increment size, and artifacts in selected criteria as issues that can influence findings. Order should follow defensible decision logic, not be selected after trying alternatives until one looks favorable. A weak baseline can make an added score appear more distinctive; a strong baseline may leave less unique information to add. The outcome needs scrutiny too. A narrow proxy may be easy to count without representing the decision people care about. A result for one behavior does not automatically establish value for broad “success.” The U.S. Office of Personnel Management’s “Designing an Assessment Strategy” describes selection comparisons between tools, but does not validate a particular personality instrument or outcome. “Statistically Controlling for Confounding Constructs Is Harder than You Think” shows how regression-based incremental construct claims can be vulnerable when related constructs’ measurement unreliability is ignored under the conditions examined. This is a caution, not proof that every increment is spurious. Ask whether the baseline resembles information available in the decision and whether the criterion is meaningful rather than convenient. Without both answers, the claim does not say what the added score contributes.

Sources: The incremental validity of psychological testing and assessment: conceptual, methodological, and statistical issues; Incremental validity of new clinical assessment measures; Statistically Controlling for Confounding Constructs Is Harder than You Think; Designing an Assessment Strategy

What can an increment statistic tell you—and what can’t it?

A fit or prediction statistic can describe how much a personality score improves a defined model after a baseline is included. By itself, it cannot show that the score is precise, its interpretation sound, or a particular person likely to behave as predicted. Separate three questions: did the model improve, how large and uncertain was the improvement, and would it matter for the decision? One common summary is the change in explained variance, written ΔR². R² describes the share of variation in an outcome accounted for by a model in the analyzed data; ΔR² is the change when predictors are added. It is not a percentage increase in a person’s chance of succeeding, nor does it show how far individual predictions miss. Other reports compare prediction error or classification accuracy. These summaries differ: explained variance may change while prediction error barely moves, and accuracy may hide which errors occur. The report should name its statistic rather than compress these distinctions into “better prediction.” The review “The incremental validity of psychological testing and assessment: conceptual, methodological, and statistical issues” identifies increment size, predictor entry order, and criterion choice as issues affecting interpretation. A ΔR² needs its outcome, comparison model, sample, and uncertainty to be meaningful. Statistical significance does not show whether a gain is substantial or useful. With enough data, a small increment may be detectable while changing no decision. With limited data, an estimate may be too uncertain to distinguish a gain from sampling variation. Significance alone cannot tell which situation applies. Score consistency is separate. Reliability concerns how consistently a measure produces scores under stated conditions. More consistent measurement can support useful prediction, while measurement error can weaken observed relationships. But consistency does not establish that a score measures the intended construct or supports a particular interpretation. “Incremental validity of new clinical assessment measures” notes that indices vary with criterion, comparison, and sample, and distinguishes an increment from cost-benefit considerations. A model improvement does not establish the score’s meaning, suitable reference group, or intended use. The U.S. Office of Personnel Management’s “Designing an Assessment Strategy” treats reliability and validity as different considerations in employment assessment, and says evidence should fit the assessment’s use. This selection guidance does not validate any particular personality report. It illustrates a broader reading rule: a statistic supports a bounded claim about a model and outcome, not an all-purpose badge of accuracy. Look for the statistic’s name, scale, criterion, comparison, sample, and uncertainty, then ask what decision the gain could change. “Significant improvement” without those details is incomplete. It does not prove the result meaningless; it leaves the reader unable to judge its size or consequence.

Sources: The incremental validity of psychological testing and assessment: conceptual, methodological, and statistical issues; Incremental validity of new clinical assessment measures; Designing an Assessment Strategy

Does the gain survive new data?

A gain measured only in the data used to build a prediction model is weaker evidence about future performance than a gain that persists in held-out or new data. In-sample means the same observations help estimate the model and assess its added value. Out-of-sample means the prediction rule is estimated on one set of observations and evaluated on different observations. This matters because a model can fit chance details in its development data. Those details may look like useful signal but fail to recur elsewhere, a problem called overfitting. A report making a predictive claim should say where its increment was evaluated. The 2024 methods paper “An out-of-sample perspective on the assessment of incremental predictive validity” examined this issue through simulations and an empirical application. The researchers varied factors including sample size and test reliability. They found a difference between within-sample and out-of-sample assessments, especially in small samples. The study offers no universal adjustment for any reported increment. It supports a narrower caution: under the simulated conditions, a within-sample estimate could give a more favorable picture than evaluation on separate observations. The paper concerns methods for incremental prediction, not personality reports generally; it does not show every gain is inflated or every independent estimate precise. A random train/test split reserves part of a dataset for evaluation: the model is fitted on the training portion and assessed on held-out observations. Cross-validation repeats this process across partitions. Independent replication tests the claim in a separately collected sample. These methods answer related questions, but none guarantees transfer. A split can leave too few observations for stable evaluation. Cross-validation uses one dataset and requires model-building to stay within each training fold. Replication can test transfer more directly, yet may not represent a different population or outcome. The design should match the claim; its label is not enough. The 2003 review “The incremental validity of psychological testing and assessment: conceptual, methodological, and statistical issues” identifies predictor-entry order, increment size, and possible artifacts in selected criteria as factors that can affect findings. It reinforces why a validation result needs its comparison and method disclosed. Sample size, criterion measurement, and uncertainty around the estimated gain shape confidence in an apparent improvement. A report should distinguish evaluation on development data from evaluation on held-out or new data, and name which supports its claim. For a reader, ask whether the added score improved performance in observations that did not help build the prediction rule. Does the report identify the validation sample, whether it was independent, and whether model choices were made without using its results? If evidence comes only from development data, describe the finding as preliminary evidence of added fit, not established future performance. Even a gain that survives evaluation applies first to the studied population, criterion, instrument, and procedure. It strengthens a predictive claim only when those conditions fit the use; it does not establish what will happen for one person.

Sources: An out-of-sample perspective on the assessment of incremental predictive validity; The incremental validity of psychological testing and assessment: conceptual, methodological, and statistical issues

When does an added score matter in practice?

An added score matters when it changes a defined decision enough to justify the burden of collecting and using it. Statistical improvement and practical usefulness are related, but they are not the same finding. “Incremental validity of new clinical assessment measures” says measure choice is moderated by the cost of acquiring data, the importance of the phenomenon, and the utility of the new information. The review concerns clinical assessment, so applying its decision logic to a personality report is a reasoned extension, not direct evidence that a report is useful. Make the practical question concrete: if this score were absent, what would the reader, coach, or decision-maker do differently? If the answer is “nothing,” a measurable gain may have little value in that setting. If it prompts a different follow-up question, its value may be reflective rather than predictive. In coaching, for instance, a suggestion that someone prefers time to organize a response could prompt them to check recent meetings: did they ask for time, and did that help them contribute? This illustrates a low-stakes use; it does not show the score predicts performance or that the interpretation is accurate for that person. The reader can accept, reject, or refine the prompt against experience. Costs also vary by setting. A private reflection exercise may ask for time and attention; a workplace assessment may involve collecting personal responses, administration effort, and consequences if an interpretation is wrong. Ask who supplies the information, who sees it, who benefits, and who bears the cost of error. A score that adds convenience for an organization but exposes an individual to an unsupported label has a different practical balance from a voluntary prompt the person controls. Privacy and fairness belong in the usefulness judgment alongside predictive performance. In employment selection, the question is narrower and stakes are higher: does the added information improve a job-related decision for the intended role and population enough to justify administration and error? The U.S. Office of Personnel Management’s “Designing an Assessment Strategy” frames incremental validity as predictive validity added by one assessment relative to another, within an assessment strategy. This is employment guidance, not validation of a particular personality instrument. It supports asking what the added tool contributes to the existing process; it does not establish that a general personality score should affect a hiring outcome. A report should connect the claimed increment to a consequence: what decision may change, what the added score costs in time and data, and what uncertainty remains. If a simpler conversation, work sample, or existing record can answer the same question with less burden, the gain must be meaningful enough to justify the added measure. The test is not merely whether a number improved, but whether that improvement changes an action for the people affected.

Sources: Incremental validity of new clinical assessment measures; Designing an Assessment Strategy

What does a real personality-selection example establish?

Work-selection research makes it plausible that personality measures can contribute to some defined criteria, but current evidence does not license a universal claim about every instrument, role, applicant, or individual. A trait’s association with performance does not show that a particular score improves prediction beyond existing information. A report that cites the first as proof of the second has skipped the comparison that incremental validity requires. One example is conscientiousness, a personality dimension studied in relation to job performance. “The Criterion-Related Validity of Conscientiousness in Personnel Selection: A Meta-Analytic Reality Check” pooled field studies of self-reported conscientiousness and supervisor-rated overall performance. The overall estimate was a correlation of .17 across 102 estimates and 23,305 participants. The positive average was modest and left much variation unexplained. This is evidence of association across included settings, not a guarantee about a person or a test’s usefulness in a particular decision. The design breakdown matters too. The authors reported that only about 12% of studies used real applicants in predictive designs, where a measure is taken before later performance data are collected. Much of the literature involved current employees and concurrent designs, where predictor and outcome data are collected around the same period. The meta-analysis found no statistically significant differences between applicant and incumbent estimates, or between predictive and concurrent estimates. This counters assuming incumbent studies necessarily exaggerate every relation. Yet applicant-predictive evidence was sparse, and the authors concluded that whether self-report conscientiousness retains predictive validity under applied conditions remains open. A non-significant difference in sparse evidence does not prove the settings interchangeable. The .17 correlation is bivariate: it summarizes the relationship between one predictor and an outcome. It does not show whether conscientiousness adds information beyond a named baseline, such as a structured interview or work sample. The study calls for more multivariate research on selection procedures in predictive settings with actual applicants. An incremental claim requires comparing the baseline alone with that baseline plus the score, for the same criterion and population. The U.S. Office of Personnel Management’s “Designing an Assessment Strategy” similarly defines incremental validity by how much predictive validity one tool adds relative to another. A trait’s standalone association cannot supply that missing result. Pooled estimates also depend on available evidence and how robustly it is examined. “The Validity of Conscientiousness Is Overestimated in the Prediction of Job Performance” analyzed 113 correlations from an earlier meta-analysis. It reported an unadjusted random-effects mean of .16 and a 90% prediction interval from .03 to .29, indicating variation across settings, not a range for any one person. Its publication-bias analyses suggested overstatement, particularly for non-contextualized measures and journal-published findings. This complicates reliance on one average. It does not show every association is spurious: some subgroups, including contextualized measures, appeared more robust, and conclusions depended on studies and methods. These findings limit what may travel from research to a report. Evidence about conscientiousness as a broad construct does not automatically validate a different instrument, its scoring, or interpretation. Current-employee evidence does not settle prediction among applicants; an association measured alongside performance does not establish prediction of later performance. A group-level increment describes average improvement under defined conditions, not certainty about one person. A report citing general trait-performance research should present it as background, not validation of its own score or a particular hiring use. For that stronger claim, it should identify the instrument, baseline, outcome, population, validation design, and size and uncertainty of the added performance. Without them, research warrants testing a bounded claim, not generalizing it.

Sources: The Criterion-Related Validity of Conscientiousness in Personnel Selection: A Meta-Analytic Reality Check; The Validity of Conscientiousness Is Overestimated in the Prediction of Job Performance; Designing an Assessment Strategy

How broad may the report’s conclusion be?

Only as broad as the supporting study. An incremental-validity result belongs to a particular instrument, sample, baseline, criterion, model, and intended use. Rewording that result cannot turn it into a universal description of a person. The review “Incremental validity of new clinical assessment measures” defines the question as how much a measure explains or predicts a phenomenon relative to other measures, and notes that the result varies with the criterion, comparison measures, and people in the sample. The scope can be followed through three steps: responses produce scores; scores are interpreted as evidence about a construct or pattern; and that interpretation is used for a purpose. Incremental validity addresses a limited question within that chain: did adding this measure contribute information beyond a specified alternative for a specified target? It does not, by itself, establish that the score measures what its label suggests, that an interpretation fits every person, or that a proposed action is justified. This is why a claim about a group-level prediction must not be translated into certainty about an individual. A model may improve its predictions across a study sample and still leave substantial uncertainty about any one person. The conclusion needs to name the population and outcome studied, and it should preserve the difference between an average pattern and a personal forecast. The U.S. Office of Personnel Management’s “Designing an Assessment Strategy” discusses added predictive value in a workplace selection context, where assessment tools are considered in relation to job performance. Evidence about conscientiousness and job performance illustrates the boundary. “The criterion-related validity of conscientiousness in personnel selection: A meta-analytic reality check” combined 102 studies and 23,305 participants, reporting an average correlation of .17. The authors also found that only about 12% of studies used real applicants in predictive designs, and called for more multivariate research. This is an average association for a trait and performance outcome, not a test of whether one named report adds value beyond a named baseline. It therefore cannot substantiate a report-wide promise, much less a conclusion about one reader’s future. A careful report separates what the evidence showed from the interpretation drawn from it and from any advice offered. A hypothesis about how a tendency may appear at work can prompt reflection; it is not the same as an empirically demonstrated prediction. A bounded claim should say what measure added information, beyond which baseline, for which population and outcome, and in what intended use. If a provider cannot specify those limits, ask for narrower wording and the underlying study before relying on the conclusion.

Sources: Incremental validity of new clinical assessment measures; Designing an Assessment Strategy; The Criterion-Related Validity of Conscientiousness in Personnel Selection: A Meta-Analytic Reality Check

Should a personality report claim incremental validity?

Yes, a personality report may claim incremental validity when evidence shows that a specified score adds information beyond a named baseline for a defined outcome and population. The claim should identify the comparison, method, size and uncertainty of the gain, and why it matters for the stated use. Without those details, describe the evidence more narrowly. The phrase alone is not a certificate of accuracy or insight into an individual.

First ask whether the comparison is interpretable: is the measure identified, is the baseline explicit, is the criterion clear, and does the sample match the people and setting named? “Incremental Validity: A Review and an Evaluation of the Literature” describes incremental validity as dependent on the criterion, comparison measures, and sample, and distinguishes evidence from cost-benefit judgment. A result against one baseline answers a particular question. It does not establish added value after every other source of information or for a different outcome.

Next ask whether the gain is credible for the conclusion. For a prediction claim, readers should be able to tell whether evaluation used data separate from the data used to build the model, and should see the gain and uncertainty. “Improving Predictive Inference about Psychological Constructs: A Comparison of Three Approaches for Developing Prediction Models” compares within-sample and out-of-sample approaches using simulations and an empirical application; its simulations found within-sample assessment overestimated out-of-sample assessment in small samples. This supports caution about development-sample gains. It does not provide a universal adjustment or show that every held-out estimate transfers to a new population. Then ask what decision changes and whether the gain is worth added time, cost, privacy burden, or error risk. The review identifies information cost, importance of the phenomenon, and utility as distinct considerations. A significance label cannot settle this judgment.

There is a fair exception to skepticism. A narrowly designed measure could have credible incremental evidence for a particular outcome, baseline, population, and use. That justifies the bounded claim without supporting a broad description of the person. Conversely, absence of an incremental-validity study does not show that a score cannot prompt useful self-reflection; it means the report has not established added predictive value beyond the specified baseline. The U.S. Office of Personnel Management’s “Assessment Strategy” guidance frames workplace increments relative to the existing assessment set, but cannot validate a particular instrument or employer’s use.

The verdict: claim only the increment the comparison supports, and state its limits beside it. If a report cannot name the measure, baseline, outcome, population, validation design, gain, and uncertainty, treat the statement as unsubstantiated or too broad, not as proof the measure has no possible value. For a personal work-pattern question, the live Work Pattern Report can be a low-stakes prompt for noticing tendencies and turning vague friction into observations. It supplies no norms, selection score, or validated employment prediction. Use it at /assessment.

Sources: Incremental validity of new clinical assessment measures; An out-of-sample perspective on the assessment of incremental predictive validity; Designing an Assessment Strategy

What should you ask before relying on the claim?

Ask the provider to complete one sentence: ‘This version of the measure added [specified information] beyond [named baseline] for [outcome] in [population], using [validation design], with [estimated gain] and [uncertainty].’ If it cannot name those parts, the claim is too broad to guide your decision. Missing details do not show the score has no possible value; the comparison is simply unclear. Request the technical source. Check that it evaluates the instrument version, sample, criterion, and intended use described in the report. “Incremental Validity of New Clinical Assessment Measures” explains that results vary with the criterion, comparison measure, and sample, and that added information must be weighed against cost and utility. The 2024 paper “An Out-of-Sample Perspective on the Assessment of Incremental Predictive Validity” reports differences between within-sample and out-of-sample estimates in simulations, especially in small samples. Neither source verifies a particular report. Until those details are available, do not treat the claim as a decisive judgment about you or your employment prospects. A transparent, replicated comparison in a relevant population, with useful added performance, would strengthen it; transfer to another setting may remain uncertain. For personal reflection, the live Work Pattern Report at /assessment offers prompts about observable work tendencies. It is not a validation demonstration or job recommendation. Browse /topics for report-literacy guidance.

Sources: Incremental validity of new clinical assessment measures; An out-of-sample perspective on the assessment of incremental predictive validity

Questions readers ask