Why Can Personality Assessment Look More Predictive in a Restricted Sample?

Start with the claim behind ‘more predictive’

A report says a personality score predicted later performance among current employees. You wonder whether that finding applies to a broader group. First ask what “predicted” means. It usually describes an association between a score and outcome in a particular sample. It does not mean the score caused performance or gives an individual probability.

A restricted-group result can look stronger for several reasons: it may be an adjusted estimate for a less restricted population, an observed association in a selected subgroup, or a finding tied to one outcome. These are not interchangeable. Range restriction often reduces an observed correlation by compressing score variation, but this is not guaranteed under every selection design. “Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors” explains why selection paths and correction assumptions matter. The APA Guidelines on Psychological Assessment and Evaluation tie validity to interpretation, setting, and purpose. Compare the number, who was studied, how they entered the sample, and the outcome before applying the finding to your case. A larger coefficient alone does not establish broader usefulness.

Sources: Guidelines for Psychological Assessment and Evaluation; Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors

What does ‘restricted range’ remove from a prediction?

Range restriction means that a study includes less spread on a score or outcome than the population about which someone hopes to draw a conclusion. For example, employees may have more similar assessment scores than the wider applicant pool. A small sample can contain varied scores, while a large one can remain tightly clustered. What matters is how much the measured values vary and which values are underrepresented. That distinction matters because Pearson’s correlation, often written as r, describes how two variables vary together relative to their own spread. The Statistics LibreTexts chapter “13.3: Correlation Considerations” explains that Pearson’s r uses covariance and the standard deviations of both variables. If one variable has less spread, that can change the observed correlation even if the broader pattern has not changed. The chapter illustrates this by taking a scatterplot with a positive relationship, removing scores below a cutoff on one axis, and showing the correlation fall from .67 to .38. Those numbers belong to the chapter’s teaching illustration, not to a personality test study. This illustrates one mechanism, not its size in a particular assessment or work setting. In ordinary language, imagine looking only at people whose assessment scores fall within a narrow band. There is less room to see whether higher scores tend to accompany higher or lower values on a later outcome. The relationship may look weaker because much of the score variation that could reveal it is absent. That explanation depends on how the band formed and what was measured. Restricting the outcome instead, selecting people through another measure, or studying a group with a different pattern can change the comparison in other ways. Range restriction is therefore a description of a sample’s limited spread, not a complete explanation of any coefficient. The Cambridge article “Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors” defines the relevant comparison in personnel selection as a study sample that is narrower than the applicant pool for which someone proposes to use a predictor. It also stresses that a correction requires credible information about spread in both groups. A report’s sample size alone cannot show whether its validation group represents the target population. A reader should look for score distributions or standard deviations for the studied group and the population the claim concerns, and how participants entered the study. If that comparison is absent, the degree of restriction remains unknown; the coefficient should not be treated as a direct forecast for a broader group.

Sources: 13.3: Correlation Considerations; Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors

Why does restriction often weaken an observed relationship?

If a selection rule removes people at the low or high end of a personality score, those left have fewer score differences to compare with an outcome such as a later performance rating. In a simple, roughly linear relationship, this compression often makes the observed correlation smaller. This weakening is called attenuation: a reduced association partly because the sample covers less of a variable’s range. It describes a pattern in a group, not how accurately a report describes one person.

The reason is visible in what a correlation summarizes. Pearson’s correlation reflects how two measures vary together, relative to how much each varies on its own. Imagine, purely as an illustration, a broad group whose assessment scores and later outcomes tend to rise together. If a study keeps only people whose scores fall in a middle-to-high band, many original score contrasts disappear. Outcome values still differ, but there is less score variation to line up with them. The association can look weaker even though the broad pattern has not simply vanished. This explains the calculation; it is not a claim about a particular test or workforce.

The chapter “13.3: Correlation Considerations” illustrates the point with a teaching scatterplot: after removing the lower half of X scores, the example’s correlation changes from .67 to .38. Those values describe that illustration only. The chapter explains that Pearson’s r uses covariance and the standard deviations of both variables, and cautions that correction can be complex and require unavailable information. The example supports the mechanism, not a universal amount of attenuation for personality assessments.

Restriction can involve the assessment score or the outcome. Selection based directly on an assessment may leave a narrower score spread among those studied. A sample could also include only people above a performance threshold. These situations do not automatically produce the same change: the correlation depends on variation in both measures and on how inclusion was determined. Truncating one end of a score distribution also differs from merely recruiting fewer people.

In “Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors,” the authors discuss how selection designs affect validity estimates and why corrections depend on the selection process. This personnel-selection methods discussion reinforces the need to know what was restricted; it is not a personality-inventory finding. If a report calls an incumbent group representative but gives no score spread or selection history, a reader cannot judge how much of the association is visible. Attenuation is a reason to inspect the sample, not a correction formula or proof that the assessment has no useful signal.

Sources: 13.3: Correlation Considerations; Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors

How can the selection path change what the sample shows?

The effect depends on how people entered the study. Direct restriction occurs when the assessment being evaluated helped determine who was selected, so the studied group has already been filtered on the focal score. Indirect restriction occurs when another measure helped select people and is related to the personality score. The filtering can still change the observed relationship, even though the assessment itself was not used as the gate. ‘Current employees’ describes who was studied; it does not tell you which route brought them there.

Consider an ordinary selection path: applicants complete several measures, a combined score helps determine who receives an offer, and a later study examines only those hired. That hired group is not simply a smaller copy of all applicants. Its members passed a rule based on the measures, and the rule may have favored different combinations of scores. A person with a lower result on one measure might have needed a higher result on another to reach the same combined score. Looking only among those who passed can therefore alter how the measures relate to one another. The relevant question is not just whether the sample is narrow, but what was narrowed and by which rule.

The 2007 article ‘A Cautionary Note on the Effects of Range Restriction on Predictor Intercorrelations’ examines this issue with a simulation, an applied example, and a reanalysis concerning ability and interview correlations. It reports that when both predictors contribute to the operational selection composite, their intercorrelation can differ substantially from its population value. That finding concerns relationships among predictors under composite selection. It does not directly establish that every personality score’s association with job performance will change in the same way. It does show why a correction based on a simpler selection story may not describe a more complex path.

The 2023 article ‘Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors’ treats selection design as consequential, while cautioning against assuming a large effect in every case. It notes that some concurrent-study situations may involve little restriction effect, depending on how the sample and selection process relate to the predictor. An employee sample is neither automatically unusable nor equivalent to an applicant population. Its history matters. Ask which criteria determined entry, whether the personality score related to those criteria, and whether the reported adjustment represents that process. If details are missing, the estimate may describe the studied group, but its extension to another group is uncertain. The sample label alone cannot resolve that uncertainty.

Sources: Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors; A Cautionary Note on the Effects of Range Restriction on Predictor Intercorrelations

How can a restricted sample look more predictive?

A restricted sample can look more predictive in at least three different senses, and they should not be treated as the same finding. The first is an observed association among the people who were actually studied. The second is a corrected estimate intended to describe a broader, less restricted population. The third is a claim that either number will carry over to a different group or decision. Consider a study of people already employed in a role. Its observed coefficient describes the association between a defined personality score and a defined work outcome among those employees, under that study’s conditions. If a correction is applied, the resulting coefficient is a model-based estimate for a target population. It may be larger because the method attempts to account for variation in scores missing from the selected group. That increase does not mean the assessment became more accurate among the employees who supplied the data. Nor is the corrected value an additional result directly observed in a broader sample. This distinction matters because correction is not a mechanical repair. “Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors” describes revisions to correction practices in personnel-selection research, notes that revised estimates can differ substantially from earlier ones, and recommends attention to study design and variability. The target, selection process, and information used in the adjustment all matter. A second possibility is that the restricted subgroup or outcome has a different relationship from the one a reader assumes. That is a question to investigate, not a default explanation. “Increased Accuracy for Range Restriction Corrections: Implications for the Role of Personality and General Mental Ability in Job and Training Performance” reports that a revised correction procedure changed estimates for specified personality measures and general cognitive ability by different amounts in its meta-analytic analysis. This shows correction choices can alter comparisons, but does not establish a universal ranking or result for a modern assessment. Finally, two coefficients may look comparable while referring to different scores, criteria, groups, or methods. The Statistics LibreTexts chapter “Correlation Considerations” uses a teaching example in which truncating part of a score range reduces an observed correlation. The example illustrates why restriction can hide an association, not a universal rule for all designs. So ask: higher than which number, for which target population, and under what correction or comparison? If the report cannot identify those details, “more predictive” remains unresolved. A corrected estimate may inform a bounded population claim, but does not by itself establish transfer to a new use.

Sources: Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors; Increased Accuracy for Range Restriction Corrections: Implications for the Role of Personality and General Mental Ability in Job and Training Performance; 13.3: Correlation Considerations

Could the apparent improvement come from a different outcome or group?

Yes. A relationship can differ because the group, criterion, or measurement changed, even when range restriction is not the reason. A personality score associated with one narrowly defined training outcome does not thereby show the same association with broad job performance, another role, or another group. Before calling one result stronger, check whether the studies asked the same question. If several features changed, the larger coefficient cannot identify which difference mattered.

A criterion is the outcome used to evaluate the score: for example, a defined task result, a supervisor rating, or a broader performance measure. These are not interchangeable labels for one fact. They capture different behaviors, periods, and sources of error. Even “performance” needs detail: what was rated, by whom, and when? An association with a specific training result may be useful for that question. It leaves open whether the score relates similarly to broader work or another setting. That is a change in the claim, not a statistical correction.

Measurement quality raises a separate issue. Criterion unreliability means the outcome measure contains imprecision; it can make an observed association harder to interpret. Range restriction concerns limited variation in the sample. Correcting for one does not automatically address the other, and neither correction substitutes for matching the criterion to the decision. In “Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors,” the authors discuss these as distinct correction problems. They explain that sample-specific information is often unavailable in meta-analyses, so applying factors drawn from the available subset requires care. This is a methods discussion about personnel selection broadly, not a finding about a particular personality assessment.

Group composition matters too. A finding among people in one role, organization, or stage of selection may not describe people elsewhere. The APA’s “Guidelines for Psychological Assessment and Evaluation” says interpretation should account for setting, purpose, and population, and that validity evidence may need evaluation when these differ from the studied context. This does not mean every new use requires an identical new study; it means a mismatch should be considered and reported rather than hidden by a single coefficient.

For a report reader, compare the exact score and version, group studied, criterion and timing, and proposed use. If estimates differ on several dimensions, do not attribute the gap to restriction alone or treat the higher number as stronger evidence for your situation. First establish that both estimates concern a comparable score–outcome relationship. Then ask whether the population and decision match the one at hand.

Sources: Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors; Guidelines for Psychological Assessment and Evaluation

Why can an adjusted number look larger—and what does that number mean?

A corrected estimate answers a different question from the observed association in the selected group. It uses data and assumptions about selection or measurement to estimate what a relationship might look like in a broader target population. Because the selected group may contain less variation in assessment scores, a correction can raise the estimated association. That larger value is not a new finding among the people studied. It is a model-based estimate under conditions.

An observed coefficient describes the measured association in the sample as collected. An adjusted coefficient attempts to account for a feature that may obscure the relationship, such as restricted score spread or an imperfectly measured outcome. A third claim—that the result applies to another population, assessment, criterion, or decision—requires evidence about that transfer. Moving from the first number to the second involves assumptions; moving from either to the third involves a further inference. “Corrected” does not make those steps disappear.

The 2023 article “Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors” describes how correction choices can change estimates in personnel-selection research. It reports that the 2022 reanalysis it discusses identified problems in earlier range-restriction corrections and produced revised operational-validity estimates that often differed from prior estimates. The article notes that the reanalysis presented observed and corrected values side by side and reported between-study variability and averages. Its authors recommend attention to study-specific conditions when correcting estimates. This is a methods discussion, not validation of every personality report or proof that every earlier adjustment is wrong.

An example shows why correction can change comparisons as well as absolute values. In “Increased accuracy for range restriction corrections” (2008), a meta-analytic study applied a revised procedure to conscientiousness, emotional stability, and general cognitive ability in job and training performance research. Its accessible abstract reports that estimated validity increased more for cognitive ability than for the two personality measures, reducing personality’s relative validity in that analysis. This concerns those constructs and selection-research literature. It cannot establish a ranking for an unnamed modern assessment or an individual reader.

A larger adjusted estimate can coexist with weaker grounds for applying it. The correction may target a population beyond the observed sample, while its assumptions may fit that target imperfectly. Ask which value was observed, which adjustment was made, what selection and score-spread information it used, and what population the result represents. Look for uncertainty and variation across settings, not only an average. Read observed and adjusted values together; treat the latter as conditional evidence, not a universal property of the assessment.

Sources: Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors; Increased Accuracy for Range Restriction Corrections: Implications for the Role of Personality and General Mental Ability in Job and Training Performance

What should a reader compare before accepting the word ‘predictive’?

Before accepting “predictive,” identify the personality score and version, the people studied, the population the claim concerns, the outcome, and when it was measured. Check whether the coefficient is observed in the sample or adjusted for selection. A result may describe a selected group, estimate a less restricted population under assumptions, or support a new decision. These are distinct claims; one does not automatically establish the next. The APA’s *Guidelines for Psychological Assessment and Evaluation* explain that validity concerns an interpretation and use in context, rather than being a permanent property of an instrument. A study of one group and outcome may not resolve whether the same score supports a different population, role, or decision. A larger sample can make an estimate more precise for the group studied without showing that it applies elsewhere. Reliability concerns the consistency or precision of scores under specified conditions. A measure can yield consistent scores while its relationship with an outcome is weak or irrelevant to another use. Measurement noise may affect an association, but differs from range restriction, which concerns reduced variation in the studied group. A reliability figure alone does not show that validity evidence transfers. The APA guidelines advise considering technical manuals, standardization samples, and validity and reliability evidence in relation to intended use. Keep three results distinct: | Result | Describes | Does not establish alone | |---|---|---| | Observed association | Relationship in the selected sample | Relationship in another population | | Corrected estimate | Model-based target estimate, conditional on inputs | New observation or universal property | | Matched-use evidence | Support for a defined score, group, setting, and outcome | Individual forecast or every other use | *Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors*, in *Industrial and Organizational Psychology*, explains why selection path matters: effects depend on variables and design, and corrections need study-specific information. The article concerns selection methods broadly, not every personality instrument. It supports a question, not a coefficient to transfer: who entered the sample, by what rule, and does the correction represent that process? The practical standard is a matched claim, not an impressive number. For self-reflection, a relevant finding can frame a tendency as a question to check against observed behavior. For a consequential employment decision, evidence must fit the score, population, outcome, and use. A general personality coefficient is not an individual verdict. Without those details, portability remains uncertain.

Sources: Guidelines for Psychological Assessment and Evaluation; Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors; APA Ethical Principles of Psychologists and Code of Conduct

How would you read a report that says the score predicted performance?

Read “the score predicted performance” as a bounded claim until the report shows what that sentence covers. Consider a hypothetical report, offered only as an illustration: it relates a personality score to supervisor ratings among current employees, then suggests using the finding for applicants to another role. The result describes an association in that group; it does not establish prediction for the different group or an individual forecast.

Start by identifying the predictor precisely. Which assessment and version was used? Was the reported variable a total score or a particular scale? Then identify the outcome: what did “performance” mean in this report, who rated it, and when? A supervisor rating at one point is not interchangeable with a different performance measure or time period. A changed score or outcome changes the claim being evaluated.

Next, trace how the participants entered the study. Were they applicants assessed before selection, or employees already chosen through interviews, experience, other tests, or several sources of information? The accessible article “Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors” explains that range restriction depends on how the validation group was selected and how the studied predictor relates to the actual selection basis. It also cautions that applying correction factors from predictive studies to concurrent studies of current employees can overcorrect; the authors describe this as a methodological risk, not a finding about every personality measure. An employee sample is not automatically unsuitable, but its selection history matters.

Now separate the number actually observed from any adjusted estimate. If the report gives both, ask what population the adjustment aims to represent, what information about applicant and employee score distributions or selection was available, and what assumptions the method requires. A corrected estimate is an inference from data and a model, not a second observed result. Without the method, you cannot judge whether the adjustment fits.

Finally, match the conclusion to the proposed use. The “APA Guidelines for Psychological Assessment and Evaluation” state that validity evidence is evaluated in the context of settings and purposes, and call for attention to population and context. A narrow conclusion might be that a particular score was associated with a specified rating among a defined employee group. The broader claim that it predicts performance for new applicants, another role, or an individual requires evidence for that interpretation and use. A score describes a tendency, not a verdict about someone’s likely conduct or worth. A focused question for the report’s provider or decision-maker is: “What population does this estimate represent, and what evidence supports using it for this outcome here?”

Sources: Guidelines for Psychological Assessment and Evaluation; Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors

What should you ask or do next?

When a report says a personality score predicted performance, ask: for whom, measured how, and in what setting? Match the claim’s population to the people and decision you care about. Identify the score and outcome. The APA Guidelines for Psychological Assessment and Evaluation explain that validity concerns a particular interpretation and use; evidence about one group does not automatically settle another question. Next, trace how people entered the study. Were they selected employees? Did selection use the personality measure, another related measure, or both? If the estimate is adjusted, ask what method and selection information it uses, and which population it describes. “Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors” explains why corrections depend on study design. An adjusted value estimates beyond the observed sample under assumptions; it is not a new observation of that broader group. Finally, choose a proportionate use. If group, score, outcome and selection path fit your question, treat the finding as one reason to take the claim seriously. If details are missing or differ, keep the result uncertain rather than making it a personal verdict. For an individual career or collaboration question, note a recent decision or feedback exchange and consider what pattern it suggests. The Work Pattern Report at /assessment can structure low-stakes reflection. It is an unvalidated self-report without norms, cutoffs or a selection score, so it cannot predict work performance. Restriction often hides variation; correction can estimate beyond the observed group. Neither travels without a defensible match to the target population and decision. Selection-path evidence could change confidence.

Sources: Guidelines for Psychological Assessment and Evaluation; Revisiting the Design of Selection Systems in Light of New Findings Regarding the Validity of Widely Used Predictors