When Does a Personality Report Need a Separate Comparison Group?

When does a personality report need a separate comparison group?

A personality report needs a separate comparison group when it makes a relative claim about a population its current reference data do not adequately represent. A change in purpose does not automatically call for new norms; it calls for evidence that the score supports the new interpretation. Ask two questions: who is the score compared with, and what conclusion or decision is it meant to support? A norm is a reference distribution for relative standing. A percentile shows where a score falls within a distribution; it is not a percentage of a personality trait. The NCME Glossary defines percentile rank in relation to scores in a distribution, so the reference population matters. Validity evidence asks whether an interpretation is supported for a particular use. The ITC Guidelines on Test Use place interpretation in context, including the reason and setting for testing. Check whether the report’s reference group fits the population named or implied by its relative claim. If that comparison is used for a new decision, ask for evidence tied to that interpretation. Both questions may need attention, or only one. Without the report’s manual and details about its reference group and intended use, adequacy cannot be determined.

Sources: ITC Guidelines on Test Use; NCME Assessment Glossary

What does a comparison group change in the report?

It changes the reference distribution against which the same score is described. A percentile means a position within a specified distribution; it is not a percentage of a trait, a grade, or a prediction about what the person will do. The National Council on Measurement in Education (NCME) defines a percentile rank as the percentage of scores in a specified score distribution below a given score. That definition makes the comparison group part of the meaning: a percentile without its reference distribution is an incomplete relative claim. Consider an illustrative report with a fixed raw score for a work-style tendency. Compared with a broad adult reference and then a narrower group, its relative position could differ because the distributions differ. The guide “What to Look for in a Personality Report’s Technical Manual” makes the same practical distinction: different publishers may compare a score with broad adult or narrower user groups, and the relevant question is whether the selected group fits the interpretation being made. A norm-referenced interpretation answers a relative question: where does this score fall within a specified reference population’s score distribution? The NCME glossary defines norms as statistics or tabular data summarizing the distribution of scores for one or more specified groups, usually intended to represent a larger reference population. “Higher than most” or percentile wording therefore needs an identified distribution. A percentile does not mean a person possesses a trait to that percentage, answered that share of items in a particular way, or will act in a matching proportion of situations. It does not establish whether the trait is beneficial. This relative comparison differs from a criterion-based interpretation. The NCME glossary describes criterion-referenced interpretation as relating performance to a defined criterion domain, including a cut score. A norm asks how a score compares with scores in a group; a criterion asks how it relates to a defined standard. Calling a result “above average” implies a comparison with a distribution. The label attached to a comparison also needs scrutiny. A representative population norm aims to describe a specified wider population. A local norm refers to a limited reference population, such as one organization or institution; NCME explicitly says local norms are not intended to represent people beyond that setting. A self-selected user group is different again: people who chose to take an online assessment may be a useful description of those users, but participation alone does not establish that they represent a broader population. Keep the score type and comparison separate: “What score was calculated?” and “Which distribution or criterion gives it this label?” The NCME definitions support that distinction, while the technical-manual guide advises checking the norm group, edition or date, and whether the report uses national, local, or user norms. If a report gives a percentile without identifying the distribution, its relative ranking is unresolved. That weakens the rank claim; it does not necessarily erase the score’s possible use as a prompt for cautious self-reflection, provided the report explains what the underlying measure covers. for the reader. A comparison table is not a second measurement of the person; it summarizes other scores used as a reference. Changing that reference can therefore change the percentile label without changing the responses. A report should make this distinction visible wherever it presents relative standing, so readers can distinguish a score description from the basis of its ranking.

Sources: NCME Assessment Glossary; Norm-referenced test; What to Look for in a Personality Report’s Technical Manual

When is a separate group justified by the population?

A separate comparison group is justified when a report makes a relative claim about a target population that its existing reference data do not adequately represent. A different demographic or occupational label alone does not show that a separate norm improves the report.

Start by identifying the claim. Is the report comparing someone with adults generally, people in a particular setting, or a defined occupational or language group? Then ask what supports that comparison. The National Council on Measurement in Education’s (NCME) glossary defines a reference population as the population represented by test norms, permitting estimation of its score distribution. It defines norm-referenced interpretation as comparison with a specified reference population. If the report does not name the population, the reader cannot tell what relative claim the percentile makes.

Next, inspect how the reference was assembled. A useful account identifies who was included, how participants were recruited, when data were collected, which instrument version and language they completed, and the administration conditions. Check whether recruitment covered the target population, and whether the dates, language, version, and administration conditions fit the people described. A sample description should make clear who could be represented and who may be missing. Check whether the report describes an intended population or only people who chose to take that assessment. NCME calls statistics from a self-selected or otherwise ill-defined group “user norms.” They describe that group; it does not follow that they represent a larger population.

Local data raise a related issue. NCME defines local norms as comparisons for a limited reference population, such as one organization or institution, and says they are not intended to represent populations beyond that setting. If an organization wants to describe patterns among its own participants, local norms may answer that narrow descriptive question. The reference’s scope should match the report’s claim.

A narrower group can make a comparison more pertinent, but narrower is not automatically better. Specificity helps only if the sample adequately supports the claim. A group assembled from a small, selected, outdated, or mismatched set of respondents may represent the target less well than a broader reference. Nor should a publisher create a separate norm simply because an observed group average differs: the difference could reflect who was sampled, administration, language, or score interpretation. Evidence needs to explain why this group is the right comparison and what limitations remain.

The review “A Comparative Review of Current Practices in Personality Assessment Norming” shows why the issue deserves attention but also calls for restraint. Its accessible abstract reports a review of literature and guidelines and an examination of norming practices across 30 personality assessments offered in the United States, United Kingdom, and Canada. The authors describe accurate, representative norms as consequential when decisions compare people with a norm group, and say existing standards give less specific guidance about norms than selection procedures. It supports asking for the rationale and evidence rather than treating a group label as sufficient.

The International Test Commission’s “ITC Guidelines on Test Use” warns against conclusions drawn from comparisons with norms that are irrelevant to the people tested or outdated. It says users should understand the characteristics and limitations of comparison groups, use appropriate groups where available, and consider relevant validity evidence for demographic groups. Splitting a limited sample can produce a narrower-looking comparison without making it more stable or representative.

Ask the publisher to identify the reference population, explain how and when its sample was recruited, and specify the instrument version, language, and conditions represented. Compare those facts with the group named or implied in the report. If the publisher cannot connect its actual reference data to that claim, qualify or omit the relative ranking.

Sources: A Comparative Review of Current Practices in Personality Assessment Norming; ITC Guidelines on Test Use; NCME Assessment Glossary

Do subgroup differences always call for separate norms?

No. A score difference between groups is a reason to examine interpretation and fairness, not proof that each group needs its own norms. Before splitting a reference group, ask whether the assessment measures a meaningful, comparable construct across those groups and whether language, access, or testing conditions could have shaped scores. A group label alone does not answer those questions. The norm describes relative position; fairness concerns whether that comparison is appropriate and interpretable for the people assessed.

A related question is measurement invariance: whether an assessment’s measurement structure operates comparably across groups. This is not the same as asking whether group averages are equal. If the intended interpretation assumes comparable score meaning, evidence for that assumption matters before treating a score difference as a difference in the underlying tendency. A report should not infer that every individual in one group shares its average. Group-level patterns, where supported, do not determine individual behavior.

There is a reasonable case for keeping a broad norm. If the report intends to describe standing relative to a broad population, and the instrument’s meaning and administration are sufficiently comparable for that purpose, one common reference can preserve a shared scale. Creating many narrow groups can fragment the comparison: each may have fewer observations, while a local sample may represent only people who chose or were able to take the assessment. The percentile can look tailored while resting on a weaker or less representative reference. A narrower label does not guarantee a better norm. Comparable measurement does not require identical averages; groups may differ while the score retains a similar meaning. Conversely, matching averages alone would not establish that each item or the full scale functions equivalently.

The counterargument is that a pooled reference can answer the wrong question. If a report explicitly describes relative standing within a particular population, a broad norm may blur that intended comparison. A defensible subgroup norm could make it clearer, provided its sample represents the named group, the test version and conditions fit, and the interpretation is meaningful. “Separate” should follow the intended claim and evidence, not a reflex to create a table for every observed difference.

The 2015 review *A Comparative Review of Current Practices in Personality Assessment Norming* examined norming practices across 30 personality assessments in the United States, United Kingdom, and Canada. Its accessible abstract describes norms as consequential and says existing standards provide less specific direction about norming than selection procedures. The review supports careful scrutiny of how norms are built; it does not show subgroup norms always improve interpretation or provide a universal threshold for splitting a sample. Its abstract gives no result for individual assessments. The practical lesson is to explain the choice and its limits rather than imply that a demographic label settles the issue.

Look for a reasoned account of the comparison. Does the report identify a broad population norm, subgroup norm, or local reference? Who was included, and does that sample match the group named? Were language versions, access, and administration considered? What evidence supports comparable interpretation and the intended use? The ITC guidelines ask test users to examine norm-group appropriateness, relevance to intended use, fairness across tested groups, and possible language and testing-condition effects. If the publisher cannot answer, the relative ranking is uncertain; that does not erase every descriptive observation in the report.

The verdict is not “always split” or “never split.” Separate norms can clarify an intended within-group comparison when the target is distinct and the reference is adequate. Pooled norms can fit a broad claim when evidence supports comparable interpretation. Where evidence about construct meaning, sample quality, language, or administration is insufficient, neither a pooled nor narrower percentile deserves confident interpretation. The publisher should explain why it chose pooled, subgroup, local, or no norm, and what evidence bears on that choice. Readers should treat a group comparison as context for reflection, never as a diagnosis or prediction about an individual.

Sources: ITC Guidelines on Test Use; Guidelines for Psychological Assessment and Evaluation; A Comparative Review of Current Practices in Personality Assessment Norming

Why are a norm sample and a validation sample not interchangeable?

A norm sample and a validation sample answer different questions. The norm sample supplies a reference distribution: it shows how scores were distributed in a defined group, so a reader can interpret a score relative to that group. Validation evidence asks whether a proposed interpretation of scores is supported for a particular use. The National Council on Measurement in Education (NCME) Assessment Glossary defines norm-referenced interpretation by comparison with a specified population’s score distribution, and defines validation as investigating the proposed interpretation for its intended uses. A percentile table can therefore support a relative statement, but it cannot by itself establish that the score supports a new conclusion or decision. Consider one fixed personality questionnaire and one fixed set of responses. Suppose its technical report provides a percentile based on a broad adult reference group. For private reflection, the report might use that comparison to prompt a question such as whether the person tends to prefer more structure than members of the reference group. The percentile has a defined descriptive role, though it does not prove why the person responded that way or determine how they will act. If the same score is later used to select applicants for a particular role, the broad-group percentile does not change: the person’s responses and reference distribution are still the same. What changes is the claim being made. The decision-maker is no longer merely describing relative standing; they are using the score to infer something relevant to a consequential outcome. That inference needs evidence appropriate to that use and population. The International Test Commission’s Guidelines on Test Use make this distinction operational. They say users should check whether technical documentation supports validity and relevance for the required use and relevant populations, and recommend using tests only for purposes where relevant validity evidence is available. They also say that when a test’s purpose changes, its validity evidence should be reevaluated. Norm adequacy and use validity are related parts of responsible interpretation, but they do not substitute for one another. The distinction also runs in the other direction. A publisher could replace the broad adult reference distribution with a narrower occupational or organizational reference. That change could move a person’s percentile because the comparison distribution has changed, even though the person’s answers have not. A higher or lower relative standing under the narrower group would not, on its own, show that the score predicts a work outcome better, supports a coaching conclusion, or improves a decision. The NCME glossary’s definition of validity is explicitly about accumulated evidence and theory supporting a specific interpretation for a given use. If a report intends multiple interpretations for different uses, NCME says validity evidence is needed for each interpretation. The two roles do not require two wholly separate sets of participants. A study may describe a score distribution and also test a proposed interpretation, provided its design and analyses support both claims. A validation sample does not automatically supply a representative norm, just as a norm sample does not automatically test whether a score interpretation works for a proposed purpose. The APA’s Standards information page identifies the American Educational Research Association, APA, and NCME as the sponsoring organizations for the Standards for Educational and Psychological Testing. a particular report’s technical manual is still needed to inspect its samples and analyses. For a reader, the practical request can be precise: “Which group produced this comparison, and what evidence supports using this score for the conclusion or decision being proposed?” If the reference group is named but evidence for a new use is absent, the relative description may remain interpretable while the new inference is unresolved. If the reference is unclear, the percentile’s meaning remains uncertain even when use evidence exists. When both the population comparison and the purpose change, ask for documentation addressing both. This avoids the common category error of treating new norms as a substitute for evidence about what a score means in practice.

Sources: NCME Assessment Glossary; Standards for Educational and Psychological Testing; ITC Guidelines on Test Use

What changes when the report is used for employment selection?

Selection changes the question from “How does this person compare with a reference group?” to “Does this score support a consequential judgment about this applicant for this job?” A job-related norm may describe applicant standing, but it does not establish that a score predicts performance, adds useful information, treats applicants fairly, or is appropriate under the rules that apply. For private reflection, a person might use a report to name a tendency and consider whether it fits examples from their own work. In coaching or development, a facilitator might use the same language to structure a conversation about collaboration or planning. In selection, the score is used to compare applicants or influence a hiring decision. That shift raises a new question about the link between the measured score and the particular job outcome being inferred. The U.S. Office of Personnel Management’s “Designing an Assessment Strategy” explains this distinction with a hypothetical use: if a personality test is intended to forecast success in a customer-service position, predictive evidence may be needed to show that test scores relate to subsequent job performance. This is guidance about what evidence would fit that claim, not a reported finding that any personality test predicts customer-service success. Comparing applicants with people in customer service might make their relative standing more relevant to that group, but the comparison alone does not show that higher or lower standing corresponds to later performance, or that using the result improves the decision. The same point holds if the applicant group is well represented in the norm sample. That sample can help answer a descriptive question: where does this applicant’s score fall relative to scores in the reference distribution? It cannot, by itself, answer a predictive question about later work, an incremental question about whether the score contributes useful information beyond other evidence, or a fairness question about who is screened out. OPM’s assessment-strategy guidance treats validity evidence as dependent on how an assessment method is used in an employment decision. It also distinguishes reliability, or score consistency, from validity; consistent scores do not alone establish a useful job inference. A publisher’s norm table and evidence supporting a selection decision therefore have different jobs, even if some underlying data contribute to both. For U.S. employment use, the Equal Employment Opportunity Commission’s “Employment Tests and Selection Procedures” states that tests should be validated for the positions and purposes for which they are used, and that results should suit the employer’s purpose. Vendor documentation may help, while responsibility for ensuring validity under the Uniform Guidelines on Employee Selection Procedures remains with the employer. The guidance addresses possible disparate impact: if a selection procedure screens out a protected group, employers should consider whether an equally effective alternative has less adverse impact. This is U.S.-specific guidance, not a summary of rules in every country. They illustrate why a technically appropriate norm cannot settle all questions about fair or lawful use. A reader reviewing an employment-related report can ask what exact job and decision the evidence covers, what outcome the score is claimed to predict, and whether the technical documentation supports that inference for the relevant population and version of the assessment. A percentile or a norm group named for an occupation does not show that a person is suited to that job. The conclusion should remain unresolved when the evidence describes only relative standing but the decision requires a job-performance claim. This distinction also limits what a self-reflection report can responsibly offer. The live Work Pattern Report is a low-stakes self-report across ten decision-and-collaboration continuums. It supplies no norm, cutoff, or selection score, and it has not been validated for hiring or role recommendation. It can help someone prepare questions about recurring work friction and compare impressions with concrete experiences. It cannot provide an employment ranking or establish whether someone should be hired. When an assessment is being used to make a selection decision, ask for job- and purpose-relevant evidence rather than treating a relative score as proof of suitability.

Sources: Designing an Assessment Strategy; Are we allowed to use personality tests to assess candidates?; Employment Tests and Selection Procedures

How can I decide whether the report needs new norms, new use evidence, or both?

Use two questions: has the target population for the relative comparison changed, and has the score interpretation or decision changed? A supported yes to the first may require different norms; a supported yes to the second requires evidence for the new use. If both change, assess both. If neither changes, existing evidence may remain relevant, subject to version, time, and context limits. A norm describes where a score stands in a reference distribution. Validity evidence concerns whether the proposed meaning and use of that score are supported. The National Council on Measurement in Education’s Assessment Glossary says each intended interpretation needs use-specific validity evidence, and self-selected test takers may not represent a defined population. The International Test Commission’s Guidelines on Test Use also places interpretation in context. | What changed? | Evidence question | Practical reading | |---|---|---| | Neither population nor use | Does documentation still fit this version, language, administration, date, and claim? | Keep it if its basis remains relevant. | | Population changed; use stayed | Does the reference represent the newly named group? Is another defensible reference available? | A different norm may be needed; a narrower label alone does not prove it is better. | | Population stayed; use changed | What evidence supports this interpretation and decision in the proposed use? | The norm may remain suitable for relative standing while the new inference needs support. | | Both changed | Is there a suitable reference, and is the interpretation supported in the new context? | Examine both questions; neither settles the other. | Keep the score and reference group fixed, but move from private reflection to a decision about applicants. The percentile stays fixed because the reference stays fixed. The inference has changed, however. Reflection invites comparison with experience; selection uses scores for consequential judgments about people and work. Documentation must support that decision rather than borrowing authority from the percentile. The U.S. Office of Personnel Management’s Designing an Assessment Strategy says relevant validity evidence depends on how an assessment is used. Its example says a personality test intended to forecast customer-service job success may need evidence linking scores with later job performance. This is a question to investigate, not validation of a report. Suppose a report documented a broad reference and is presented as showing relative standing in a narrower group. Answers stay the same; the distribution may differ. A percentile is not an intrinsic personal quantity; it belongs to a score interpreted against a specified reference. Ask whether documentation supports the narrower comparison or whether another reference was built for it. If the report cannot show who was included, when data were gathered, or why the group fits its wording, the ranking may be unresolved. This limits ranking, not every description of responses. A yes prompts an evidence check, not automatically a new norm. A small or self-selected group can be a poor basis for broad claims even when its label sounds relevant. Local data may fit an explicitly local comparison if the report describes their scope. NCME defines local norms as referring scores to a limited population of interest and says they are not intended to represent populations beyond that setting. It distinguishes these from user norms lacking a well-defined reference population. Neither label proves sample quality. Ask the publisher or assessment user who was sampled and how that sample relates to the stated population; when data were collected; which version, language, and administration were used; what score and reference support the report’s claim; and what inference the score is meant to inform. For a changed use, ask what supports it in context. For selection or another consequential decision, ask what relevant outcome was examined and fairness issues. OPM’s guidance is U.S.-specific. A self-selected user group, changed language or version, or more consequential decision may alter the verdict. Name the missing link: ask for a different comparison when the population claim lacks support, use evidence when interpretation changes, both when both axes change, or narrower wording when neither is established. This makes the request concrete.

Sources: ITC Guidelines on Test Use; NCME Assessment Glossary; Designing an Assessment Strategy

What should I ask before relying on this comparison?

Before relying on a comparison, ask: “Which population produced this reference, and does it match the group named in the report?” Then ask what score is being interpreted. A raw score, percentile, or criterion comparison answers a different question. The NCME Glossary defines percentile rank relative to a specified distribution; without that distribution, a relative label cannot be checked. “What to Look for in a Personality Report’s Technical Manual” identifies where to look for the stated population and purpose. Next, name the decision: reflection, development, or selection. Ask what evidence supports that interpretation for that use. A reference group describes relative standing; it does not establish that the score supports a new decision. If the manual does not identify the comparison or intended use, treat that conclusion as unresolved. The missing information limits the ranking claim, but does not automatically erase every descriptive observation. If your question is how your work tendencies combine, the live Work Pattern Report offers private, low-stakes reflection across ten decision-and-collaboration continuums. It has no norm, cutoff, or selection score and cannot support hiring, promotion, compensation, performance management, diagnosis, or role matching. Use it to identify a pattern to discuss, separately from claims about population rank or job suitability.

Sources: What to Look for in a Personality Report’s Technical Manual; NCME Assessment Glossary