How to Evaluate an Automated Personality Assessment

What claim is the report asking you to accept?

Start with the claim the report asks you to accept and the decision it informs. A result may summarize your answers about behavior, infer a tendency from digital traces, place a score within a comparison group, or predict an outcome. These claims are not interchangeable. Automation describes how information is processed; it does not establish that an interpretation is sound. The National Council on Measurement in Education’s glossary defines validity as evidence and theory supporting a particular score interpretation for a particular use. “Is this assessment valid?” is too broad. Ask what the score says, about whom, and for what purpose. A reflection prompt, coaching conversation, workplace development exercise, screening step, and hiring decision carry different consequences. Evidence for one use does not automatically support another. A polished explanation or fast result does not settle what a report means. “How to Evaluate an Automated Personality Assessment,” the existing article, distinguishes automation from objectivity; that principle does not prove anything about a specific product. Treat a result as a tentative description of tendencies, not a diagnosis, fixed identity, or stand-alone career or hiring verdict. If the report omits its intended use, withhold stronger conclusions until that gap is addressed.

Sources: NCME Assessment Glossary; How to Evaluate an Automated Personality Assessment

Trace the route from input to reported score

Start with the input, not the polished paragraph at the end. Find what information the system uses and how it becomes a score or narrative. Questionnaire answers, recorded speech, resume text, and social-media material are different evidence, even when software processes them alike. Ask what the system observes, what it infers, and which steps turn the inference into the result shown. Map five links: source; response or captured signal; preparation or feature extraction; scoring rule or model; and reported scale, category, or narrative. If a result describes a tendency, find out whether it came from answers about behavior, a pattern detected in speech, or a combination. The first is direct self-report. The second uses a proxy, an observable feature standing in for the characteristic being measured. That intended characteristic is called the construct. A proxy may be informative, but the connection needs evidence; the feature itself is not the personality tendency. The NCME Assessment Glossary defines a construct as the concept or characteristic a test is designed to measure. It also defines a computer-prepared interpretive report as a programmed interpretation of test results based on empirical data and/or expert judgment, presented as narrative, table, or graph. The report’s sentences are interpretations, not raw observations. Polished wording does not show whether the steps behind it are justified. Consider two hypothetical inputs. A person answers a prompt about whether they prefer to plan before beginning a task. A report might summarize that stated preference, subject to how the question is worded and scored. In another system, software analyzes speech rate to infer sociability. Speech rate is observable; sociability is inferred. Ask what evidence connects the captured feature to the construct, and whether other explanations for the signal have been considered. These examples illustrate different routes, not findings about a particular assessment. Request the instrument name and version, a description of its inputs, and how they affect the reported scale or wording. If it uses indirect traces, ask which features matter and what supports interpreting them as personality. Ask whether you can correct a factual error in the source material. You need enough information to follow the claim, not source code: a provider can protect implementation details while explaining input categories, scoring logic, and supporting evidence. Labels such as “automated,” “adaptive,” or “personalized” do little by themselves to establish what a report means. The 2022 paper “An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring” examined two hiring systems that inferred personality from job seekers’ digital materials. Its authors report substantial instability in key measurement facets and describe stability as necessary but insufficient for validity. The study concerns those systems and that setting; it does not establish how every current tool works. It shows why the input route matters when indirect material becomes a claim. Once the chain is visible, ask: is the result a summary of what someone reported, or an inference from a proxy? What transformation connects input to scale? Which parts are explained, and which remain unknown? If the provider cannot answer well enough for you to understand the claim, treat the narrative as an interpretation with an unclear basis and seek clarification before accepting it as a description of the person.

Sources: NCME Assessment Glossary; An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring

What does the number compare you with?

Before interpreting a number, identify its type and reference. A raw score is calculated directly from responses. A scale score is transformed to make interpretation easier. A percentile rank describes the percentage of scores in a specified distribution that fall below a given score. These labels answer different questions: how responses were tallied, how the result was put on a reporting scale, and where it sits relative to other scores. The NCME Assessment Glossary makes the comparison group part of a norm-referenced interpretation. Without that group, a percentile is incomplete. A percentile is not the percentage of a personality trait someone has, the share of situations in which they will behave a certain way, or a probability of success. It marks relative position within a distribution. “High” may mean only that a score falls toward the upper part of the named group. It does not by itself mean better or suited to a job. A value judgment requires a criterion and evidence connecting the score to it. Otherwise, “higher” and “lower” are descriptions. Ask who supplied the comparison scores. The NCME glossary defines norms as score statistics for specified groups, usually intended to represent a larger reference population. It separately defines “user norms” as statistics for a group that does not represent a well-defined population, such as a self-selected group. User norms can describe that group, but should not be presented as if they locate a reader in a broader population. A report or technical manual should explain how the group was defined, when data were collected, and the language and conditions of assessment. Ask whether this group suits the person and question. A demographic label alone does not show a norm is appropriate. “What to Look for in a Personality Report’s Technical Manual” recommends checking the norm description, its date or edition, and whether low, average, or high labels refer to national, local, or user norms. A well-described norm group does not establish that an assessment measures the intended tendency; it clarifies the comparison behind the score. Conversely, a carefully measured trait does not make every comparison group suitable. Evidence about what a scale means and whom a comparison describes are related, but not interchangeable. The NCME glossary’s definition does not certify a product or claim. Consider an unnamed report that says “high” but does not explain whether its reference consists of a defined population or people who chose to complete the same online questionnaire. The responses may still be summarized descriptively if scoring rules are clear. But the reader cannot infer a broader population standing from an unspecified comparison, or turn the label into a prediction about work behavior. Note the score type, comparison group, and the report’s meaning for the scale. If one is missing, ask the provider before treating the number as comparative evidence. If the norm group does not match the reader or question, withhold the comparative interpretation. If no group is named, “high,” “average,” and percentile have no clear reference. Missing disclosure is not proof of inaccuracy; it is a reason to limit the claim. Read a number comparatively only when its score type, reference group, and scale meaning are clear.

Sources: NCME Assessment Glossary; What to Look for in a Personality Report’s Technical Manual

How much could the score vary?

Reliability, also called precision in measurement, concerns how consistent scores are under specified conditions and how much random measurement error affects them. It is a question about score dependability, not whether the score captures the personality tendency named in a report. The NCME Assessment Glossary defines reliability/precision in terms of consistency across repeated applications and freedom from random error. Reliability claims need the conditions: what was repeated, with whom, and when. Different estimates answer different questions. Internal consistency examines how responses or parts of a scale relate to one another. Test-retest stability asks whether scores remain similar when the same people take a measure again after an interval. Agreement across forms or raters concerns whether alternate versions or different observers yield comparable results. The Society for Industrial and Organizational Psychology’s Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection notes that consistency may be examined across items, samples of behavior, raters, algorithmic predictions, or repeated assessments. In employment selection, it says the appropriate estimate depends on the intended interpretation and use. A strong figure for one kind of consistency cannot stand in for every other kind. For an individual result, ask how much uncertainty surrounds the displayed value. The standard error of measurement estimates the spread expected in observed scores across repeated administrations under the same conditions. The NCME glossary notes that it is usually estimated from group data, since identical repeated administrations cannot generally be collected for each person. A provider may show uncertainty as an interval or another account of expected score variation, helping keep a precise-looking number from implying more certainty than the measure supports. Uncertainty matters when a report draws a line between nearby results, such as assigning different labels to scores just above and below a band threshold. If uncertainty around an individual score is substantial relative to that distinction, the evidence may not justify treating the labels as meaningfully different. Look for an explanation of when scores can and cannot be distinguished; do not assume every decimal or category carries a separate conclusion. SIOP's recommendations say reliability estimates in selection should be accompanied by the study design, data collection, and interpretations they support. The practical question is whether precision fits the interpretation. There is no universal reliability coefficient that makes every personality assessment adequate. The needed consistency depends on what the score is asked to do. A broad prompt for private reflection can tolerate more uncertainty than a narrow distinction used for a consequential decision; this is an inference from the different costs of error, not a cutoff supplied by the sources. Judge evidence against the proposed use, not a number detached from context. A stable result can still be irrelevant. If a system repeatedly assigns the same score because it reacts to a feature unrelated to the intended characteristic, reliability will not fix that problem. Consistency addresses random variation; evidence for the score's meaning and use is separate. Ask what kind of reliability was studied, what uncertainty applies to an individual result, and whether the evidence fits the conclusion invited by the report. If those answers are missing, avoid fine distinctions and keep the score's role narrow. A result presented as exact deserves less trust when its precision, conditions, and limits are unexplained.

Sources: NCME Assessment Glossary; Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection

Does the evidence match the use you have in mind?

Ask whether evidence supports this score’s interpretation for this purpose and population. Evidence that a scale behaves as expected does not automatically establish a prediction about job performance or justify selection. The higher the consequence, the more specific the evidence and safeguards need to be. “Validated” is incomplete unless the provider says what use the evidence supports.

The NCME Assessment Glossary defines convergent evidence as a relationship between scores and measures of the same or a related construct. That can show whether a scale behaves as expected, but not that it predicts a particular outcome. Outcome evidence asks whether the measure relates to a relevant criterion, such as a defined aspect of performance. Even a demonstrated relationship does not settle whether the score should guide a decision. That depends on use, who is affected, and what other information is considered.

Follow each claim back to its conditions. Who took part? What did the study measure, and when? Was the outcome a self-description, rating, or observed result? Was the assessment version the same as the one now offered? Do participants and conditions resemble the people and setting at issue? A finding in one population may not support the same conclusion elsewhere. NCME’s Assessment Glossary describes validity as evidence and theory for a specified interpretation and use; it does not certify a product. Ask the provider to name the claim, population, and use covered.

There is a fair case for automation. A well-documented system may standardize scoring, ease administration, or offer access an alternative does not. Those are practical benefits, but consistency and convenience do not establish a sound personality interpretation. Nor is a conventional questionnaire automatically better because a person answers direct questions. Compare each method’s input, score meaning, population, intended use, and review process. The choice is between evidence and conditions, not “human” and “computer.”

Employment calls for a higher-stakes review. The Society for Industrial and Organizational Psychology’s Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection addresses selection and promotion. It calls for scrutiny of job-relevant accuracy, score consistency, fairness, operational use, and documentation for verification and auditing. This is professional employment guidance, not a vendor validation study or a universal rule for private reflection. An employment claim needs evidence tied to the work and decision. A personality description alone cannot establish that someone will perform well in a role or justify a hiring score or job recommendation.

Coaching differs from screening an applicant. A tentative pattern may prompt questions about planning, feedback, or ambiguity; it is not a prediction or fixed account of the person. As an inference from the use-specific evidence principle, private reflection can tolerate more uncertainty when it does not determine access to an opportunity. Compare the result with experience and context. If evidence covers only self-reflection, using the report in selection exceeds its support.

Match the conclusion to evidence. If documentation supports a description of tendencies for a relevant group, read it as bounded. For an outcome claim, check that the current tool, population, and decision resemble the studied case. If details are missing, ask before consequential reliance. Missing public documentation is a reason to limit confidence and seek clarification; it does not prove inaccuracy. Until resolved, do not treat the score as a verdict.

Sources: NCME Assessment Glossary; Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection

What changes when the input changes?

For systems that infer personality from digital traces, ask whether small, irrelevant input changes alter the profile. Stability means that inputs equivalent for the measurement produce sufficiently similar outputs. It is a useful check, not proof that a score measures the intended tendency. A system can be consistent while relying on a weak signal; a score can also change when a person’s circumstances genuinely change. Distinguish those changes from file or format changes that should not matter. The 2022 paper “An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring” examined two commercial systems, Humantic AI and Crystal, which inferred personality from job seekers’ resumes or social profiles. Researchers varied input sources and, for one system, resume file formats. They reported substantial instability in important facets and concluded that the systems studied could not be considered valid testing instruments. This is evidence about those systems under the audit conditions, not every current product or a comparison of all questionnaires with all digital systems. Use it as a documented risk and a method for asking questions, not a universal verdict. Ask a provider what it has tested. Does it compare reasonable alternate formats or source variations? Does it examine repeat administrations when a tendency should be stable? Which changes are expected because a person’s situation has changed, and what counts as a meaningful score difference? Was the same system version used? A file conversion offers a clear contrast: if content is preserved but the profile moves, the result may be sensitive to a nuisance feature rather than the intended signal. A new response after a genuine contextual change is different; it could reflect changed circumstances, changed answers, or ordinary measurement variation. The changed input determines what the comparison can show. The Journal of Applied Psychology study record, “Automated Video Interview Personality Assessments: Reliability, Validity, and Generalizability Investigations,” illustrates why this category needs finer distinctions. It reports models developed and cross-validated across three mock interview samples, with test-retest examination in a fourth. Models trained on interviewer reports showed mixed reliability evidence, convergent and discriminant relations, and prediction of academic outcomes; models trained on self-reports showed little evidence of reliability or validity. These findings concern studied methods in mock video interviews, not every vendor. The academic outcome also does not establish prediction of workplace performance. The signal, criterion, sample, and study design all matter. If a provider cannot say whether it tested stability across irrelevant input changes, that absence does not prove inaccuracy. It does mean this kind of consistency cannot support confidence in the score. Be especially cautious when a system infers traits from incidental traces but does not explain its input path. For private reflection, treat the output as a tentative prompt and compare it with concrete behavior and context. Before a consequential decision, ask for relevant documentation and review; if unavailable, do not treat the profile as an individual verdict. A clear account of version, variation tests, and limits can support cautious interpretation. Unexplained sensitivity or silence leaves the inference unresolved.

Sources: An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring; Automated Video Interview Personality Assessments: Reliability, Validity, and Generalizability Investigations; Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection

Who can access the assessment and challenge its use?

An assessment can give everyone the same screen and still place some people at a disadvantage. Check whether a person can complete its tasks accessibly, whether the provider has examined relevant group differences, and whether there is a route to request an explanation, correction, accommodation, or review. Identical software conditions do not establish that scores mean the same thing for everyone. The NCME Assessment Glossary defines fairness in relation to whether score interpretations are valid for relevant subgroups, and explains that features unrelated to the intended construct can distort scores. This concern applies across assessment formats. Start with what the assessment asks a person to do. A timed response may depend on typing speed; a spoken response may be affected by speech patterns; a visual interface may require sight; a mobile task may assume a particular device or motor control. Language and time limits can shape responses too. These conditions matter when they influence a score for reasons separate from the personality tendency the tool claims to describe. Their presence is a reason to investigate, not proof of bias. The U.S. Equal Employment Opportunity Commission’s 2022 worker guidance illustrates this risk in employment settings. It describes a timed keyboard test that could screen out a person with severe arthritis when typing quickly is not needed for the job, and a visual memory game that could disadvantage a blind applicant. It also describes a speech-pattern system assessing problem-solving that could score poorly for a qualified person who stutters. They show how response format or a scoring feature can become an obstacle unrelated to the ability being assessed. For hiring or promotion in the United States, the EEOC advises workers to ask what skills a test assesses and request a format that shows their ability to do the job. This guidance is specific to disability and employment, not a general legal rule for private self-reflection. The Society for Industrial and Organizational Psychology’s recommendations for AI-based employee selection likewise call for attention to fairness, operational use, and documentation in employment decisions. Ask whether a person can request an accessible format or review. A provider’s evidence should fit the concern. Ask whether it examined score behavior across relevant groups and tested offered response modes or accommodations. Ask what data are collected, who can access them, how long they are retained, and whom to contact about an error or challenge. These are distinct checks: secure storage does not show that a personality interpretation is sound, while fairness evidence does not reveal who can retain the data. If a consequential process cannot explain what a task measures or offers no meaningful way to raise an access problem, do not treat its score as self-explanatory. Request relevant information or an appropriate alternative before relying on it. For private reflection, inaccessible design or thin documentation may mean the output is a poor fit; it does not prove the tool inaccurate for everyone. Keep confidence within what the evidence supports, and do not turn one automated result into a verdict about personality, capability, or employment potential.

Sources: The Americans with Disabilities Act and the Use of Software, Algorithms, and Artificial Intelligence to Assess Job Applicants and Employees; NCME Assessment Glossary; Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection

Can the result be explained, reviewed, and kept current?

A usable report should identify its limits, explain how its score was produced, and give a route for questions or correction. If the provider updates the scoring system, ask how it tracks versions and checks whether earlier evidence still supports the current output. Governance guidance can frame these questions, but it cannot establish that a personality interpretation is sound. Start with three requests: the system or instrument version and date used for this result; a change history identifying material revisions to scoring or interpretation; and a named contact or review route. These do not require release of proprietary code. They give the reader a reference point for what produced the report, whether its explanation applies to this version, and where to raise a factual error. Ask whether the provider retains enough records to answer those questions later. The Society for Industrial and Organizational Psychology’s 2023 recommendations address AI-based assessments used in employee selection and promotion. They call for documenting development and scoring decisions so the process can be verified and audited, and treat changes in organizational demands as relevant to review. Its scope matters: it concerns employment decisions. It does not establish that every private reflection tool needs the same validation program or certify any product. For employment use, ask who is responsible for interpreting the result, whether a qualified person can review it, and how the organization checks that the version remains appropriate for its purpose. For private reflection, the proportionate version may be simpler: understand the tool’s limits and know how to contact its provider about an unclear result. The National Institute of Standards and Technology’s AI Risk Management Framework Playbook offers a broader, bounded lens. Its suggestions are organized under Govern, Map, Measure, and Manage. NIST says the Playbook is voluntary, is not a checklist or mandatory sequence, and can be adapted to context. It is general AI risk-management guidance, not a personality-testing standard. It cannot show that a score measures a trait or validate an interpretation. A change does not automatically make a report invalid. Changes may preserve the score’s meaning, or alter its inputs and interpretation. Without a change record, the reader cannot tell which is true. That uncertainty warrants limited confidence and a request for clarification, not a conclusion that the system is inaccurate. Privacy questions are related but separate. Ask what information was collected, who can access it, how long it is retained, and what options exist to correct or remove it, based on provider disclosures and applicable rules. A clear data policy does not show that a personality score is meaningful; measurement evidence does not explain who keeps source material. If the version, responsible owner, or review route is unclear, treat the result as a provisional prompt for reflection at most. Do not rely on it for a consequential judgment until its process and evidence can be examined for that use. A clear explanation, traceable version, and real review route make an assessment easier to question and use responsibly; they do not, by themselves, make its claims true.

Sources: NIST AI Risk Management Framework Playbook; Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection

What should you do with this report now?

Write down the report’s main claim, the evidence offered, and the decision at hand. If it says you prefer structure, translate that into an observable tendency: you may find it easier to begin when steps and deadlines are clear. Treat this as a possibility, not a fixed description. Note one situation where it fits and one where circumstances might explain a different response. Deadlines or team support can change behavior. This check keeps the report connected to behavior and context.

Ask one specific question about the missing link. If the report gives a percentile, ask which reference group it uses; the National Council on Measurement in Education’s glossary defines norm-referenced interpretation as comparison with a specified population. For a broader claim, ask what evidence supports it for your intended use. If unclear, keep it as a reflection prompt, not a basis for consequential decisions. In coaching or a work conversation, offer it as a hypothesis to test against examples, not a diagnosis, hiring score, or job recommendation.

For reflection, the publication’s Work Pattern Report at [/assessment](/assessment) offers a private self-report across ten decision-and-collaboration continuums, with response spread and paired interactions. It has no norms, cutoff, type, or selection score and is not validated for hiring or career matching. Use it to frame questions about work friction and compare them with actual situations. The verdict is conditional: rely on an automated report only as far as evidence supports its interpretation and use. Clear documentation, relevant comparison data, and a review route could justify more confidence; their absence calls for restraint, not a conclusion that the assessment is inaccurate.

Sources: NCME Assessment Glossary; Work Pattern Report assessment