Symptom Validity: Some observations and Comments about Over-Reporting

In my view, many of the most important and most interesting questions we have about symptom validity remains either unanswered, or rarely explored. The purpose of this paper is to outline some patterns I have observed, and to describe what I believe are critical steps for the future of the field – and for the development of the scientific practice of validity interpretation more specifically.

1. Feigning detection typically within a standardized band of effect, regardless of instrument used, condition, and even study type in most cases. These effects differ according to the specific statistical analysis.
   a. Mean effect differences typically range between .70 and 1.30, with a standard deviation approximately half of the mean effect
   b. Sensitivity ranges from .10 to .50 with an average of .30, with specificity set to .90. This standardized effect range has been termed the Larabee limit” by some.
   c. Correlations between SVTs are high, often falling within a large effect range. These between domain correlations appear robust, and do not appear to differ much between distinct symptom domains of over-reporting (e.g., somatic, cognitive, and psychopathology)
   i. These associations will typically be between r = .75 and .85
   ii. Differences are typically around a small effect (r~|.10|)
2. Moderation patterns are generally typical across instruments, and have not changed as a function of the development of new instrument versions (e.g., transition from the MMPI-2 to the MMPI-2-RF, or MMPI-2-RF to MMPI-3). This pattern is consistent with assessment instrument developmental broadly [e.g., portion of cognitive test changes between WAIS/WISC versions have similarly declined in time. For instance, the purpose of an evaluation, the type of client, their diagnosis, are their racial and ethnic background are each major and common moderators of effect. Moderation tends to be a question only of meta-analysis, which tend to reaffirm these patterns such that they can be acknowledge but not explored in follow-up study.
   a. Moderation patterns are rarely explored in experimental (simulation) designs, which is a missed opportunity to advance understanding of how, when, and why people respond in the manner that they do.
   b. Meta-analysis evaluating criterion variables (e.g., PVT and SVT used to create criterion groups) is limited, and not possible because of how these groups are created in the literature (See Herring et al., in press – PAI Meta).
3. Participatory research is rare in feigning research on personality assessment, leading to a potential for over-interpretation based on assumptions. In general, it is my perspective that the study method approaches used in the development of self-report assessment often serve to reify the ideas measured based on face validity, from the perspective of the test developer. Such approaches are similarly common on substantive scales, which is why recent work has suggested the scales are not viewed in the same pathological way in a variety of groups (e.g., MMPI-3’s RC6/RC8/CMP scales).
4. Most individuals who are identified as over-reporting during research studies (e.g., Known-Group Designs) score below the recommended cutoff scores for scales. Thus, positive predictive power is often low for scales, indicating that it is common for them to go undetected and highlighting some of the concerns raised by Leonhard’s thoughtful papers from 2024.
5. In research, the following are not typical but seem useful to developing and advancing our theory and practice of over-reporting detection. As such, we do not have a clear consensus of what effect requirements to support more clear and concise measurement, and thoughtful discussion and debate is needed to set benchmarks for when and how to answer specific interpretive questions. Advancing methods is a key want to improve this discussion. I sometimes describe this process as one in which assessment psychologists have used face validity to reify theory, rather than taking the next step and testing assumptive processes, including:
   a. SEM models to test moderation and multivariate patterns.
   b. Additional group analysis inclusive of each PVT/SVT criterion, providing fruit for long term Meta-analytic interpretation.
   c. Correlations Matrix of between scale relationship
   d. Comparison of individual scale effects and determination of incremental utility
   e. Consideration of elevation patterns following configural approaches (e.g., awareness of their multivariate nature as a lens through which meaningful interpretation is possible).