Chapter Four · failure evidence
What Multilevel & Hierarchical Linear Modeling got wrong, from 57 dissertations
Multilevel and hierarchical models frequently encounter practical limitations when datasets exhibit minimal clustering or numerical non-convergence during estimation. Analysts also report challenges with overparameterized random slopes, predictor multicollinearity, and participant mobility across clusters that violate strict hierarchical structures. These records come from PhD theses at 20 institutions, 2021 to 2026. Each links to its thesis. They were extracted by language models reading the full text, so treat each as a lead to read, not a verdict.
Negligible intraclass correlation coefficients render hierarchical clustering unnecessary
Datasets often exhibit very low intraclass correlation coefficients, indicating that grouping factors account for negligible outcome variance. In these situations, hierarchical models provide no advantage over single-level regressions and are frequently discarded due to small cluster counts or near-zero clustering dependency.
Considered and rejected
Considered and rejected: Rejected 3-level and 2-level hierarchical/multilevel modeling in favor of linear regression with dummy-coded site predictors due to minimal cluster variance (ICC = .003 for parent empowerment, .021 for outpatient service use, 0 for child symptoms) and single-participant courses.
Tried and failed
multilevel random intercept modeling with cluster-robust variance applied to hierarchical patient transfer outcome prediction. Reason: clustering adjustments did not meaningfully alter regression estimates compared to standard logistic regression
On healthcare self-organisation: a complexity approach to exploring patient journeys and outcomes · Imperial
Tried and failed
hierarchical linear modeling with random effects applied to hierarchical institutional and course data. Outcome: no signal. Reason: intraclass correlation coefficients were below the 5% threshold, rendering multilevel modeling unnecessary
Course withdrawals and college student success · UT Austin
Considered and rejected
Considered and rejected: Rejected multilevel hierarchical logistic regression for pandemic phase due to lack of level-2 clustering (ICC = 0.0018) and small number of clusters.
Immigrants’ Use of Online Mental Health Services during the COVID-19 Pandemic · Queens University Institutional Repository
Considered and rejected
Considered and rejected: Decided against multilevel regression for Aim 3 because the neighborhood intraclass correlation coefficient was virtually zero (ICC = 0.004), adopting single-level models instead.
HOUSING DISCRIMINATION AND HEALTH: EXPLORING THE RELATIONSHIP BETWEEN STRUCTURAL RACISM AND MENTAL HEALTH AMONG BLACK AMERICAN ADULTS · JScholarship
Considered and rejected
Considered and rejected: Rejected analyzing PSC via multilevel modelling (MLM) at group level due to limited clusters (two hospitals) and ICC variance <5%, opting for individual-level SEM.
The role of healthcare accreditation in influencing burnout and work engagement levels among healthcare professionals in The United Arab Emirates · University of Nottingham Repository
Considered and rejected
Considered and rejected: Rejected multilevel modeling for Group 2 nested data because the number of clusters (4 genetic counselors) was too few and ICC was too small (0.03)
Considered and rejected
Considered and rejected: Rejected mixed-effects logistic regression with site as a random intercept in favor of standard logistic regression due to negligible clustering (ICC ~ 0)
Considered and rejected
Considered and rejected: Multilevel modeling (hierarchical linear modeling) was rejected due to having only 35 school clusters (below the recommended 50) and non-significant ICCs (p = 0.26 and p = 0.13), reverting to OLS regression.
Supporting Youth Mental Health in Schools: An Investigation of Teachers as Gateway Providers · DSpace at SUNY Buffalo
Considered and rejected
Considered and rejected: Multilevel modeling (hierarchical linear modeling) was rejected after a null model yielded an intraclass correlation coefficient < 1% across home visitors.
The role of the home visitor: How intervention improves stressed parent-child interactions · Iowa State
Considered and rejected
Considered and rejected: Rejected multilevel factor analysis despite nested data (3 targets per participant) because intraclass correlation coefficients were very low (ICCs 0 to .27) and single-level analysis yielded identical factor structures.
Racialized and Multifaceted: Impacts of the Feminist Label and Race/Ethnicity on Perceptions of Men’s Masculinity · Queens University Institutional Repository
Considered and rejected
Considered and rejected: Rejected multilevel structural equation modeling (ML-SEM/HLM) because school-level intraclass correlation coefficients (ICCs) ranged from .0015 to .0222, well below the .05 threshold.
Malleable mental health factors in undergraduate engineering students · UT Austin
Considered and rejected
Considered and rejected: Rejected using multilevel / hierarchical linear modeling due to small sample size and insufficient number of classroom clusters (<20-30 clusters).
Optimization and numerical algorithms fail to converge during model estimation
Multilevel estimations frequently terminate without converging due to non-linearity, numerical integration difficulties, or small group-level sample sizes. These numerical optimization failures prevent researchers from obtaining valid higher-level standard errors or stable model fits across strata.
Tried and failed
Linear mixed-effects modeling applied to longitudinal plant growth count data. Outcome: did not converge. Reason: Singular fit errors caused by non-linearity in the response variable
Ecology and evolution of the African ant acacia, Vachellia drepanolobium, and its multiple symbionts · Harvard
Tried and failed
multilevel modeling with aggregate contextual predictors applied to longitudinal adolescent mental health data. Outcome: did not converge. Reason: insufficient convergence to compute higher-level standard errors
Tried and failed
multilevel structural equation modeling applied to hierarchical survey data. Outcome: did not converge. Reason: insufficient group-level sample size and between-cluster variance
Tried and failed
multilevel multinomial logit meta-regression with continuous moderators applied to multinomial meta-analytic count data. Outcome: did not converge. Reason: numerical integration via Gauss-Hermite quadrature failed to optimize with the continuous covariate specification
Patterns of continuity and discontinuity of childhood maltreatment across generations: A meta-analysis · Cambridge
Tried and failed
linear mixed-effects models with random slopes applied to pooled multi-cohort longitudinal trajectory data. Outcome: did not converge. Reason: None
Accounting for Heterogeneity in Health Decision Analysis · Harvard
Lost to a baseline
In Study II weight interpolation with dropping third-trimester weights, the cubic polynomial mixed-effects model failed to converge.
Lifestyle, Weight Gain, and Pregnancy Complications from a Life-Course Perspective · Harvard
Considered and rejected
Considered and rejected: Simultaneous multi-trait multilevel regression predicting goal progress rejected due to non-convergence
Personality-Goal Fit: Benefits and Consequences of Aligning Personality Traits and Personal Goals · Carleton University Institutional Repository
Considered and rejected
Considered and rejected: Rejected entering multiple PBAT items into a single multilevel model due to lack of model convergence
Considered and rejected
Considered and rejected: Rejected full multivariable adjustment in multilevel variance estimation models due to convergence failure, retaining only time as a fixed effect.
Diffusion of Procedure Innovation Among Cardiac Catheterization Operators at the Veterans Health Administration · ResearchWorks
Considered and rejected
Considered and rejected: Rejected mixed-effects logistic regression stratified across all wealth quintiles because the model could not converge at all quintile strata
ASSESSING THE PROVISION AND EQUITABILITY OF PRIMARY CARE IN A LOW-RESOURCE SETTING · JScholarship
Considered and rejected
Considered and rejected: Multilevel modeling rejected in favor of single-level path analysis due to non-convergence and low sibling clustering dependency (ICC = 0.013).
Considered and rejected
Considered and rejected: Multilevel modelling (using R lme4) was rejected due to model convergence failure on EDI Dysphoria, very low ICCs (2.6%–5%), and design effects < 2.
Therapist Factors on Autistic Children’s Socio-emotional Outcomes After Participating in a Cognitive Behavioural Intervention · Carleton University Institutional Repository
Considered and rejected
Considered and rejected: Rejected multilevel modeling with random intercepts due to very low ICCs (< 0.10) and model convergence/fitting failures, adopting standard logistic, negative binomial, and OLS regression with divisional fixed controls instead.
Investigating U.S. State-Level Income Inequality as a Determinant of Population Health: Theory, Evidence, and Directions Forward · DSpace at SUNY Buffalo
Complex random slope specifications lead to singular fits and overparameterization
Attempting to fit maximal random slopes or complex cross-level interactions often causes singular fit errors when data support is limited. Researchers repeatedly drop random slopes because having few observations per subject causes non-convergence and fails to improve model fit.
Tried and failed
maximal random slopes mixed-effects linear regression applied to hierarchical experimental survey response data. Outcome: did not converge. Reason: model complexity exceeded data support causing singular or non-converging optimization
Considered and rejected
Considered and rejected: Modeling random slopes in multilevel regression models, rejected due to model convergence issues caused by having only two within-person observations.
Comparing the Utility of Scoring Methods for the Hinting Task in a Heterogeneous Clinical Sample · Virginia Tech
Considered and rejected
Considered and rejected: Multilevel modeling with dyad members nested within the dyad was considered and attempted, but decided against because models failed to converge due to insufficient statistical power for 3-way cross-level interactions.
Examining Mother-Adolescent Emotion Dynamics in the Context of Perturbations at Two Time Scales: Parent-Adolescent Conflict and Major Life Stress · Queens University Institutional Repository
Considered and rejected
Considered and rejected: Rejected random slopes for time in multilevel models because they failed to improve model fit and caused convergence issues.
Considered and rejected
Considered and rejected: Random slopes in linear mixed-effects models rejected due to convergence issues
A Computational Approach to Recontextualization in Human Reading Behavior · Harvard
Considered and rejected
Considered and rejected: Rejected multilevel model with both random slopes and random intercepts because it included too many random effects for dataset size
Taking Flight: Overcoming Challenges in Airport Development · Harvard
Mixed-effects specifications underperform simpler fixed-effects models or produce estimation bias
Regularized and standard mixed-effects models can exhibit lower classification precision, higher root mean square errors, or signs of overfitting compared to fixed-effects baselines. Mixed-effects estimators can also introduce severe estimation bias when dynamic specifications like lagged dependent variables or unobserved cluster correlations are present.
Lost to a baseline
Mixed-effects LASSO had lower classification precision than fixed-effects LASSO across multiple SNR and cluster settings.
The Importance of Random Effects in Variable Selection: A Case Study of Early Childhood Education · Harvard
Lost to a baseline
Mixed-effects LASSO had higher coefficient estimation RMSE than fixed-effects LASSO across cluster variances (e.g. at higher variance, fixed-effects LASSO estimated true betas more accurately).
The Importance of Random Effects in Variable Selection: A Case Study of Early Childhood Education · Harvard
Lost to a baseline
Fixed-effects parameter estimates derived from mixed-effects models yielded worse MAB and RMSE than models fit directly with fixed effects alone
Modeling Stem Taper of Southern Appalachian Red Spruce · Virginia Tech
Considered and rejected
Considered and rejected: Decided against mixed-effects GLMs (with sampling round and site as random effects) in favor of fixed-effects GLMs due to observed indicators of model overfitting.
Advancing Rural Public Health: From Drinking Water Quality and Health Outcome Meta-analyses to Wastewater-based Pathogen Monitoring · Virginia Tech
Considered and rejected
Considered and rejected: Rejected mixed-effects models for longitudinal whole-brain lesion analysis due to high computational complexity and potential inconsistency if unobserved cluster heterogeneity correlates with covariates.
Advances in statistical methods for large-scale binary-valued neuroimaging data · Oxford
Considered and rejected
Considered and rejected: Rejected multilevel modeling with lagged dependent variables in SEM and test-gain linear regressions due to severe known estimation bias, opting for single-level clustered robust models.
Rigid nesting structures cannot handle participant mobility or non-hierarchical groupings
Longitudinal cohorts face substantial sample loss when participants move between school campuses or categories during a study window. Analysts rejected temporally fixed multilevel models and hierarchical mixed-effects structures because cluster migration, overlapping categories, and singleton clusters distort cluster-level representations.
Considered and rejected
Considered and rejected: Rejected multilevel (hierarchical) modeling due to overlapping participation across ECA categories and small subgroup sizes, opting for single-level multivariate OLS regression.
Lost to a baseline
12,572 students (out of 128,611) were excluded from the longitudinal high school cohort due to moving school campuses during the 4-year study window, as the multilevel model required a single campus grouping factor.
Examining equity of access and participation in computer science education · UT Austin
Considered and rejected
Considered and rejected: Temporally fixed multilevel model for regression analyses was rejected because base-year school-level characteristics ceased being representative as students migrated between high schools across waves.
Considered and rejected
Considered and rejected: Rejected Hierarchical Mixed-Effects (Multi-Level) Models for empirical data because singleton patterns at the industry-country level confound random intercepts with overall error terms.
Multicollinearity and high predictor correlation cause estimation instability
Including highly correlated scales or collinear contextual predictors induces severe linear dependencies in multilevel regression models. This collinearity leads to computational instability, estimation failures, and sampling breakdowns in small clustered datasets.
Tried and failed
multilevel logistic regression with competing event predictor applied to hierarchical student transfer outcome modeling. Outcome: did not converge. Reason: linear dependency and data collinearity among predictors causing estimation failure
Considered and rejected
Considered and rejected: Rejected a three-block hierarchical regression including state FIPS, Title IV status, programmatic completion rates, and student-to-faculty ratios due to severe missingness and multicollinearity/collinearity.
Tried and failed
multilevel modeling with collinear scale covariates applied to dyadic longitudinal survey data. Outcome: unstable. Reason: simultaneous inclusion of highly correlated coping and relationship quality scales caused severe multicollinearity
Tried and failed
multilevel multinomial logistic regression with random effects applied to small sample clustered longitudinal data. Outcome: unstable. Reason: cluster-period random effects caused computational instability and MCMC sampling failures in small samples
Causal inference methods for cluster-randomized trials under complex selection mechanisms · Penn
Violations of residual distributional assumptions and covariance structures undermine model fit
Hierarchical linear modeling was abandoned when underlying requirements for error independence, homoscedasticity, and residual normality were violated. In other settings, latent variable models yielded poor fit indices or standard compound symmetric covariance structures failed to capture spatial correlation decay.
Tried and failed
multilevel structural equation modeling applied to longitudinal survey diary data. Reason: Poor model fit indices for latent variables
Examining the Benefits of Cisgender Allyship Behavior for Allies of Transgender and Gender Non-conforming Individuals · Texas Tech
Considered and rejected
Considered and rejected: Decided against classical hierarchical linear models (multilevel models) because requirements for independence, homoscedasticity, and normality of residuals were violated, risking model instability
Einfluss von Struktur- und Prozessmerkmalen stationärer Einrichtungen auf die Qualität der Versorgung am Beispiel des akuten Schlaganfalls Influence of structural and process characteristics of inpatient facilities on the quality of care using the example of acute stroke · open_UMR Marburg DSpace 10.0
Considered and rejected
Considered and rejected: Rejected compound symmetric covariance structure in linear mixed-effects modeling in favor of a first-order autoregressive structure due to spatial correlation decay between distant disc levels.
Development of Imaging-Based Models for Analyzing the Spatiotemporal Function of Intervertebral Discs · DukeSpace
Left open by the authors
Problems the authors named and did not get to.
Left open
Conduct polynomial or semiparametric regression to test for curvilinear too-much-of-a-good-thing effects of work engagement on organizational outcomes. Blocker: Requires the private multilevel organizational survey dataset used in the thesis.
High-quality leader-member exchange relationship as a key to employee work engagement · Osuva
Left open
Apply multilevel modeling to the high school survey dataset to examine higher-level teacher effects on math achievement goals and attitudes. Blocker: Requires access to the private high school student and teacher survey dataset.
Left open
Apply multilevel modeling with bootstrapping to analyze within-group variance and non-independence in the cocaine treatment group study dataset. Blocker: Requires private clinical trial dataset collected for the thesis
Left open
Establish theoretical conditions and optimal weighting sequences for Multilevel Monte Carlo when beta <= 1 in pharmacokinetic experimental design models. Blocker: None
Variational, Monte Carlo and policy-based approaches to Bayesian experimental design · Oxford
Left open
Meta-analyze the effects of childhood maltreatment severity and chronicity on intergenerational continuity using multilevel multinomial logistic regression. Blocker: No specific dataset, coding scheme, or defined operationalization provided for the severity and chronicity dimensions across studies
Patterns of continuity and discontinuity of childhood maltreatment across generations: A meta-analysis. · Cambridge
Left open
Evaluate mixed-effects conformal inference and BNN frameworks on diverse ML algorithms and multi-level hierarchical structures using public data. Blocker: None
Prediction and monitoring with machine learning under complex dependencies for swine production data · Iowa State
Left open
Perform multilevel multinomial logit meta-regression testing parental age at birth and race/ethnicity as moderators of maltreatment continuity across generations. Blocker: None
Patterns of continuity and discontinuity of childhood maltreatment across generations: A meta-analysis · Cambridge
Left open
Extend the ECov hierarchical regression methodology to other generalized linear model families beyond logistic and Gaussian regression. Blocker: None
Left open
Compare mixed-effects location-scale models to CRVE methods to evaluate if CCREMs can relax exogeneity and homoscedasticity simultaneously via covariate centering. Blocker: None
A comparison of methods for centering covariates in cross-classified random effects models · UT Austin
Left open
Analyze fMRI mixed-effects model variance components against physical variances and evaluate residual patterns after mixed-effects partitioning. Blocker: None
Extracting Feature Vectors From Event-Related fMRI Data to Enable Machine Learning Analysis · Virginia Tech
Checking a claim in this area?
We can run the same search on any method or claim. If nothing turns up, we will say so, and that proves nothing on its own.