Chapter Four · failure evidence
What active learning got wrong, from 57 dissertations
Papers report the method that worked. Theses also record what didn't. These are the active learning failures, baseline losses and rejected designs we found in PhD theses from 17 institutions, 2021 to 2026. Each one links to its thesis. They were extracted by language models reading the full text, so treat each as a lead to read, not a verdict.
Uncertainty sampling focuses on redundant, unrepresentative, or pathological data points
Active learning strategies that rely purely on uncertainty or variance measures often fail to improve upon baseline methods. These approaches tend to sample redundant instances from overlapping clusters or boundary regions, rather than exploring diverse and unrepresented areas of the data distribution.
Tried and failed
Maximum variance sampling applied to interactive active learning. Outcome: worse than baseline. Reason: Maximizing input variance alone matched random sampling rather than providing informative active learning queries.
Feedback Coding for Efficient Interactive Machine Learning · Georgia Tech
Tried and failed
pure uncertainty sampling in active learning applied to iterative surrogate model training. Outcome: no signal. Reason: selected redundant candidates from overlapping clusters instead of exploring diverse unrepresented regions
Tried and failed
Maximum variance active learning selection applied to interactive binary feedback learning. Outcome: worse than baseline. Reason: Pure variance exploration fails to outperform random sampling in bisection-based candidate refinement
Feedback Coding for Efficient Interactive Machine Learning · Georgia Tech
Tried and failed
exploration-based k-center active learning applied to budget-constrained pool-based active learning. Outcome: worse than baseline. Reason: pure exploration was less effective than simple uncertainty sampling under a limited query budget
Optimizing resource allocation in computational sustainability: Models, algorithms and tools · Georgia Tech
Tried and failed
active learning with variance-based acquisition applied to Gaussian process surrogate modeling. Outcome: too slow. Reason: frequent retraining overhead negated sampling efficiency, underperforming random sampling for fast simulations
Time-dependent damage of soft materials with bond breaking and healing kinetics · Cornell
Lost to a baseline
BALD active learning performance on Repeated MNIST fell behind random acquisition due to acquiring redundant near-duplicate points.
Uncertainty Estimation: single forward pass methods and applications in Active Learning · Oxford
Considered and rejected
Considered and rejected: Rejected static single-snapshot acquisition functions in active learning because they rely heavily on arbitrary acquisition epochs and limit hypothesis diversity.
Considered and rejected
Considered and rejected: Rejected Active Learning Cohn (ALC) / Active Learning MacKay (ALM) maximum predictive variance sampling for computer model calibration designs because it tends to concentrate acquisitions along the boundary of the input space rather than space-filling the interior.
Bayesian Methods for Mineral Processing Operations · Virginia Tech
Tried and failed
likelihood-weighted total uncertainty active learning applied to surrogate model training. Outcome: did not converge. Reason: sampling using total uncertainty causes active learning to get trapped in pathological attractor states near the mean
Tried and failed
pure uncertainty or pure diversity active learning sampling applied to domain adaptation under large distribution shift. Outcome: worse than baseline. Reason: isolated uncertainty selects outliers while pure diversity ignores shift severity, failing under hard domain shifts
Towards Reliable Computer Vision Systems · Georgia Tech
Considered and rejected
Considered and rejected: Rejected building active learning data selection policies solely from heuristic uncertainty measures because they risk sampling redundant data points from narrow regions of the distribution without ensuring representativeness
Considered and rejected
Considered and rejected: Rejected pure uncertainty-based deep active learning query acquisition functions due to severe x-space mode collapse under variable pool ratios.
Deep Learning for the modeling and design of artificial electromagnetic materials · DukeSpace
Active learning requires significant manual intervention or wet-lab validation that defeats the purpose
Active learning workflows are frequently abandoned or left unfinished when they require slow, expensive physical experiments to validate the selected samples. This bottleneck prevents the rapid iteration needed for effective active learning cycles.
Left open
Implement active learning workflows to evaluate and generalize thermoset model ensembles on novel chemistries like strand-cleaving crosslinkers. Blocker: Requires wet lab experimental synthesis and testing of novel polymer chemistries to complete the active learning cycle
Designing Macromolecules using Machine Learning and Simulations · MIT
Left open
Implement an active learning framework to guide experimental drug synergy validation and recalibrate the graph neural network synergy model. Blocker: Requires wet-lab experiments to generate validation data for active learning iterations
Therapeutic lead discovery using graph deep learning · Imperial
Considered and rejected
Considered and rejected: Multi-view machine learning and active learning workflows were rejected/abandoned due to the lack of wet-lab validation throughput to confirm gold-standard training data.
Adapting Genome-wide microRNA Discovery and Target Prediction to Specific Species · Carleton University Institutional Repository
Left open
Expand active learning polymer discovery models to predict Tg, polymerization compatibility, thermal conductivity, resin viscosity, and printability parameters. Blocker: Requires experimental synthesis and characterization data for specialized dual-network photo-thermal resins to train and validate active learning loops.
Tandem Development of Novel Materials and Manufacturing Techniques for Direct Ink Write 3D Printing · Georgia Tech
Left open
Simulate and characterize surface restructuring mechanisms and reverse segregation cooperativity in the presence of reactive adsorbates (CO, O, H) using active learning MD. Blocker: Requires substantial computational resources and domain-specific knowledge to configure reactive DFT/active learning simulations without specific target alloy/surface details.
Left open
Implement Bayesian optimization active learning to guide iterative experimental exploration of g-C3N4 material synthesis spaces. Blocker: Requires wet lab synthesis and experimental characterization feedback loop for iterative validation
Left open
Integrate active learning and uncertainty quantification into reinforcement learning pipelines for autonomous experimental chemistry platforms. Blocker: Requires integration with an autonomous physical/chemical experimental platform and lack of specific pipeline details
Bayesian optimisation in chemical problems · Imperial
Left open
Integrate automated robotics with active learning algorithms for closed-loop autonomous discovery of high-entropy multimetallic alloy catalysts. Blocker: Requires a physical automated robotic synthesis and testing laboratory apparatus.
Designing Electrocatalysts for the Production and Oxidation of Liquid Fuels · MIT
Bayesian optimization acquisition functions struggle with high dimensionality, complex objectives, or poor uncertainty estimates
Active learning using Bayesian optimization often fails because standard acquisition functions do not scale well to high-dimensional spaces or multiple constrained objectives. Furthermore, poor uncertainty quantification in the underlying surrogate models leads to ineffective sample selection.
Tried and failed
deep Gaussian process active learning applied to aerodynamic dynamics emulation. Outcome: worse than baseline. Reason: suboptimal Bayesian MCMC parameter inference hindered performance compared to standard kriging
Active Learning Methods for Emulation and Inverse Design · Georgia Tech
Tried and failed
multi-objective Bayesian optimization with L-BFGS acquisition applied to high-dimensional benchmark functions. Outcome: did not converge. Reason: acquisition function optimization became under-parameterized when objectives approached input dimensionality
Multiscale Modeling and Microstructure Design of Biodegradable Polymeric Scaffolds · Georgia Tech
Tried and failed
weighted variance multi-output active learning acquisition applied to multi-output Gaussian process surrogate modeling. Outcome: worse than baseline. Reason: underperformed simpler maximum variance and round-robin sampling strategies across multi-output benchmarks
Tried and failed
regret-based Bayesian optimization acquisition function applied to privacy-constrained personalization querying. Outcome: worse than baseline. Reason: Failed to outperform uniform querying across privacy-constrained personalization simulations.
Essays on the Decision Value of Data in Marketing Measurement and Targeting · Penn
Tried and failed
Gaussian approximation of deep Gaussian process posteriors applied to Bayesian optimization uncertainty estimation. Reason: yielded poor uncertainty quantification and inaccurate variance estimates for acquisition functions
Physics-informed Machine Learning for Digital Twins of Metal Additive Manufacturing · Virginia Tech
Tried and failed
random sampling for acquisition function optimization applied to high-dimensional constrained Bayesian optimization. Outcome: worse than baseline. Reason: random sampling struggles to find optimal acquisition points in high-dimensional or constrained search spaces
Sequential black-box optimization via global optimization of tree ensembles · Imperial
Lost to a baseline
Expected Improvement (EI) acquisition function was outperformed by Upper Confidence Bound (UCB) during Bayesian optimization hyperparameter selection.
Data-Driven Design of Recycling-Friendly Aluminium Alloys · MIT
Considered and rejected
Considered and rejected: Rejected Lower Confidence Bound (LCB) acquisition function in favour of Probability of Improvement (PI) and Expected Improvement (EI) due to failure to locate feasible regions.
Robust optimisation of wing aerostructural response · Cranfield
Considered and rejected
Considered and rejected: Rejected optimizing GPR kernel hyperparameters via marginal likelihood due to overfitting on micro-data and search acquisition instability.
Parameter Space Abstractions for Diversity-based Policy Search · Imperial
Active learning underperforms simple random sampling in data-starved or limited budget regimes
In scenarios with very few initial training samples or a strictly constrained labeling budget, active learning strategies often fail to outperform uniform random sampling. Initial models trained on minimal data yield uninformative metrics, making active selection no better than random choice.
Tried and failed
distance-thresholded active learning exploration applied to chemical reaction optimization. Outcome: worse than baseline. Reason: Similarity constraints failed to outperform random sampling in small sample-size regimes
Lost to a baseline
Random selection occasionally outperformed the active learning sequential design strategy initially when fewer than 50 training microstructures were evaluated.
Neural Inverse Microstructure Design with Bayesian Scale-Bridging · Georgia Tech
Lost to a baseline
Active learning with Tanimoto similarity cutoffs (cutoff 0.49) did not achieve lower loss than random learning for n < 1000 on Suzuki cross-coupling data.
Lost to a baseline
Random learning matched or beat active learning on the Suzuki dataset prior to adding ~1,000 training reactions.
Lost to a baseline
Uniform sampling beats greedy active learning baselines (Uncertainty, Entropy) on FashionMNIST and ImageNet benchmarks in data-starved regimes.
Tried and failed
uncertainty-based pre-filtering for batch active learning applied to subset selection with few initial samples. Outcome: no signal. Reason: initial models trained on minimal data yield uninformative uncertainty estimates
Active Learning Under Limited Interaction with Data Labeler · Virginia Tech
Lost to a baseline
Quivr achieved higher F1 scores on trajectory queries than EQUI-VOCAL when the labeling budget was strictly restricted to 12 examples (with 0 active learning iterations) due to full search space enumeration.
AI-assisted video data management for compositional and high-level queries · ResearchWorks
Lost to a baseline
Quivr achieved higher F1 scores on trajectory tasks when the labeling budget was strictly restricted to 12 examples (no active learning rounds) due to full search space enumeration.
AI-assisted video data management for compositional and high-level queries · ResearchWorks
Active learning struggles to acquire sufficient samples of rare classes under extreme data imbalance
When applied to severely imbalanced datasets, standard active learning strategies often fail to sample enough minority class instances. Both uncertainty-based and random selection methods are ineffective when positive samples are extremely scarce.
Tried and failed
active learning with ensemble exploration applied to severe class imbalance classification. Outcome: worse than baseline. Reason: positive samples were too scarce to justify exploration over pure exploitation
Data Exchange for Artificial Intelligence Incubation in Manufacturing Industrial Internet · Virginia Tech
Tried and failed
unsupervised active learning and uncertainty sampling applied to severely imbalanced classification. Outcome: data insufficient. Reason: sampling strategy failed to acquire sufficient minority class instances under extreme class imbalance
Machine Learning for VLSI Computer Aided Design · UT Austin
Tried and failed
uncertainty-based active learning applied to cold-start few-shot text classification. Outcome: worse than baseline. Reason: severe class imbalance and poorly calibrated model uncertainty during cold-start few-shot regimes
Towards Efficiently and Reliably Harnessing Pre-trained Language Models: A Data-centric Lens · Georgia Tech
Active learning is sensitive to noise, outliers, and intermediate model variance
Active learning methods can be derailed by noisy data or high variance in intermediate model checkpoints during training. Greedy selection strategies are particularly vulnerable to sampling unphysical edge-cases or misleading outliers.
Lost to a baseline
In some individual in vivo sessions, random sampling outperformed active learning because querying unmodeled noisy data points misled subsequent sample selection.
An active learning framework for quantifying the effect of neuromodulation · Georgia Tech
Considered and rejected
Considered and rejected: Rejected Active Learning (AL) for data rebalancing due to poor handling of high noise/fluctuations in brittle cracking simulations
Real-time assessment of tunnelling-induced damage to structures within the Building Information Modelling Framework NA · University of Nottingham Repository
Considered and rejected
Considered and rejected: Rejected naive greedy selection in active learning because variance across training checkpoints and stochastic optimization induces severe outlier sensitivity.
Tried and failed
greedy active learning data acquisition applied to neural network training. Outcome: worse than baseline. Reason: outlier sensitivity and variance in intermediate neural network models degrade acquired sample quality
Tried and failed
active learning with equivariant neural network force fields applied to molecular adsorption property prediction. Outcome: did not generalise. Reason: unphysical edge-case samples entered the calibration pool, preventing convergence and inflating validation error
Adsorption Isotherm Prediction of Diverse Adsorbates in Metal–Organic Frameworks using Machine Learning · Georgia Tech
Tried and failed
deterministic active learning with spectral embedding applied to stochastic simulation optimization. Outcome: unstable. Reason: broke down under significant non-stationary stochastic simulation noise
Accelerated Simulation-Based Analysis of Emergent and Stochastic Behavior in Military Capability Design · Georgia Tech
Active learning metrics derived from structural constraints or heuristics fail to capture true informativeness
Active learning strategies based on structural constraints, specific clustering initializations, or heuristic parameters often perform poorly. These methods can limit the capacity to match optimal distributions or degrade the representative diversity of the selected samples.
Tried and failed
matrix symmetry as active learning uncertainty metric applied to active learning query selection. Outcome: worse than baseline. Reason: caused overfitting and degraded performance compared to standard active learning sampling
Machine Learning Approaches to Predicting Postoperative Hemodynamics After Cardiac Surgeries · Georgia Tech
Lost to a baseline
Plain GAAL using symmetry metric (82.65% accuracy) was beaten by baseline active learning using symmetry on unaugmented data (83.39% accuracy).
Machine Learning Approaches to Predicting Postoperative Hemodynamics After Cardiac Surgeries · Georgia Tech
Tried and failed
Feedback coding for active learning query selection applied to Bayesian logistic regression. Reason: Unlabeled samples cannot exactly match the theoretical capacity-achieving input distribution under log-concave priors
Feedback Coding for Efficient Interactive Machine Learning · Georgia Tech
Tried and failed
multidimensional posterior matching feedback coding applied to active learning query selection. Reason: structural model constraints prevent finding examples that induce the exact capacity-achieving posterior distribution
Feedback Coding for Efficient Interactive Machine Learning · Georgia Tech
Considered and rejected
Considered and rejected: Direct multidimensional posterior matching in active learning rejected due to structural encoder constraints.
Feedback Coding for Efficient Interactive Machine Learning · Georgia Tech
Considered and rejected
Considered and rejected: Rejected direct multidimensional posterior matching in active learning because structural encoder constraints prevent exact matching to the capacity-achieving distribution.
Feedback Coding for Efficient Interactive Machine Learning · Georgia Tech
Considered and rejected
Considered and rejected: Rejected hand-engineered heuristics for active learning informativeness, replacing them with a meta-learned acquisition function to prevent domain-specificity breakdown and inaccurate proxies.
Data-driven personalization techniques to account for heterogeneity in human-machine interaction · Georgia Tech
Distribution shift between training phases or domains misguides active learning sample selection
When there is a significant distribution shift between source and target domains, active learning can become biased during early sample selection. This bias prevents the model from recovering and adapting to the new distribution effectively.
Considered and rejected
Considered and rejected: Rejected relying on non-adaptive (fixed) acquisition proposals for active testing under distribution shift, as both LURE and ASE perform poorly without retraining the auxiliary model.
Tried and failed
cross-domain warm starting in active learning applied to tabular predictive modeling across sites. Outcome: worse than baseline. Reason: Distribution shift between source and target domains biased early sample selection, preventing recovery.
Lost to a baseline
On the distribution shift experiment, LURE using XWED or BALD acquisition proposals fails to outperform the naive MC baseline due to high variance.
The computational overhead of active learning outweighs the benefits
The process of repeatedly evaluating acquisition functions and retraining models during active learning can introduce severe computational bottlenecks. In some cases, the runtime becomes prohibitively slow, negating any potential efficiency gains over simpler methods.
Left open
Apply combinatorial coverage density sampling to deep neural network active learning to reduce retraining overhead. Blocker: None
Active Learning with Combinatorial Coverage · Virginia Tech
Tried and failed
projected gradient descent uncertainty maximization active learning applied to active learning sample selection. Outcome: too slow. Reason: Trapped in local optima, requiring an order of magnitude more computation without accuracy improvements
Fiber-Wireless Integration with Enhanced Adaptability for Next Generation Radio Access Networks · Georgia Tech
Tried and failed
gradient-based clustering for active learning applied to text classification. Outcome: too slow. Reason: extremely high computational overhead without outperforming simpler selection baselines
Probabilistic language models with model efficiency and data efficiency · UT Austin
Tried and failed
Fine-grained demographic subgroup fairness constraints applied to Multimodal active learning. Outcome: worse than baseline. Reason: Granular group partitioning degraded model performance and caused prohibitively long active selection runtimes.
Left open by the authors
Problems the authors named and did not get to.
Left open
Evaluate the reproducibility of active learning versus random sampling across multi-patient cohorts using public Restoring Active Memory (RAM) project data. Blocker: None
An active learning framework for quantifying the effect of neuromodulation · Georgia Tech
Left open
Investigate and mitigate the imbalanced selection of blank, easy, and hard samples during active learning training in medical image analysis. Blocker: None
Weakly-supervised Learning for Cost-Effective Medical Image Analysis Weakly-Supervised learning for Label-Effective Medical Image Analysis · Research Repository UCD
Left open
Develop an active learning framework for ALS point cloud classification to automatically select diverse training samples across spatial regions. Blocker: None
Contextual semantic classification of ALS point clouds in urban environment · DSpace-CRIS at TU Wien
Left open
Analyze properties of inverse design benchmark problems (like ADM) that cause active learning strategies to fail or underperform. Blocker: The task is an open-ended theoretical direction rather than a concrete software or algorithmic specification
Deep Learning for the modeling and design of artificial electromagnetic materials · DukeSpace
Left open
Evaluate Bayesian deep ensembles with randomized Jacobian-vector products on active learning and reinforcement learning benchmarks. Blocker: None
Left open
Derive a theoretical stopping criterion for active learning to determine the minimum number of demonstrations required for generalization. Blocker: No concrete theoretical framework or approach is specified for deriving the stopping threshold
Learning and optimization of anticipatory feedback controllers for robot manipulation · EPFL
Left open
Study how to customize synthetic data generation to enhance active learning performance. Blocker: The goal and method for customizing synthetic data generation are entirely unstated
Active Learning Under Limited Interaction with Data Labeler · Virginia Tech
Left open
Implement differentiable sampling with adversarial uncertainty attacks in active learning to improve ML-predicted excited state lifetimes. Blocker: None
Accelerating drug discovery with quantum chemistry, machine learning, and molecular dynamics · Harvard
Left open
Implement GraphSAGE or GraphSAINT sampling procedures to scale GCN-based active learning samplers instead of random subset pre-sampling. Blocker: None
Parametric active learning techniques for 3D hand pose estimation · Imperial
Left open
Adapt myopic margin uncertainty sampling into a batch-aware active learning algorithm and evaluate it on standard benchmark datasets. Blocker: None
Optimizing resource allocation in computational sustainability: Models, algorithms and tools · Georgia Tech
Left open
Design a model-data dependent active learning acquisition algorithm for noisy labelers. Blocker: The unfinished work lacks specific target metrics, mathematical formulations, or explicit architectural requirements.
Advancing human-AI systems : on robustness, decision making, and beyond · UT Austin
Left open
Implement an active learning strategy to query training data at high-uncertainty operating points for GP-based optimal power flow approximations. Blocker: None
Gaussian Processes for Power System Monitoring, Optimization, and Planning · Virginia Tech
Checking an idea in your area?
We can run the same search on any method or claim. If nothing turns up, we will say so, and that proves nothing on its own.