Jesse van Remmerden

Jesse van Remmerden

PhD candidate at TU/e. Reinforcement learning, diffusion models and LLMs for scheduling, routing and maintenance planning.

Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutions

Published in Transactions on Machine Learning Research, 2026

Project page: figures, results and code

Authors & Affiliations
Jesse van Remmerden, Zaharah Bukhsh, Yingqian Zhang — Information Systems, Department of Industrial Engineering & Innovation Sciences, Eindhoven University of Technology (TU/e), The Netherlands.
Corresponding author: j.v.remmerden@tue.nl

Summary

Online deep reinforcement learning has become the dominant learning-based approach to job shop scheduling, but it carries a demanding prerequisite: a simulator faithful enough to support millions of trial-and-error interactions. For realistic production environments such simulators are costly to build and frequently do not exist at all.

CDQAC (Conservative Discrete Quantile Actor–Critic) removes that prerequisite. It learns scheduling policies entirely offline, from a fixed set of previously executed schedules, and improves upon the quality of the data it was trained on. Three findings stand out:

  • Sample efficiency. CDQAC reaches state-of-the-art performance from only 10–25 training instances, where online RL methods require millions of simulated interactions.
  • Generalization. Policies trained on small instances transfer to problems with ten times as many jobs, outperforming industrial solvers in that regime.
  • Coverage beats quality. CDQAC performs best when trained on data generated by a random heuristic, outperforming training on higher-quality genetic algorithm and priority dispatch rule data. The rank correlation between the quality of the data-generating heuristic and the quality of the resulting policy is −0.95: what matters is how much of the decision space the dataset covers, not how good its solutions are.

The third finding is the one with reach beyond scheduling. It inverts the standard intuition that offline RL datasets should be as close to expert as possible, and it supplies a concrete and cheap prescription for building training data for learned decision-making systems.

Abstract
The Job-Shop Scheduling Problem (JSP) and Flexible Job-Shop Scheduling Problem (FJSP) are canonical combinatorial optimization problems with wide-ranging applications in industrial operations. Recent online RL approaches require millions of simulated interactions and suffer from poor sample efficiency. We propose Conservative Discrete Quantile Actor–Critic (CDQAC), a novel offline RL algorithm that learns effective scheduling policies directly from historical data, eliminating online interactions while improving upon suboptimal training data. CDQAC couples a quantile-based critic with a delayed policy update to estimate the return distribution of each machine–operation pair. Extensive experiments show that CDQAC outperforms the data-generating heuristics and state-of-the-art offline and online RL baselines, and is highly sample efficient—needing only 10–25 training instances. Surprisingly, CDQAC performs best when trained on data generated by a random heuristic, outperforming training on higher-quality GA and PDR data.

Method
CDQAC couples a quantile-based critic, which estimates the full return distribution of each machine–operation pair rather than a point estimate, with a delayed policy update that stabilizes learning under distributional shift. Conservative Q-learning keeps value estimates pessimistic on out-of-distribution actions, which is what permits the policy to exceed the behaviour that generated its data without exploiting value overestimation.

Keywords
Offline reinforcement learning; Job shop scheduling; Flexible job shop scheduling; Distributional RL; Quantile critic; Conservative Q-learning; Neural combinatorial optimization

Links

BibTeX

@article{vanremmerden2026generalizing,
  title   = {Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutions},
  author  = {van Remmerden, Jesse and Bukhsh, Zaharah and Zhang, Yingqian},
  journal = {Transactions on Machine Learning Research},
  year    = {2026},
  url     = {https://openreview.net/forum?id=4vVhm4817Q}
}

Recommended citation: van Remmerden, J., Bukhsh, Z., & Zhang, Y. (2026). Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutions. Transactions on Machine Learning Research.
Download Paper