Offline Reinforcement Learning for Learning to Dispatch for Job Shop Scheduling

Authors
Jesse van Remmerden, Zaharah Bukhsh, Yingqian Zhang
Information Systems (IE&IS), Eindhoven University of Technology, Eindhoven, The Netherlands.
Corresponding author: j.v.remmerden@tue.nl

Published in Machine Learning 114(8), 191 (Springer). Presented in the Journal Track of ECML PKDD 2025, Porto, 15–19 September 2025.

Summary

Online reinforcement learning can find acceptable job shop schedules quickly, but it learns by trial and error inside a simulator. That requirement is the binding constraint in practice: faithful simulators of real production environments are expensive to build and often do not exist, and starting from scratch discards the high-quality solutions that constraint programming and other established methods have already produced.

Offline-LD is the first fully end-to-end offline RL approach for the JSSP. It learns dispatching policies purely from a fixed set of previously executed schedules, with no environment interaction whatsoever.

Method. Offline-LD introduces maskable variants of two Q-learning methods — Maskable Quantile Regression DQN (mQRDQN) and discrete maskable Soft Actor–Critic (d-mSAC) — trained through Conservative Q-Learning. Making these methods work under the JSSP’s action-masking structure required two new components: a novel entropy bonus modification for d-mSAC adapted to maskable action spaces, and a reward normalisation scheme for the offline scheduling setting.

Key results

• Outperforms online RL on both generated and standard benchmark instances when trained on only 100 solutions generated by constraint programming.
• Adding noise to the expert dataset yields comparable or superior results to using the clean expert dataset of the same size — a direct indication that these methods are viable on real industrial logs, which are inherently noisy and imperfect.

That last observation is what motivated the follow-up work, CDQAC (project page), which takes the premise to its conclusion: training on entirely random solutions, and beating simulator-trained methods anyway.

Abstract
The Job Shop Scheduling Problem (JSSP) is a complex combinatorial optimization problem. While online Reinforcement Learning (RL) has shown promise by quickly finding acceptable solutions for JSSP, it faces key limitations: it requires extensive training interactions from scratch leading to sample inefficiency, cannot leverage existing high-quality solutions from traditional methods like Constraint Programming (CP), and require simulated environments to train in, which are impracticable to build for complex scheduling environments. We introduce Offline Learned Dispatching (Offline-LD), an offline reinforcement learning approach for JSSP, which addresses these limitations by learning from historical scheduling data. Our approach is motivated by scenarios where historical scheduling data and expert solutions are available or scenarios where online training of RL approaches with simulated environments is impracticable. Offline-LD introduces maskable variants of two Q-learning methods, namely, Maskable Quantile Regression DQN (mQRDQN) and discrete maskable Soft Actor-Critic (d-mSAC), that are able to learn from historical data, through Conservative Q-Learning (CQL), whereby we present a novel entropy bonus modification for d-mSAC, for maskable action spaces. Moreover, we introduce a novel reward normalization method for JSSP in an offline RL setting. Our experiments demonstrate that Offline-LD outperforms online RL on both generated and benchmark instances when trained on only 100 solutions generated by CP. Notably, introducing noise to the expert dataset yields comparable or superior results to using the expert dataset, with the same amount of instances, a promising finding for real-world applications, where data is inherently noisy and imperfect.

Keywords
Job Shop Scheduling; Reinforcement Learning; Offline RL; Maskable action spaces; Conservative Q-Learning; mQRDQN; d-mSAC

Links

• DOI: https://doi.org/10.1007/s10994-025-06826-w
• arXiv: https://arxiv.org/abs/2409.10589
• Code: https://github.com/jesserem/Offline-LD

BibTeX

@article{vanRemmerdenOfflineJssp2025,
	abstract = {The Job Shop Scheduling Problem (JSSP) is a complex combinatorial optimization problem. While online Reinforcement Learning (RL) has shown promise by quickly finding acceptable solutions for JSSP, it faces key limitations: it requires extensive training interactions from scratch leading to sample inefficiency, cannot leverage existing high-quality solutions from traditional methods like Constraint Programming (CP), and require simulated environments to train in, which are impracticable to build for complex scheduling environments. We introduce Offline Learned Dispatching (Offline-LD), an offline reinforcement learning approach for JSSP, which addresses these limitations by learning from historical scheduling data. Our approach is motivated by scenarios where historical scheduling data and expert solutions are available or scenarios where online training of RL approaches with simulated environments is impracticable. Offline-LD introduces maskable variants of two Q-learning methods, namely, Maskable Quantile Regression DQN (mQRDQN) and discrete maskable Soft Actor-Critic (d-mSAC), that are able to learn from historical data, through Conservative Q-Learning (CQL), whereby we present a novel entropy bonus modification for d-mSAC, for maskable action spaces. Moreover, we introduce a novel reward normalization method for JSSP in an offline RL setting. Our experiments demonstrate that Offline-LD outperforms online RL on both generated and benchmark instances when trained on only 100 solutions generated by CP. Notably, introducing noise to the expert dataset yields comparable or superior results to using the expert dataset, with the same amount of instances, a promising finding for real-world applications, where data is inherently noisy and imperfect.},
	author = {Remmerden, Jesse van and Bukhsh, Zaharah and Zhang, Yingqian},
	date = {2025/07/15},
	doi = {10.1007/s10994-025-06826-w},
	id = {Remmerden2025},
	isbn = {1573-0565},
	journal = {Machine Learning},
	number = {8},
	pages = {191},
	title = {Offline reinforcement learning for learning to dispatch for job shop scheduling},
	url = {https://doi.org/10.1007/s10994-025-06826-w},
	volume = {114},
	year = {2025},

Recommended citation: van Remmerden, J., Bukhsh, Z., & Zhang, Y. (2025). Offline reinforcement learning for learning to dispatch for job shop scheduling. Machine Learning, 114(8), 191. https://doi.org/10.1007/s10994-025-06826-w
Download Paper