Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutions

Offline RL for scheduling, without a simulator or expert data

Online reinforcement learning methods for job shop scheduling, such as L2D and DANIEL, learn by interacting with a simulated shop floor, often for millions of steps. That is slow, and a simulator rarely captures every detail of a real factory.

Offline RL learns from a fixed dataset instead, for example the schedules a plant has already run. Because it learns what each decision is worth rather than copying the decisions it sees, it can end up better than the data it was trained on. Our earlier method, Offline-LD, showed that this works for JSP, but it relied on near-optimal solutions from a constraint programming solver, which are expensive to produce and intractable for large instances. CDQAC drops that requirement: it learns strong policies for both JSP and FJSP from suboptimal data, down to schedules made at random.

How CDQAC works

CDQAC builds a schedule one decision at a time. At every step it looks at the partial schedule and picks a machine–operation pair: which operation runs next, and on which machine. The reward for a step is the negative increase in the partial makespan, so the rewards of a finished schedule add up to exactly minus its makespan.

Conservative Q-learning keeps unseen actions in check

An offline agent never sees what happens after actions that are missing from its dataset, so a naive critic can overestimate them and the policy will chase those errors. CDQAC adds a Conservative Q-Learning (CQL) penalty that pushes down the value of pairs outside the data.

A quantile critic for every machine–operation pair

A Dual Attention Network (DAN) encoder, adapted from DANIEL, embeds the operations and machines. On top of it, the critic predicts a distribution over returns, as a set of quantiles in the style of QR-DQN, for each candidate pair instead of a single value. It uses a dueling design with separate inputs: the value stream sees a global embedding of the schedule, and the advantage stream sees the operation, machine and pair embeddings. That split suits scheduling, where the number of candidate pairs changes at every step.

Delayed policy updates

The actor, which is the scheduling policy itself, is updated only once every η critic updates. This stops errors in the bootstrapped critic from compounding into the policy. In the ablation study this is the component that matters most, and it matters more on larger instances with larger action spaces.

Why random data works: coverage beats quality

On standard offline RL benchmarks, expert data beats random data. In scheduling we found the opposite, and traced it to two structural properties:

• A dense reward aligned with the objective. Every decision receives a reward that is a calibrated share of the final makespan, so even a bad schedule tells the critic what each of its steps cost.
• Equal-length trajectories. Every heuristic, good or bad, makes exactly one decision per operation, so all trajectories for an instance have the same length and none is favored for being shorter.

With both properties in place, what limits offline RL is state–action coverage: how many different situations and decisions the data contains. A random policy maximizes coverage by construction.

The paper tests this explanation directly. Breaking either property, by cutting episodes short or by switching to a sparse or proxy reward, hurts the random dataset the most.

Results on job shop and flexible job shop benchmarks

Flexible job shop (FJSP): the weakest data gives the best policy

We trained CDQAC and five other offline methods (behavior cloning, IQL, TD3-AWR and two Offline-LD variants) on four datasets, each produced by a different heuristic: priority dispatching rules (PDR), a genetic algorithm (GA), both combined, or uniformly random decisions. Averaged over all FJSP evaluation sets, CDQAC beats every other method on every dataset, and its best result comes from the random data, even though the random heuristic itself is by far the weakest.

Against online reinforcement learning

CDQAC trained on random data also surpasses the state-of-the-art online RL baselines on the FJSP and JSP benchmarks, including DANIEL, the online method whose encoder it builds on. On the Taillard JSP benchmark:

Mean gap (%) on the Taillard benchmark, lower is better. For reference, exact MIP and CP solvers with a 30-minute limit per instance reach 8.4% and 2.0%. CDQAC was trained on the random 10×5 dataset. The paper also compares further JSP-specific methods.

What each component contributes

Removing the delayed policy update hurts most: the gap grows by 54.3% on average, and by 112.8% when training on 20×10 instances. The dueling architecture shows the same pattern at a smaller scale (8.9% on average, 16.7% on 20×10), while removing the quantile critic or the CQL penalty costs 4.2% and 2.5% on average.

Sample efficiency

CDQAC reaches competitive performance with 10 to 25 training instances, typically 1 to 5% of the data that online RL baselines train on. Dataset size barely matters: from 5 to 500 random training instances, the gap on the Brandimarte and Hurink FJSP benchmarks stays around 6% with sampling and moves by at most about one and a half percentage points with greedy decoding.

Code, datasets and pretrained checkpoints

Everything needed to reproduce the paper is public. The code on GitHub (MIT license) trains CDQAC and the offline baselines (behavior cloning, IQL, TD3-AWR and both Offline-LD variants) from a single entry point. The datasets and checkpoints on Hugging Face hold the JSP and FJSP training data from every heuristic (dispatching rules, genetic algorithm, random and ε-greedy dispatching rules), the evaluation instances, and pretrained policies.

git clone https://github.com/jesserem/CDQAC_scheduling
cd CDQAC_scheduling
pip install -r requirements.txt
python download_datasets.py eval checkpoints

# Evaluate a pretrained CDQAC policy (trained on random FJSP data, 10×5)
python eval_model.py checkpoints/fjsp/cdqac/cdqac_random_10_5.pt --eval_folder ./data/SD1/10x5

Cite this paper

@article{remmerden2026generalizing,
    title={Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutions},
    author={Jesse van Remmerden and Zaharah Bukhsh and Yingqian Zhang},
    journal={Transactions on Machine Learning Research},
    issn={2835-8856},
    year={2026},
    url={https://openreview.net/forum?id=4vVhm4817Q}
}

Copy BibTeX

Earlier arXiv versions of this paper were titled “Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Data”.

Related work

• Offline Reinforcement Learning for Learning to Dispatch for Job Shop Scheduling (Offline-LD). J. van Remmerden, Z. Bukhsh, Y. Zhang. Machine Learning 114(8), 191, 2025. Our first offline RL method for JSP, trained on constraint programming solutions.
• Flexible Job Shop Scheduling via Dual Attention Network-Based Reinforcement Learning (DANIEL). R. Wang, G. Wang, J. Sun, F. Deng, J. Chen. IEEE TNNLS, 2023. Source of the DAN encoder and the FJSP environment.
• Job Shop Scheduling Benchmark: Environments and Instances for Learning and Non-learning Methods. R. Reijnen, I. G. Smit, H. Zhang, Y. Wu, Z. Bukhsh, Y. Zhang. Annals of Mathematics and Artificial Intelligence, 2026. Used to generate the genetic-algorithm datasets; code.