Introducing CleanQRL: A Lightweight Library for Quantum Reinforcement Learning

We present CleanQRL, an open-source library that provides concise, single-file implementations of quantum reinforcement learning (QRL) algorithms. Modeled after the widely used CleanRL library for classical reinforcement learning, CleanQRL aims to lower the entry barrier for researchers exploring quantum-enhanced agents.

This library accompanies our recent preprint, "CleanQRL: Lightweight Single-file Implementations of Quantum Reinforcement Learning Algorithms" (arXiv:2507.07593). By standardizing QRL implementations, CleanQRL facilitates reproducible experiments and practical exploration of quantum advantages in reinforcement learning.

1. Addressing Challenges in Quantum Reinforcement Learning

Quantum reinforcement learning combines quantum computing with reinforcement learning, offering potential efficiency gains for complex decision-making problems. Progress in the field has been slowed, however, by the absence of accessible, standardized implementations that allow direct comparison between quantum and classical approaches.

CleanQRL meets this need by supplying minimal, self-contained scripts for both quantum and classical algorithms. The library is designed for straightforward modification and integrates seamlessly with established environments such as Gymnasium, MuJoCo, and Jumanji.

Fig. 1: Overview of CleanQRL architecture and agent implementations.

2. Core Features and Integrations

CleanQRL currently includes single-file implementations of several agents, including quantum variants of REINFORCE. It supports hyperparameter optimization through Ray Tune and experiment tracking via Weights & Biases, enabling rigorous benchmarking against classical baselines.

Additional features include TensorBoard logging, deterministic reproducibility through seeding, and worked examples demonstrating quantum agents on benchmark tasks such as CartPole.

Fig. 2: Example of a quantum agent configuration and training loop in CleanQRL.

3. Benchmarks and Performance

The provided implementations have been carefully tuned and evaluated on standard environments, allowing direct performance comparisons between quantum and classical agents. Support for distributed training enables large-scale experiments and systematic hyperparameter searches.

Fig. 3: Performance benchmarks comparing QRL agents to classical counterparts on standard environments.

Why This Matters

CleanQRL contributes to democratizing quantum reinforcement learning research. By offering clean, extensible code, it helps bridge the gap between theoretical proposals and practical implementation, supporting investigations into quantum advantages for applications ranging from optimization problems to power electronics and beyond.

Researchers are invited to explore the GitHub repository, consult the documentation, and read the accompanying preprint.

Future releases will expand the library with additional algorithms and features.

References

[1] G. Kruse, R. Coelho, A. Rosskopf, R. Wille, and J. M. Lorenz, "CleanQRL: Lightweight Single-file Implementations of Quantum Reinforcement Learning Algorithms," arXiv:2507.07593 [cs.LG], 2025. arXiv Link
[2] CleanQRL GitHub Repository. GitHub Link
[3] CleanQRL Documentation. Docs Link

Introducing CleanQRL: Quantum Reinforcement Learning with Clean, Single-File Implementations

Authors