Reinforcement learning trading environments, from someone who has built the bots
I build market-simulation environments for training and evaluating trading agents, with tasks and graders built to your spec. I have built an LLM trading bot with its own runtime, a PPO-based RL trader with a literature survey behind it, and a long-running research wiki on machine learning for markets. The environments grade process, risk rules and robustness, not profit.
Finance is one of the domains AI labs most want environments for, and one of the easiest to get wrong. A trading agent that scores well on one stretch of history has usually learned that stretch, not the market.
What I've built
- An LLM trading bot with a standalone runtime: the model reads market state and proposes actions, and the runtime enforces what it may actually do.
- An RL trader trained with PPO on rented B200 GPUs, backed by a survey of the reinforcement-learning trading literature.
- A research wiki on machine learning for markets, kept up over months.
Earlier work in the same line:
- A MetaTrader 5 bot running in Windows inside Docker, where an LLM decides from market data and news.
- An intraday scanner for narrow-range breakout setups across a national equity market, with LLM analysis of the candidates.
- Commodity arbitrage across regional agricultural markets, from daily price data: an unusual real-world data source.
All of these are private. I don't publish profit and loss figures, and nothing here is investment advice.
One environment in detail
The RL trader's environment shows the shape I build to:
- Causal timing: the agent observes a completed five-minute bar, its order executes at the next bar's open, and it is marked at that bar's close. Nothing from the future leaks in.
- Costs in the reward: the reward is the log change in equity after taker fees, half the spread, deterministic slippage, real funding events and closing out at the end.
- Honest episodes: random seven-day sessions, with insolvency, maximum drawdown and forced settlement as terminal states.
- A sealed test set: training uses 2024 to mid-2025, validation uses fixed episodes from late 2025, and every training environment refuses 2026 data, which is kept as the final test.
- Deterministic validation: 26 fixed seven-day episodes spread across the validation half-year, run at regular intervals during training.
- A rich observation: a chart image read by a frozen vision encoder, plus aligned numeric history with missingness and data-age signals, and the position, account and time state.
What makes a trading environment useful
- Simulation you can trust: fees, slippage, latency and position limits modelled explicitly, so an agent can't win by exploiting the simulator.
- Out-of-sample by construction: tasks drawn from periods and instruments the agent hasn't trained on.
- Graders on process and risk: did the agent follow position limits, stop rules and the stated strategy? Did it behave sensibly in a crash? Profit alone is a noisy and easily gamed reward.
- Tool-using agents, not just policies: tasks where an LLM agent must read data, write and run analysis code, and act through an order interface with guard rails.
What you can buy
Market-simulation RL environments built to your spec: task sets, graders for risk and process, simulators with explicit costs, and runners. Delivered as HUD tasksets or as a repository.
FAQ
Can reinforcement learning be used for trading?
Yes, but results on historical data overstate live performance unless costs, slippage and out-of-sample testing are handled carefully. Environments should make those failure modes visible.
Do you share trading results?
No. I don't publish profit and loss, and the environments I build grade process and risk rather than promising returns.
What do trading environments for LLM agents test?
Whether an agent can read market data, analyse it with code, follow stated risk rules and act through a guarded order interface, across conditions it hasn't seen.
License it, or have me build an environment in the same style.