Training Agents

Training agents over many episodes with train_scenarios - the episode hooks, what survives from one episode to the next, and where learned knowledge has to be kept
Keywords

agent-based modeling, abm, training, episodes, reinforcement learning, q-learning, bptk, bptk-py, python

Training Agents

An agent that learns has to play the same game many times: try something, see what it earned, and do better the next time. Each of these games is an episode, and bptk.train_scenarios runs a scenario for as many episodes as you ask for. It works for agent-based and hybrid models.

The library supplies the episodes. What an agent learns, and how, is up to you: this page builds the simplest learner there is, and Training AI to Play the Beer Game applies Q-learning to a supply chain.

The Episode Hooks

In every episode, train_scenarios calls

  1. begin_episode(episode_no) on the model, which calls it on every agent,
  2. runs the scenario from its start to its stop time - with begin_round and end_round on the model in every round, and act on every agent,
  3. end_episode(episode_no) on the model, which calls it on every agent.

begin_episode is the place to choose what to try, end_episode the place to learn from the result, and end_round the place for a reward that comes during the game.

Two Things to Know Before You Start

Nothing is reset between episodes. The model and its agents are the same from one episode to the next, and so are their properties: a profit that is added up in act goes on adding up across episodes. What has to start afresh, begin_episode sets back itself.

Learned knowledge must not live on the model instance. The scenario manager builds the model again when training starts, and the scenario you run afterwards is a model of its own: an attribute set on the instance during training is no longer there. Keep what the agents learn somewhere that survives - as an attribute of the class, as the beer game keeps its q-tables, or in a module of its own.

A Seller Learns Its Price

A seller can charge 4, 6, 8 or 10 per unit. Demand falls with the price, each unit costs 2, and the seller does not know which price earns most. In every episode it picks a price - usually the best one so far, sometimes one at random - plays five rounds and remembers what that price earned on average:

The scenario is an ordinary agent-based one, with one seller:

Training

train_scenarios takes the scenario, the number of episodes and the agent properties to report - one row per episode, the value at the end of it. Choose how long the seller trains:

For the first episodes the seller sticks to 4, the first price it tried; once a random choice has found 8, it keeps to that, apart from the odd excursion. What it has learned is on the class - the average profit of each price, and how often it was tried:

Keeping What Was Learned

Training can take hours, so what the agents learned is worth keeping. That is ordinary Python - json or pickle on the learned structure. The beer game writes its q-tables with dump_q_tables and reads them back with load_q_tables, so that its pages can use agents trained for 50,000 episodes.

progress_bar=True shows how far the training has got; in the browser there are no progress bars - see Limitations. Every parameter of the method is on bptk.train_scenarios.