Benchmark: A Model That Calls Back Into Python

What a user-defined function costs on the Rust engine, measured over how many of them a model has.

Keywords

benchmark, performance, rust, execution engine, user-defined functions, system dynamics, bptk, bptk-py, python

Benchmark: A Model That Calls Back Into Python

A model with user-defined functions runs on the Rust engine, which calls back into Python at those nodes — once per node and timestep. This page measures what that costs.

The axis is neither of the other two. The scalar benchmark asks what a longer run gains and the arrayed benchmark what a wider model costs; this one asks what a callback costs, over how many of them there are.

The model

The workforce chain of the arrayed benchmark at twelve levels, plus a hundred scalar converters that all compute the same thing. The parameter is how many of the hundred are computed by a Python function rather than by an expression, so the two versions of the model differ in nothing else:

uplift = model.function("uplift", lambda model, t, value: value * 1.05)

for i in range(100):
    converter = model.converter(f"extra_{i}")
    if i < callbacks:
        converter.equation = uplift(total_headcount)   # a call into Python
    else:
        converter.equation = total_headcount * 1.05    # the same thing, as an expression

Both runs go through run_scenarios; only the backend argument differs. Median of three runs, each on a freshly built model. Both engines are compared, not just timed.

Results

Apple M3, Python 3.13:

Timesteps	Callbacks	Python engine	Rust engine	Speedup
400	0 %	181.7 ms	3.1 ms	58.7×
400	1 %	174.5 ms	3.0 ms	57.7×
400	10 %	176.3 ms	4.2 ms	41.6×
400	50 %	177.4 ms	8.6 ms	20.6×
400	100 %	178.3 ms	14.1 ms	12.6×
40,000	0 %	18,348.5 ms	155.2 ms	118.2×
40,000	1 %	18,300.1 ms	169.2 ms	108.1×
40,000	10 %	18,295.7 ms	258.8 ms	70.7×
40,000	50 %	18,352.7 ms	639.4 ms	28.7×
40,000	100 %	18,004.8 ms	1,172.0 ms	15.4×

The two engines agree exactly. The largest absolute difference across every equation of every row is zero.

Reading the table

A callback costs about a quarter of a microsecond. At 40,000 timesteps, ten callbacks add 103.5 ms over roughly 400,000 calls, and a hundred add 1,016.8 ms over roughly 4,000,000 — 0.26 and 0.25 microseconds each. That is the price of crossing between the two runtimes, paid once per call.

The price is visible. The same model over 40,000 timesteps takes 155 ms with no callbacks and 1,172 ms when all hundred converters are Python functions: 7.5× slower on the engine. That is what makes the speedup column fall from 118× to 15×, and nothing else does — the Python column is flat, because on that engine a user-defined function is one Python call among many.

The advantage does not disappear. Even with every one of the hundred converters computed in Python, the engine is 15× faster over a long run. The model around the callbacks keeps running natively: the chain is 279 entities the engine evaluates by itself, and only the hundred nodes cross over. Work moved out of a user-defined function and into an equation is work the engine does at its own speed.