Benchmark: A Model That Calls Back Into Python

What a user-defined function costs on the Rust engine: the crossing into Python, and the work the function does once it is there.

Keywords

benchmark, performance, rust, execution engine, user-defined functions, system dynamics, bptk, bptk-py, python

Benchmark: A Model That Calls Back Into Python

A model with user-defined functions runs on the Rust engine, which calls back into Python at those nodes — once per node and timestep. This page measures what that costs.

The axis is neither of the other two. The scalar benchmark asks what a longer run gains and the arrayed benchmark what a wider model costs; this one asks what a callback costs - the crossing into Python, and the work the function does once it is there.

The model

The workforce chain of the arrayed benchmark at twelve levels, with a market wage per level: the wage at which a labour pool fills that level’s openings. The pool’s supply rises with the wage along a curve no formula inverts, so the wage is found by bisection - a loop, which an equation cannot express, and so a function you would actually write in Python:

def clearing_wage(openings, pool, reference, iterations):
    """The wage at which `pool` supplies `openings` people."""
    def supply(wage):
        x = wage / reference
        return pool * (1.0 - math.exp(-x)) * (1.0 + 0.5 * math.sin(x))

    low, high = 0.0, 2.0 * reference
    for _ in range(iterations):
        middle = (low + high) / 2.0
        if supply(middle) < openings:
            low = middle
        else:
            high = middle
    return (low + high) / 2.0


openings.equation = promotion + attrition
hiring_cost.equation = hiring * wage

wage is arrayed, so a function computing it is called once per level - twelve calls a timestep. The benchmark builds the model once per variant and times each build on both engines; the builds differ only in the equation for wage:

for variant in ("expression", "crossing only", 10, 40, 160):
    model = build(levels=12, stoptime=steps, variant=variant)
    ...  # run it on the Python engine and on the Rust engine, compare, time


# inside build(), the part that depends on the variant:
if variant == "expression":
    # no function at all: an equation the engine evaluates by itself
    wage.equation = salary * 1.1
elif variant == "crossing only":
    # a function that does nothing but return its argument: what the call alone costs
    market_wage = model.function(
        "market_wage", lambda model, t, openings, pool, reference: reference)
    wage.equation = market_wage(openings, pool, salary)
else:
    # the real thing, with 10, 40 or 160 bisection steps
    market_wage = model.function(
        "market_wage", lambda model, t, openings, pool, reference:
        clearing_wage(openings, pool, reference, variant))
    wage.equation = market_wage(openings, pool, salary)

So the first row is the engine at its own speed, the second adds the crossing into Python and back, and the bisection rows add the work the function does once it is there.

Both runs go through run_scenarios; only the backend argument differs. Median of three runs, each on a freshly built model. Both engines are compared, not just timed. The script is beside this page: callback_benchmark.py.

Results

Apple M3, Python 3.13:

Timesteps	Wage computed by	Python engine	Rust engine	Speedup
400	expression	241.2 ms	4.6 ms	52.8×
400	crossing only	288.5 ms	6.2 ms	46.8×
400	bisection, 10 iterations	278.4 ms	12.2 ms	22.9×
400	bisection, 40 iterations	296.6 ms	26.2 ms	11.3×
400	bisection, 160 iterations	350.4 ms	82.1 ms	4.3×
4,000	expression	2,256.8 ms	29.4 ms	76.6×
4,000	crossing only	2,711.9 ms	43.4 ms	62.4×
4,000	bisection, 10 iterations	2,773.5 ms	100.9 ms	27.5×
4,000	bisection, 40 iterations	2,910.2 ms	240.7 ms	12.1×
4,000	bisection, 160 iterations	3,478.9 ms	795.8 ms	4.4×

The two engines agree exactly. The largest absolute difference across every equation of every row is zero.

Reading the table

The crossing is cheap. At 4,000 timesteps the function that only returns its argument adds 14 ms to the engine’s run over about 48,000 calls: 0.3 microseconds a call. That is the price of going from one runtime to the other and back.

The function is not. Each bisection step costs about 0.1 microseconds, so 160 of them are 15.7 microseconds a call - fifty times the crossing. At 4,000 timesteps the engine’s run grows from 43 ms to 796 ms, while the Python engine’s grows by 28 %. On the Python engine the function is one Python call among many; on the Rust engine it is the only part that is not native, and it becomes the floor of the run. That is what takes the speedup from 62× down to 4×, and nothing else does.

The engine cannot make Python faster. What runs inside a user-defined function runs at Python’s speed on either engine. Two things help: keep in the DSL whatever an equation can express - the expression row is the engine at its own speed - and make the function itself cheap. Even at 160 iterations the model around the function still runs natively, and the engine is four times faster than the Python engine.