The benchmark in the ReadMe seems to execute a too slow pure-python version

[Tortar]

I stumbled across this repository, and while the approach seems interesting, actually the benchmark compares apple-to-oranges because the pure Python version is way too slow than what is expected. The running time seems also quadratic, which doesn't make a lot of sense to me because the money model seems linear in the number of agents. By digging into the source code I think that the culprit is at https://github.com/adamamer20/mesa-frames/blob/main/docs/scripts/readme_plot.py#L31 because self.model.schedule.agents creates a copy of the structure containing the agents, which makes the model much slower than what it should be, and explains also the quadratic behaviour.

Indeed an Agents.jl version of the model is still like 40x faster than Polars:

julia> using Agents, Random

julia> @agent struct WealthAgent(NoSpaceAgent)
           wealth::Int
       end

julia> function wealth_model(; numagents = 100, initwealth = 1)
           model = ABM(WealthAgent; agent_step!, scheduler = Schedulers.Randomly(), 
                       rng = Xoshiro(42), container = Vector)
           for _ in 1:numagents
               add_agent!(model, initwealth)
           end
           return model
       end
wealth_model (generic function with 1 method)

julia> function agent_step!(agent, model)
           agent.wealth == 0 && return
           agent.wealth -= 1
           random_agent(model).wealth += 1
       end
agent_step! (generic function with 1 method)

julia> m = wealth_model(; numagents=9000);

julia> @time step!(m, 1);
  0.067163 seconds (94.95 k allocations: 4.839 MiB, 99.34% compilation time)

julia> @time step!(m, 100);
  0.011685 seconds

Hope this helps you to find a better benchmark :-)

[rht]

I simplified the schedule to be just a list of agents and do shuffle / choice on the list. Got this
. From the scaling, it's definitely worth it to use mesa-frames for much higher number of agents still, but at least 10k of agents. And this gives indication that the DataFrame version of Agents.jl would be even faster.

[Tortar]

Yes, this seems much more what is expected. I'd like to see the speed-up with e.g. 500k though. This is what I see in Agents.jl

julia> m = wealth_model(; numagents=500000);

julia> @time step!(m, 100);
  1.785942 seconds (2 allocations: 3.815 MiB, 0.44% gc time)

And this gives indication that the DataFrame version of Agents.jl would be even faster.

I don't think so, at least if we compare single core dataframe vs. current approach of Agents. I think the fact that Agents.jl is 40x faster than Polars gives the opposite indication

[rht]

I can see that a vectorized+immutable Julia code could have more allocations in it (i.e. could be slightly slower), but what about the possibility of running a model on a GPU?

[Tortar]

yes, this would give a clear speed-up in my opinion, but I'm not sure by how much, probably it will depend on the operations.

Also the fact that one is able to use multiple-cores on a CPU with a DataFrame-model is nice, but I'm unsure if a DataFrame is the best semantic, for once it is restrictive to how the model can be constructed, secondly vectorized+immutable as you say will probably have some perf penalty actually.

[adamamer20]

@Tortar Thank you very much for digging deeper into the mesa benchmark! I personally just copied the examples from the mesa docs and didn't check what the issues were with the performance.
I heard of Agents.jl but didn't know it was this fast, that's great! Agents.jl definitely gives more flexibility in model construction and already has excellent timing but I hope that in the future mesa-frames won't be chosen just out of performance but also ease of use.
I think using DataFrames to store agents information is fairly intuitive and allows users to easily "see" how the agents are evolving. The API for pandas and polars already supports many optimized methods out-of-the-box, which should make developing custom methods relatively quick (though thinking in terms of vectorized operations rather than loops can sometimes be challenging). Then there are extension packages for plotting, GPU integration, spatial operations, etc. Polars also has some useful features like handling larger-than-RAM DataFrames and lazy evaluations.
I guess after #6 is closed, we can start implementing more examples to have more useful benchmarks beyond this simple model.
Here are the benchmarks from 100k to 1M agents with mesa-frames. I think the results with polars are pretty good, and there are some potential optimizations that could speed up the concise version further (though the current focus is a stable API, with performance optimizations planned for the future).

Execution times:
---------------
mesa-frames (pl concise):
  Number of agents: 100000, Time: 0.77 seconds
  Number of agents: 200000, Time: 1.29 seconds
  Number of agents: 300000, Time: 1.95 seconds
  Number of agents: 400000, Time: 2.58 seconds
  Number of agents: 500000, Time: 4.84 seconds
  Number of agents: 600000, Time: 5.47 seconds
  Number of agents: 700000, Time: 6.00 seconds
  Number of agents: 800000, Time: 6.97 seconds
  Number of agents: 900000, Time: 9.58 seconds
  Number of agents: 1000000, Time: 11.97 seconds
---------------
---------------
mesa-frames (pl native):
  Number of agents: 100000, Time: 0.33 seconds
  Number of agents: 200000, Time: 0.59 seconds
  Number of agents: 300000, Time: 0.89 seconds
  Number of agents: 400000, Time: 1.19 seconds
  Number of agents: 500000, Time: 1.55 seconds
  Number of agents: 600000, Time: 1.86 seconds
  Number of agents: 700000, Time: 2.56 seconds
  Number of agents: 800000, Time: 3.14 seconds
  Number of agents: 900000, Time: 3.70 seconds
  Number of agents: 1000000, Time: 4.37 seconds
---------------
---------------
mesa-frames (pd concise):
  Number of agents: 100000, Time: 2.40 seconds
  Number of agents: 200000, Time: 4.61 seconds
  Number of agents: 300000, Time: 7.24 seconds
  Number of agents: 400000, Time: 9.66 seconds
  Number of agents: 500000, Time: 12.82 seconds
  Number of agents: 600000, Time: 15.00 seconds
  Number of agents: 700000, Time: 17.31 seconds
  Number of agents: 800000, Time: 20.11 seconds
  Number of agents: 900000, Time: 26.35 seconds
  Number of agents: 1000000, Time: 29.73 seconds
---------------
---------------
mesa-frames (pd native):
  Number of agents: 100000, Time: 1.64 seconds
  Number of agents: 200000, Time: 3.48 seconds
  Number of agents: 300000, Time: 5.52 seconds
  Number of agents: 400000, Time: 7.33 seconds
  Number of agents: 500000, Time: 9.37 seconds
  Number of agents: 600000, Time: 11.41 seconds
  Number of agents: 700000, Time: 13.15 seconds
  Number of agents: 800000, Time: 15.11 seconds
  Number of agents: 900000, Time: 19.59 seconds
  Number of agents: 1000000, Time: 21.89 seconds
---------------