Elle txns generator doesn't currently support choosing between read/write intensive workload.

[georgyshamteev]

Elle's generator doesn't currently support enabling read/write intensive workload. It just generates ops like this:

let [f (rand-nth [:r :w])

In my opinions it would be interesting and very useful to support some optional flags like:
--read-intensive [rate]
--write-intensive [rate]
Thus, when enabled it would do something like this instead:

let [f (if (< (rand) (:rate opts)) :w :r)]

Or maybe make it one flag, e.g --write-intensity and those who want read intensity would use 1-rate.

I'm ready to make a pull request, if you think this is a useful feature.

[aphyr]

Try it out! Let me know if it finds new bugs! My intuition is that highly read-intensive workloads are unlikely to find new behavior because there's no mutation, and highly write-intensive workloads are unlikely to find bugs because we need lots of concurrent reads to actually see an error.But! Read-only transactions (which is a little different than what you're proposing, yeah?) may go through a special path, so I often wrap Elle's generator in one that rewrites some fraction of ops to be purely reads. Because that's fairly straightforward, and also something that I usually want to bind to *specific processes*, I haven't bothered pushing down additional logic into the list-append generator itself.On Apr 4, 2025 05:24, georgyshamteev ***@***.***> wrote: Elle's generator doesn't currently support enabling read/write intensive workload. It just generates ops like this: let [f (rand-nth [:r :w]) In my opinions it would be interesting to support some optional flags like: --read-intensive [rate] --write-intensive [rate] Thus, when enabled it would do something like this instead: let [f (if (< (rand) (:rate opts)) :w :r)] Or maybe make it one flag, e.g --write-intensity and those who want read intensity would use 1-rate. I'm ready to make a pull request, if you think this is a useful feature.—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> georgyshamteev created an issue (jepsen-io/elle#29) Elle's generator doesn't currently support enabling read/write intensive workload. It just generates ops like this: let [f (rand-nth [:r :w]) In my opinions it would be interesting to support some optional flags like: --read-intensive [rate] --write-intensive [rate] Thus, when enabled it would do something like this instead: let [f (if (< (rand) (:rate opts)) :w :r)] Or maybe make it one flag, e.g --write-intensity and those who want read intensity would use 1-rate. I'm ready to make a pull request, if you think this is a useful feature. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[georgyshamteev]

Well, after a long thought and some test simulations I can say that you have a good point there about write intensity! Indeed, a lot of writes without reads might be no use. And if we want a high write workload we can use default gen. But! I hardcoded or simulated low write-intensity workload (just instantly failing 95% of :append txns without even executing them). And I managed to find :anomalies {:G-single-item-realtime :cycle anomaly. (I believe it is also just a stale read, but elle called it cycle, so let it be so).
The point with read-intensity workload is that we execute small number of :append txns, so they don't interfere with each other and observe them with high intensity at any given point with reads. Default gen couldn't manage to do that, because there were not enough reads per second or reads were intensive, but it also created a lot of :appends, which interfered with each other and didn't let reads observe the bug.

Thus, I believe a --read-intensity [rate] flag might be useful. What do you think?

[aphyr]

Ah, interesting. Can I ask what kind of throughput you're getting, and how many ops you're doing in a test total? I imagine this problem might be more pronounced for low throughout or short histories.Another thing I'd like to understand--is the difference due to just doing *more* reads, or is it that you're generating more *read-only transactions*? If all ops and txns have roughly equal cost, you should only be able to double the read frequency at most. If it's because (e.g.) read-only transactions are faster, more available, or use a different codepath, I think there are even better ways to get what you're after.

[georgyshamteev]

Can I ask what kind of throughput you're getting

It was a short test. 60 second. About 400 ops. About 4 writes, others are reads. Rate is about 20 ops per sec.

Another thing I'd like to understand--is the difference due to just doing more reads, or is it that you're generating more read-only transactions?

Well, I'm currently testing our system with txns that consist of single op. Thus, one txn is either a read-only or a write-only. So it is due to both: read only txns and their increased number.
Elle supports multiple operations when generating txns, e.g. [op1, op2, op3...]. But I couldn't find API for making Elle generate txns that are read-only or write-only - [:read1, :read2, ...] / [:append1, :append2, ...]. That is why I test with parameter --max-txn-length = 1.

To sum up, it would be nice to have both:

1. write/append -only txns, read-only txns
2. read-intensive workload

[aphyr]

Ah, that makes sense, this is a suuuuuper tiny test. Assuming the bottleneck is the DB, you're probably going to want to run these for much, much longer. Elle is built for millions of txns per history.As far as read-only txns go, I suggest you actually bind them to specific threads or nodes. See https://github.com/jepsen-io/mysql/blob/main/src/jepsen/mysql/append.clj#L179 for an example.

[georgyshamteev]

Elle is built for millions of txns per history

It is actually an interesting take. After running a (debatably) huge test I believe Elle (JVM?) OOMed while analysing and it was just 50k txns.
The issue might have been that it is a list-append test and every :read txn by the end of test consists of approximately 50k values.

See https://github.com/jepsen-io/mysql/blob/main/src/jepsen/mysql/append.clj#L179 for an example.

Thank you very much for advice!

I suggest you actually bind them to specific threads or nodes.

This implies writing my own read-only gen and it is ok. What about generating write-only txns (e.g. [:append 1, :append 2, :append3 ..] )? It is if I want my txns to be more than --max-txn-length = 1, but have only appends. Also to write my own generator?

upd: or maaybe just skip reads while processing txns with len>1, but it is more of a workaround, than a solution

What about supporting that in Elle out of the box? Writing read-only gen might be easy - that is true. Adding possibility of generating write-only txns with all the logic that Elle already has - seems like a possible place for framework improvement? Right now for write-only gen with all existing logic I think I might have to just copy a lot of code from Elle and add a flag for that...

[georgyshamteev]

Assuming the bottleneck is the DB

Also, I would say that it is not the DB bottleneck, it just looking at testing from another angle. I have heavy workload tests - hundreds of txns per second. The test I was referring to was a light workload to accurately inspect strict serializability - with low write intensity and high read intensity.

I think this is an interesting angle to look from at DB testing and it actually worked out. Maybe Elle doesn't need support for read-intensity rate - it can be expressed via read-only gens and usual gen with stagger. But having something for generating write-only txns on Elle side might be useful? What do you think from your experience?

[aphyr]

The issue might have been that it is a list-append test and every :read txn by the end of test consists of approximately 50k values.

What on earth? The default is 32. Of course the software behaves poorly when you ask it to do something three orders of magnitude bigger. I firmly feel that you are creating problems for yourself that do not need to exist, and are asking me to burn time fixing them.

[aphyr]

But having something for generating write-only txns on Elle side might be useful?

Maybe—but I don’t know of a system where this would reveal new bugs, it didn’t seem to help in your case, and it means pushing down complexity that can be expressed straightforwardly via gen/map. Heck, for your single-op transactions, it’s a one-liner: (gen/filter (comp #{:append} first first :value) …).

More to the point, if your system responds differently to read-only vs mixed vs write-only transactions, you probably want to separate those classes of transaction onto different threads, to avoid starvation when one type of transaction becomes unavailable. I really would like to see that change, because it a.) makes it easy to compose all kinds of transaction mixtures, including the ones you’re asking for, and b.) would significantly simplify the example that I linked, which is a common use case in Jepsen.

The challenge is that Elle doesn’t actually provide a Jepsen generator—it returns a single, lazy sequence. You can use that as a generator, but providing separate read-only and general transactions needs jepsen.generator’s contextual wrappers to exchange information between the two streams of transactions. However, Elle doesn’t (presently) depend on Jepsen, so the generator namespace is unavailable! That’s why I’ve so far punted on pushing this sort of thing down into Elle. There is, I think, a reasonable argument for creating a new, reasonably-small, Jepsen utility library, moving generators into it, and letting Elle depend on those generators. Alternatively we could duplicate the relevant interfaces and write a shim between Jepsen and Elle. Both of these have so far entailed more work than just rewriting transactions using gen/map and gen/filter, which is why the current state of affairs persists.