apply AFL to regex

[BurntSushi]

Currently, the parser is "fuzzed" to some extent with quickcheck, which ensures that a randomly generated AST can be roundtripped to concrete syntax and back. However, we should also try to apply more generic fuzzing techniques not only to the parser, but to regex searching itself. AFL seems like an ideal approach.

[SeanRBurton]

I'm willing to do this if you want.

[BurntSushi]

@SeanRBurton Yes, that'd be lovely, thank you!

[lukaslueg]

I'm currently hitting regex with AFL, calling RegEx::new on a AFL-generated pattern and ::is_match(), ::find() and ::captures() on a static test-string. So far I've only seen some "crashes" because regex exceeds the memory limit set by AFL - this could probably be circumvented by using ::with_size_limit().
I wonder what a more thorough strategy would look like.

[BurntSushi]

@lukaslueg Could you please open issues with test cases that reproduce your problem? What is the memory limit?

[lukaslueg]

The memory limit is AFL's default of 50mb VM-size. Re-running with a manually-set ulimit -v reproduced the out-of-memory-hard-abort. I'll post an example when I get one again - I did not pay too much attention to it since there is always ::with_size_limit().

Does it make sense to test the generated patterns against a randomly generated instead of a static string? Basically the only thing AFL is looking for right now are arithmetic overflows and panics...

[BurntSushi]

I'm definitely interested in regexes that use more than 50MB of heap. If you could create a new issue with examples that'd be great!

Does it make sense to test the generated patterns against a randomly generated instead of a static string? Basically the only thing AFL is looking for right now are arithmetic overflows and panics...

Yes, definitely.

[lukaslueg]

Is there a minimum length for the haystack? We want the regex to switch strategies (if it does that at all)

[BurntSushi]

It can switch strategies depending on the length, but the specific length isn't fixed. If the combined size of the input and the regex exceeds ~256KB, then the capturing engine will switch from backtracking to the Pike VM. So if your haystack is over 256KB, that should do it. Note though that this only happens when extracting captures (the captures method but not is_match or find).

[lukaslueg]

I'll continue running AFL-generated regexes against a small haystack since matching a string of 260kb is painfully slow. Two crashing bugs and one OOM-situation so far - I'll switch strategies when things quite down on the small haystack

[BurntSushi]

@lukaslueg Thank you! :-)

[lukaslueg]

Can you give some info on when regex actually executes the simd path?

[BurntSushi]

@lukaslueg Any time there are multiple literals in the prefix. For example, aaa|bbb|ccc|ddd|eee|fff should execute the SIMD algorithm.

Note that you need to compile like so:

RUSTFLAGS="-C target-feature=+ssse3" cargo build --release --features simd-accel

And certainly, you'll need a nightly compiler.

[lukaslueg]

That's what I'm doing. I've also patched MAX_SIZE_BYTES in backtrack.rs to be 256 instead of 256kb in order to give PikeVM a rub without slowing things down too much. Any more suggestions?