Add append/prepend common (pass)word rules

[solardiz]

Maybe we should add a pre-defined ruleset produced with a script from our default password.lst (and updated along with it), appending or prepending each one of those words, also with or without a space, e.g.:

[List.Rules:Password]
\r[:$:^][ ] A\p\r[zz00]"password"

and so on for the full password.lst. Maybe we should also add length checks to ensure the added string isn't fully beyond the format's maximum length (when appending) and doesn't fully push out the original word beyond the format's maximum length (when prepending).

And we should include that ruleset from [List.Rules:All], which currently produces over 6 million candidates from one input word, while [List.Rules:Jumbo] produces about 5000. The addition of another ~3500 is definitely acceptable for "All", but perhaps not for "Jumbo".

[solardiz]

@AlekseyCherepanov Would you contribute this, please? (Since you had a related ruleset in a recent contest.)

[AlekseyCherepanov]

There are some commons passwords appended, prepended and inserted in hashcat's rules included into All. Also some of words are combined with [ulc] rules.

I checked pure prepend (left) and append (right):

$ perl -C0 -ne 'print if /^([\^\$]).(\1.)+$/' JohnTheRipper/run/rules/* \
| grep '^\^' \
| perl -pe 's/\^(.)/$1/g' > h_left.txt

$ perl -C0 -ne 'print if /^([\^\$]).(\1.)+$/' JohnTheRipper/run/rules/* \
| grep '^\$' \
| perl -pe 's/\$(.)/$1/g' > h_right.txt

$ wc -l h_left.txt h_right.txt
 1384 h_left.txt
 2938 h_right.txt

$ grep -cxFf h_right.txt JohnTheRipper/run/password.lst
75

$ grep -cxFf h_left.txt JohnTheRipper/run/password.lst
10

I checked run/rules/T0XlC-insert_top_100_passwords_1_G.rule:

# TOP 100 words shifted from position 0 to position 18 

$ grep '^i1' JohnTheRipper/run/rules/T0XlC-insert_top_100_passwords_1_G.rule \
| perl -C0 -pe 's/i.(.)/$1/g' > h_top100.txt

$ wc -l h_top100.txt
100 h_top100.txt

$ grep -cxFf h_top100.txt JohnTheRipper/run/password.lst
93

$ grep -vxFf JohnTheRipper/run/password.lst h_top100.txt
asd123
asdasd
qwe123
qweasd
12341234
qwer1234
qwerty123

[solardiz]

Aleksey, in other words, you found that most lines in password.lst do not overlap with what's already added with those hashcat rules.

I am more concerned that sometimes the ruleset will be run with password.lst as the wordlist, in which case we'll have 50% of dupes from doing both appends and prepends. Such runs can be interrupted at 50% as they wouldn't crack anything more in the other 50% of run time. That's unfortunate, but separating this ruleset in two would require double the number of lines.

Also, I am wondering if we could efficiently determine common prefixes and suffixes e.g. from RockYou - see what RockYou commonly prepends/appends e.g. to what we have in password.lst - then create the prepend/append rules with those substrings. This could be more efficient than what I had suggested with using password.lst for the appends/prepends as-is, but I think both approaches make sense for different things.