Add Controlize rules

[solardiz]

I think we need a pre-defined ruleset called Controlize, e.g.:

[List.Rules:Controlize]
s[aAbB]\p\r[\x01\x01\x02\x02] Q

And so on for other letters, and perhaps listing more characters per line. I was just lazy here.

We should also consider rules for other ways to "controlize" - e.g., of only some (not all) instances of a character, or of multiple different characters at once. Maybe we should add some rule commands into JtR itself to do this efficiently. This is similar to case toggling.

Maybe we should skip producing '\n' for now, as it would break the john.pot file format until we finally implement hex-encoding there (#2354).

[solardiz]

@AlekseyCherepanov Would you contribute this, please? (Since you had a related ruleset in a recent contest.)

[AlekseyCherepanov]

I think / command would work a bit faster than Q. But it produces unwanted candidates when the letter to be replaced is after position 125: / can find it and accept the candidate, Q would reject.

Search after 125:

$ python -c 'print "a" * 126 + "b"' | ./JohnTheRipper/run/john --pipe --stdout --rules=': sbc Q'
Using default input encoding: UTF-8
0p 0:00:00:00 0,00% 0p/s

$ python -c 'print "a" * 126 + "b"' | ./JohnTheRipper/run/john --pipe --stdout --rules=': /b sbc'
Using default input encoding: UTF-8
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
1p 0:00:00:00 0,00% 14.28p/s [...]

Comparison of speeds:

$ ./JohnTheRipper/run/john --wordlist=rockyou.txt --stdout --verbosity=1 --rules=': s[aAbBcCdDeEfFgG]\p\r[\x01\x01\x02\x02\x03\x03\x04\x04\x05\x05\x06\x06\x07\x07] Q'
Warning: Verbosity decreased to minimum, candidates will not be printed!
Using default input encoding: UTF-8
21703902p 0:00:00:09 100,00% (2019-11-04 17:52) 2366Kp/s [...]

$ ./JohnTheRipper/run/john --wordlist=rockyou.txt --stdout --verbosity=1 --rules=': /[aAbBcCdDeEfFgG] s\1\p\r[\x01\x01\x02\x02\x03\x03\x04\x04\x05\x05\x06\x06\x07\x07]'
Warning: Verbosity decreased to minimum, candidates will not be printed!
Using default input encoding: UTF-8
21703908p 0:00:00:07 100,00% (2019-11-04 17:53) 3010Kp/s [...]

Code to populate rules:

# -*- coding: utf-8 -*-

def populate(ll):
    l = "".join("{:c}{:c}".format(i + ord("a") - 1, i + ord("A") - 1)
                for i in ll)
    r = "".join(r"\x{0:02x}\x{0:02x}".format(i)
                for i in ll)
    return l, r

def out(l, r):
    print
    print r"s[{}]\p\r[{}] Q".format(l, r)
    print
    print r"/[{}] s\1\p\r[{}]".format(l, r)

t = range(1, 27)
l, r = populate(t)
print r'# full range'
out(l, r)
print

t.remove(ord('\n'))
l, r = populate(t)
print r'# without \n'
out(l, r)

Output:

# full range

s[aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ]\p\r[\x01\x01\x02\x02\x03\x03\x04\x04\x05\x05\x06\x06\x07\x07\x08\x08\x09\x09\x0a\x0a\x0b\x0b\x0c\x0c\x0d\x0d\x0e\x0e\x0f\x0f\x10\x10\x11\x11\x12\x12\x13\x13\x14\x14\x15\x15\x16\x16\x17\x17\x18\x18\x19\x19\x1a\x1a] Q

/[aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ] s\1\p\r[\x01\x01\x02\x02\x03\x03\x04\x04\x05\x05\x06\x06\x07\x07\x08\x08\x09\x09\x0a\x0a\x0b\x0b\x0c\x0c\x0d\x0d\x0e\x0e\x0f\x0f\x10\x10\x11\x11\x12\x12\x13\x13\x14\x14\x15\x15\x16\x16\x17\x17\x18\x18\x19\x19\x1a\x1a]

# without \n

s[aAbBcCdDeEfFgGhHiIkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ]\p\r[\x01\x01\x02\x02\x03\x03\x04\x04\x05\x05\x06\x06\x07\x07\x08\x08\x09\x09\x0b\x0b\x0c\x0c\x0d\x0d\x0e\x0e\x0f\x0f\x10\x10\x11\x11\x12\x12\x13\x13\x14\x14\x15\x15\x16\x16\x17\x17\x18\x18\x19\x19\x1a\x1a] Q

/[aAbBcCdDeEfFgGhHiIkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ] s\1\p\r[\x01\x01\x02\x02\x03\x03\x04\x04\x05\x05\x06\x06\x07\x07\x08\x08\x09\x09\x0b\x0b\x0c\x0c\x0d\x0d\x0e\x0e\x0f\x0f\x10\x10\x11\x11\x12\x12\x13\x13\x14\x14\x15\x15\x16\x16\x17\x17\x18\x18\x19\x19\x1a\x1a]

[solardiz]

Thanks Aleksey, you're quick!

125 is when you test in --stdout mode. The actual length that Q checks up to is the current format's maximum supported length - e.g., 8 for descrypt.

Which approach is better depends not only on speeds, but more on whether the resulting candidates would be directly used with the current format or subject to another ruleset or mask stacked on top of these rules. In fact, this is a more generic issue now discussed as #4135.

For this ruleset, I think using Q will provide more consistent behavior with other rules we have.

Will you make a PR using this line, please? -

s[aAbBcCdDeEfFgGhHiIkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ]\p\r[\x01\x01\x02\x02\x03\x03\x04\x04\x05\x05\x06\x06\x07\x07\x08\x08\x09\x09\x0b\x0b\x0c\x0c\x0d\x0d\x0e\x0e\x0f\x0f\x10\x10\x11\x11\x12\x12\x13\x13\x14\x14\x15\x15\x16\x16\x17\x17\x18\x18\x19\x19\x1a\x1a] Q

[AlekseyCherepanov]

JFYI --stdout=8 can emulate truncation for descrypt. So in rules, '* may be used to truncate candidate and reduce length for search. Truncation and search work a bit faster than Q.

$ ./JohnTheRipper/run/john --wordlist=rockyou.txt --stdout=8 --verbosity=1 --rules=': s[aAbBcCdDeEfFgG]\p\r[\x01\x01\x02\x02\x03\x03\x04\x04\x05\x05\x06\x06\x07\x07] Q'
Warning: Verbosity decreased to minimum, candidates will not be printed!
Using default input encoding: UTF-8
17093014p 0:00:00:09 100,00% (2019-11-04 18:44) 1897Kp/s

$ ./JohnTheRipper/run/john --wordlist=rockyou.txt --stdout=8 --verbosity=1 --rules=': \x27* /[aAbBcCdDeEfFgG] s\1\p\r[\x01\x01\x02\x02\x03\x03\x04\x04\x05\x05\x06\x06\x07\x07]'
Warning: Verbosity decreased to minimum, candidates will not be printed!
Using default input encoding: UTF-8
17093014p 0:00:00:07 100,00% (2019-11-04 18:49) 2285Kp/s

I am not sure that I am not missing something.

I'll make a PR with the line you asked.

[solardiz]

Aleksey, I don't mind you using in the PR the faster version you've identified in your previous comment. Thanks!