TLS Session Context refactoring #73

alexmgr · 2016-10-04T21:58:58Z

I'm looking at adding TLS 1.3 to scapy_ssl_tls. But before that, I think that TLSSessionCtx needs some refactoring.
It's quite difficult to integrate new things now. Here is my grand masterplan ;). @tintinweb, let me know if that works for you:

Add an asymmetrical keystore to hold RSA/DSA keys. tls_ctx will hold a reference to both a client and server keystore. Then get rid of self.crypto.client.rsa.privkey and co to use the keystore. Mostly done
Introduce a key exchange keystore to hold EDH/ECDH/... keys. tls_ctx will hold a reference to both a client and server keystore. Then get rid of crypto.server.dh.p and co to use the keystore.
Introduce a symmetrical keystore to hold generated keys. tls_ctx will hold a reference to both a client and server keystore. Then get rid of crypto.session.key.client.mac and co to use the keystore.
Introduce a TLSContext which will hold references to the above keystores. Per session, 2 TLS contexts will be created, one server side, one client side. Today we duplicate this through client/server distinction, but we can abstract this away
TLSSessionCtx will bind 2 TLSContexts (client/server) together and continue to hold the general session info (master_secret, negotiated stuff). I think this will make Allow loading of session secrets directly #64 easier, and can merge at that point all the session serialization
Finally refactor _process()

This has the advantage that all seperate components are in charge of loading/saving/printing as they which, and makes iterative changes (and UT) easier.

Let me know what you think. I'll try and conserve the external interfaces to minimize changes, but references to tls_ctx variables from outside will break.

- Introducing a keystore to abstract asymetrical keys from TLS context - Encapsulated RSA utility functions into a new module: ssl_tls_keystore - Added a test module to make keystore UT simpler and independant from crypto layer - Refactored ssl_tls_crypto to use the asymetrical keystore layer, as well as various examples and tests

- Removed all references to rsa.pubkey and rsa.privkey. Everything uses the RSA keystore - Moved TLSSessionCtx string templating to the RSAKeystore class. Each sub element can be responsible for it's formating. Looks weird right now, until I have completed the refactoring of other elements

- Use double quotes consistently - Moved camel casing to lower_case_with_underscores casing

- Moved DH Kex parameter handling into the keystore - Refactored the TLSSessionCtx to use the new DH keystore - Refactored UT

tintinweb · 2016-10-05T21:04:14Z

definitely the way to go! that will make it pretty straight forward to implement the idea of #64 - so thats a big like from my side. I can probably review this next week as I am currently travelling an plan to release/push v1.2.3 to pypi by the end of this week (fixing lots of nasties)

- Made keystore naming consistent with kex_keystore - Print the size of RSA keys in use

- UT needed to work with RSA keys for the size calculation to work

- Refactored the TLSSessionCtx to use the new ECDH keystore

- Client side and server side keys are maintained in seperate key stores

- TLScontext holds all crypto and session material for one end of the connection - TLSSessionCtx holds a reference to 2 TLSContexts

alexmgr · 2016-10-06T23:57:03Z

This was tedious, but I think it's in a decent state now.
Still need to finally refactor the _process() layer, but should be simple now.

- Preserved the stateless nature of __process. Just moved state handlers to seperate private functions

alexmgr · 2016-10-10T22:30:09Z

New output from session ctx looks something like this:

TLS Session Context:
    negotiated.version: TLS_1_2
    negotiated.ciphersuite: ECDHE_RSA_WITH_AES_128_CBC_SHA256
    negotiated.key_exchange: ECDHE
    negotiated.encryption: ('AES', 16, 'CBC')
    negotiated.mac: SHA256
    negotiated.compression: NULL
    encrypted_premaster_secret: None
    premaster_secret: '\xc9\x83\xc6\x1d\x07\xc9\xc3\xa4\xcew\xce\x13\xcd\\\xd0\x8b\xd6\xc0\\\xee[\x96\xb1D\xf23T\xf0\xdaq\t\x1f'
    master_secret: 'j\x8b;\x83a\x97\xc9)\xd5PM\x00A\xfa\xae\x85\xeahN\xca\xe9\xd9=P[/\xf1\xefn(\xc1*\xe8\x96\xd8\xaa2\xdbs"\xa4\x89\x93\x02}\x84\x93\xdf'

    Client TLS context:
        random: 'W\xfc\x15`\xa8:\x03\xdb<\xb3:\x93\xc4\xf6\x9d\xdcFGe\xd3\xb0\xb1\xffR5\xe3I\x9a\x82\x90*\xdf'
        session_id: ''
        Empty Asymmetrical Keystore:
            certificate: None
            size: 0
            public: None
            private: None
        ECDH Keystore:
            curve: secp256r1
            size: 256
            public: (113275496704084431291487477677909195595759109595762060638950447132726474002456, 77954217625571679719663409439576132777386731898169341442210820337739525519775) on "secp256r1" => y^2 = x^3 + 115792089210356248762697446949407573530086143415290314195533631308867097853948x + 41058363725152142129326129780047268409114441015993725554835256314039467401291 (mod 115792089210356248762697446949407573530086143415290314195533631308867097853951)
            private: 64156282720036847036300497222265754557565650929814408823437749519510182577437
        ECDHE_RSA_WITH_AES_128_CBC_SHA256 Keystore:
            AES cipher:
                mode: CBC
                key: 'K\xca\xff\xf8\xbd\\"\xd6<`#\xa7&\xdc\x10\xe1'
                size: 128
                block_size: 128
                iv: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
            SHA256 hmac:
                key: '?\xae\x9f\x8d~\xc9h\x9e\xcb\xb8\xfd\x05"C(\xcb\xaa\xd4oS\xa5K2\x1e3\xf1qe\x0bz\x05\xca'
                size: 256

    Server TLS context:
        random: '\xe6\x7f\x84\x16\x81\x03e\x9f\xd8\xe6W\x12\xbb\x19\x97\x07\xb5\xd5Z\xde\xcf\x91\xaf\x87e\xf0\x92/%\x9f\xf8\xd9'
        session_id: '\x1a\xf2\xde\x0c\xe3\xf5\xb7\x84Fq\x063HS\xa3\xdf(iEkf\xf2".3\xf9y\x13]\xfc7\xef'
        RSA Keystore:
            certificate: '0\x82\x04n0\x82\x03V\xa0\x03\x02\x01\x02\x02\t\x00\xd1\xe1\xf5:\x92\x03%\x1a0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x05\x05\x000\x81\x801\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x130\x11\x06\x03U\x04\x08\x13\nSome-State1\x120\x10\x06\x03U\x04\x07\x13\tSome-city1\x150\x13\x06\x03U\x04\n\x13\x0cSome-company1\x100\x0e\x06\x03U\x04\x0b\x13\x07Some-OU1\x1f0\x1d\x06\x03U\x04\x03\x13\x16some-server.some.where0\x1e\x17\r150522191612Z\x17\r250519191612Z0\x81\x801\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x130\x11\x06\x03U\x04\x08\x13\nSome-State1\x120\x10\x06\x03U\x04\x07\x13\tSome-city1\x150\x13\x06\x03U\x04\n\x13\x0cSome-company1\x100\x0e\x06\x03U\x04\x0b\x13\x07Some-OU1\x1f0\x1d\x06\x03U\x04\x03\x13\x16some-server.some.where0\x82\x01"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0f\x000\x82\x01\n\x02\x82\x01\x01\x00\xcd{qe\xeeu(\xf1\x07\xcffn\xdcg>\xed\xc8cTN\xbe\x8c\xc3t\x13F\x01^\xea\x18*s\xa9\xe1\x8e&\xf6\xf1U=\x83\x84=+\xda\xcd\xd4P\x1f\xae\xc7\xb4\xf5Dk\x87\x90\x05?\x15.#\xf7\r\x12\x1c\xa7\xf6:"\xa6WSn\xe4\xb5\x0b\x87wV\x8e\xf4i\x90\\\xe0R\x11\x17\x8d\xd9\xeb\xe2#\xb2\x12F\xcc\xe4\xba\xf3Q\xd0\xb8\x1bFH0\xe1_\xb7\x17\x8c\xf5\xf3\x9evs\xdewy\xe5\xdb\xbdz=.\xa9\x85\x89\xb0\xd6\x0065Dv\x93\xed.\xc62\xc3\xdb\xb62\xac%N;\x8c\xd7\x8e\x1e\xa1`\x98&\'\xe2\xcd:6\x9cK\xb4<HaA\xb9\x7f\xbb\xd9\xd3\xcb\x01K\x92\xe0\xecn\xcfF\xde\xd6GI\xbb\xec\xfbo\x98\xd0\xd2\xf4Y\xd5\xcf\x00T\xa6R"\x80\xaf\x96\x1d\xfc\xbe\x16P\x93q\x80\xf4=\xec\xf2\xf8r[\x94\xee\xec\x10$\x8c\xdcp\xac\xadc\xbc\xc3\xcdSp\xd0\xdc\x0f<\xba\x8d6\x99\t\xc6\xb9\x17\xf2C\xe5\xe5\xbc\'\x02\x03\x01\x00\x01\xa3\x81\xe80\x81\xe50\x1d\x06\x03U\x1d\x0e\x04\x16\x04\x14<\xc2\xf7\xfc\x85\xdb\xbeK\x05f\xb3[\xdetD\x84C\x8a\xe8>0\x81\xb5\x06\x03U\x1d#\x04\x81\xad0\x81\xaa\x80\x14<\xc2\xf7\xfc\x85\xdb\xbeK\x05f\xb3[\xdetD\x84C\x8a\xe8>\xa1\x81\x86\xa4\x81\x830\x81\x801\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x130\x11\x06\x03U\x04\x08\x13\nSome-State1\x120\x10\x06\x03U\x04\x07\x13\tSome-city1\x150\x13\x06\x03U\x04\n\x13\x0cSome-company1\x100\x0e\x06\x03U\x04\x0b\x13\x07Some-OU1\x1f0\x1d\x06\x03U\x04\x03\x13\x16some-server.some.where\x82\t\x00\xd1\xe1\xf5:\x92\x03%\x1a0\x0c\x06\x03U\x1d\x13\x04\x050\x03\x01\x01\xff0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x05\x05\x00\x03\x82\x01\x01\x00\x01s\x8e)\x85i-\x829\xfb\x17\x95\xe6\xea\x07\x18u\\\xf1\x06\xcds\x9fq\x13\xaf\xd3\xa0t\xad\xd0\x7f\x98\x1b\x06\xf3K\x9d\xf3\xe1e\x8c\x153U\xc5\x06\x1b6\x9d`\xd3A\xebL\xce\xfd\xd9\x8dmg\x90\xbeI\x9c\xde\x8b\xd5p]\x1a\x8a\x89\xbb\x14\x15\x99\xf31\x99\x14\xf8S\x9e)HH\xc1\x068b\x18\xd8g\x9d\xa4k\xa9\n,\xe7Xre\xcbU\xd6\xa6)V\x9beX\x1e\xe2\xe8\x8d\xed&K\x81\xdf\xf1\xc1\x1e,Ur\x8e\xfe\x17\r\xfeOvpo\xbb\xda\x13{\x02\xe0\xfa\x98sU\xb0\xcf\xdb?\x867\xe3Ts\xe4\xa6\xec\xcd\xcb\xc2}U\xd1\xf9V\xa5\xf2\xc4T\xe97\xdfq\xd4.!\xd4]"tw\xe2`S\xb8\xbe\x00?\xa5\'tk\x16;=K\x9aX](`\xe5\x08\x0e\xd9s}L_\xa5\xa3.\xeeE\xa4\xe5m\x8a\x03T#Ia\x90\x84X\x0c\xc9\xc6\xc2[\x1a\xc7\xf3\x85KP\x14#\xea\xfd\xd3(\x96\xaf\x92\xce\x8c\xa6\x929G\xd7|'
            size: 2048
            public: <_RSAobj @0x104cc7ab8 n(2048),e>
            private: None
        ECDH Keystore:
            curve: secp256r1
            size: 256
            public: (57839194958121498880865767025301393551536204925944902627237688213575334463653, 44403111253405849644576056020789267189925076126762403782064647303951800520579) on "secp256r1" => y^2 = x^3 + 115792089210356248762697446949407573530086143415290314195533631308867097853948x + 41058363725152142129326129780047268409114441015993725554835256314039467401291 (mod 115792089210356248762697446949407573530086143415290314195533631308867097853951)
            private: None
        ECDHE_RSA_WITH_AES_128_CBC_SHA256 Keystore:
            AES cipher:
                mode: CBC
                key: "\x82\xc8\xec\xa6\t\xccl':E\xee_\xec\xa3(\xf5"
                size: 128
                block_size: 128
                iv: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
            SHA256 hmac:
                key: '\xed\xfcRn\x02L\x9co\x8f\x10\xb9X\xe5\x0fK\x8f\x94W\x1e\x81\xaa\xe3\xe2\xfcWW\xa2?\xde\\^\xa6'
                size: 256

Formatting should be super easy to tweak. Should also be much easier to refactor further.
It also allows to serialize/restore any crypto state (interface TBD).

I'll put this through a PEP8 formatter and call it a day. After painfully rebasing #72 on top of this ;)

- Replaced remaining single quotes with double quotes - Normalize the resulting files with: autopep8 -a -a -i - Added pep8 section to setup.cfg to allow usage of long lines

tintinweb · 2016-10-11T07:23:35Z

thats looking pretty!
I started to rebase this off master after merging #72 and had to postpone it due to some brainteaser merge conflicts but I could probably help with that today.

- Resolved conflicts from session resumption into the refactoring work

alexmgr · 2016-10-11T17:43:16Z

Just saw your comment. Has this almost done yesterday, so pushing this after merge.
Should solve all conflicts.

I'll put the whole project through autopep8 and we can merge this.

@tintinweb, would you mind giving this a quick test? I think all is good, but just to make sure. I ran most of the examples, including the automata stuff, so this should probably ll be OK.

An integration script would be nice at some stage to, to test all the examples @ commit time. Will see if I can do this later.

- Ran "autopep8 -a -a -i -r ." on project. Formatting should be more standardized now

tintinweb · 2016-10-11T21:30:20Z

sure, will check if my scripts have any issues and report back.

I can also take care of the integration suite if you want. it's on my todo list for some time and I keep getting reminded on this when manually checking our scripts for upcoming releases.

alexmgr · 2016-10-11T21:36:03Z

Sounds good. Let me know how this goes.

For the integration scripts, sounds good also. Feel free to send to master directly if you wish.

TLSExtSignatureAndHashAlgorithm; hash_algorithm -> hash_alg; signature_algorithm -> sig_alg fixes security_scanner SLOTH checks due to attrib name change in TLSExtSignatureAndHashAlgorithm adds TLS_1_4

tintinweb · 2016-10-12T09:06:59Z

fixed an attrib name inconsistency (doc/layer/examples)
added TLS_1_4 magic

nosetests shows an "Unknown client key exchange" warning, will have to investigate.

alexmgr · 2016-10-12T16:44:03Z

nosetests shows an "Unknown client key exchange" warning, will have to investigate

That's "normal". It was a happening in the background before. I just added an else clause to raise a warning when that happens. Didn't bother digging into the root cause though since it's not a result of the refactor.

- RSA key loading should happen in the TLS context, not the session context. Avoids the `if client` logic - Added a warning for unknown server kex

alexmgr · 2016-10-12T17:18:12Z

This concludes it for me ;). Feel free to merge if you're happy with the testing and changes.

- resumption property was printed as compression

tintinweb · 2016-10-12T18:03:17Z

this is supernice! finished testing. I also see the warning for the full_Rsa_connection_with_application_data example but I agree it does not look like a regression.
good work!

Alex Moneger added 4 commits October 4, 2016 14:38

rsa_public_from_der_certificate PEP8 fixes

1a4bda1

- Use double quotes consistently - Moved camel casing to lower_case_with_underscores casing

Introduced the Kex keystore in TLSSessionCtx

954d1ff

- Moved DH Kex parameter handling into the keystore - Refactored the TLSSessionCtx to use the new DH keystore - Refactored UT

Alex Moneger added 5 commits October 5, 2016 14:54

Renamed keystore to asym_keystore

5a99613

- Made keystore naming consistent with kex_keystore - Print the size of RSA keys in use

Fixed error in RSAKeyStore testing

b7b467d

- UT needed to work with RSA keys for the size calculation to work

Introduced the ECDH Keystore

6e77c62

- Refactored the TLSSessionCtx to use the new ECDH keystore

Introduced a Symmetric keystore

d2a86bb

- Client side and server side keys are maintained in seperate key stores

Added TLSContext to hold endpoint context

732c581

- TLScontext holds all crypto and session material for one end of the connection - TLSSessionCtx holds a reference to 2 TLSContexts

tintinweb added the enhancement label Oct 8, 2016

Refactored _process()

21fef13

- Preserved the stateless nature of __process. Just moved state handlers to seperate private functions

Applied autopep8 on the resulting files

cf28cca

- Replaced remaining single quotes with double quotes - Normalize the resulting files with: autopep8 -a -a -i - Added pep8 section to setup.cfg to allow usage of long lines

Merge branch 'master' into tls_ctx_refactoring

b254e45

- Resolved conflicts from session resumption into the refactoring work

PEP8 formatting changes

a357ed0

- Ran "autopep8 -a -a -i -r ." on project. Formatting should be more standardized now

alexmgr mentioned this pull request Oct 11, 2016

How to do SSL renegotiation using the scapy-ssl_tls tool? #75

Open

alexmgr self-assigned this Oct 11, 2016

fixes layer/doc due to attrib name change with d0a32ca

c3c947a

TLSExtSignatureAndHashAlgorithm; hash_algorithm -> hash_alg; signature_algorithm -> sig_alg fixes security_scanner SLOTH checks due to attrib name change in TLSExtSignatureAndHashAlgorithm adds TLS_1_4

Moved RSA key loading into TLSContext()

57e8786

- RSA key loading should happen in the TLS context, not the session context. Avoids the `if client` logic - Added a warning for unknown server kex

Fixed typo in TLS string template

0ce5e56

- resumption property was printed as compression

tintinweb merged commit ebc8219 into master Oct 12, 2016

alexmgr deleted the tls_ctx_refactoring branch October 12, 2016 18:11

trolldbois mentioned this pull request Jun 16, 2017

refresh with new scapy layers trolldbois/sslsnoop#2

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

TLS Session Context refactoring #73

TLS Session Context refactoring #73

Uh oh!

alexmgr commented Oct 4, 2016

Uh oh!

tintinweb commented Oct 5, 2016

Uh oh!

alexmgr commented Oct 6, 2016

Uh oh!

alexmgr commented Oct 10, 2016

Uh oh!

tintinweb commented Oct 11, 2016

Uh oh!

alexmgr commented Oct 11, 2016

Uh oh!

tintinweb commented Oct 11, 2016

Uh oh!

alexmgr commented Oct 11, 2016

Uh oh!

tintinweb commented Oct 12, 2016

Uh oh!

alexmgr commented Oct 12, 2016 •

edited

Loading

Uh oh!

alexmgr commented Oct 12, 2016

Uh oh!

tintinweb commented Oct 12, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TLS Session Context refactoring #73

TLS Session Context refactoring #73

Uh oh!

Conversation

alexmgr commented Oct 4, 2016

Uh oh!

tintinweb commented Oct 5, 2016

Uh oh!

alexmgr commented Oct 6, 2016

Uh oh!

alexmgr commented Oct 10, 2016

Uh oh!

tintinweb commented Oct 11, 2016

Uh oh!

alexmgr commented Oct 11, 2016

Uh oh!

tintinweb commented Oct 11, 2016

Uh oh!

alexmgr commented Oct 11, 2016

Uh oh!

tintinweb commented Oct 12, 2016

Uh oh!

alexmgr commented Oct 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexmgr commented Oct 12, 2016

Uh oh!

tintinweb commented Oct 12, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexmgr commented Oct 12, 2016 •

edited

Loading