Introduce new storage type: prefixed map

[gui1117]

Context

Prefixed map is a new storage available in decl_storage.

it differ from map as it stores it keys at twox_256(prefix)++map_hasher(key) instead of map_hasher(prefix ++ key).

Thus it allows to remove all key of the map using sr-io::kill_prefix.

But the final goal is to make it also iterable using #3883

In the end this should replace linked_map as iteration is more efficient (just there is no longer ordering). and also Maps as it allows to have more functionality

To discuss:

• should we directly put this in a child trie instead of behind a prefix in top trie ?

  • I would prefer waiting for Possible child trie key collision when pruning #3648 before having child trie in decl_storage actually.

• the value is stored at twox_256(prefix)++map_hasher(key):

  • the first part is using twox_256 for hashing the prefix because we have to be sure that no other value share the same prefix. in case we only use twox_128 then an attacker could try to craft a key which have blake2_256(key)[0..16] == twox_128(prefix). Thus inserting a wrong value into this map.

    Introducing this map has the consequence to limit the size of our hashers. introducing a hasher with an output longer than 256 bit is no longer relevant as it can already collision with this map.

    If this is an issue then it can be solve in two way:

    • waiting for child trie
    • allow to remove key with a given prefix and a given size only. thus we would call sr-io::kill_prefix_with_size(MAP_PREFIX, 32+size_of_map_hasher)

  • the second part is map_hasher(key). currently a hasher is mandatory though it seems it could be interesting to introduce a no_hasher variant (or hasher=identity). thus value would be stored at twox_256(prefix) ++ key.encode(). The only interest I see from this is that it would avoid some hasher call.

[xlc]

This looks like double_map with with first key of ()?

substrate/srml/system/src/lib.rs

Line 416 in ec7c6cf

EventTopics get(fn event_topics): double_map hasher(blake2_256) (), blake2_256(T::Hash)

[gavofyork]

It's not so great to have a long, insecure hash as the first key as it's not inconceivable that someone could craft a module which intentionally has storage keys that collided with another module. A better way of forming the first key could be to simply append a short twox hash (8 byte, perhaps) of the item name to either the module name or a twox hash of it. This more or less guarantees it being unrealistic to craft a storage item collision, at least making it super-obvious in the module name.

[gui1117]

So finally we will do instead of a new storage type just an optional attribute on linked_map

MyLinkedMap prefixed(): linked_map u32 => u32

stored at twox(ModuleName)++twox(MyLinkedMap)++hasher(key).
This would implemente the same trait StorageLinkedMap (though with change of the generator to have this two prefixes), also in the metadata we need a new field for the additional prefix as well (or a complete new storage variant)

(Or otherwise we could also make use of blake2hasher if we can ensure this is done at compile time but I prefer the first way)

I'll work on this starting monday next week after my days off

[gavofyork]

sounds good.

[bkchr]

As we want to remove LinkedMap in the future anyway. I would propose that you add the new prefixed() attribute to StorageMap.
If it is a prefixed StorageMap, we also implement Iterator or a custom trait to make it iterable like a LinkedMap.

[gavofyork]

how far off is this? i could do with it for a PR i'm working on...

[gavofyork]

in particular, i could do with iteration on a double_map - is that already prototyped?

[gui1117]

for double_map you want to iterate over all values?

I can implement Iterator<Item=(Key1, Key2, Value)> similarly to the incoming prefixed map it seems.

But if you want to iterate on first keys only it seems it requires new function in sr-io.

[gui1117]

superseeded by #4185