Field size limits in the protocol

🌌 Research Coordicide
1

For the Shimmer network, a set of protocol parameters was chosen.
And I think most values like Min PoW Score, Below Max Depth or VByte Cost where chosen with good reason and/or simulations done beforehand.

However, the maximum length for metadata fields (and subsequently also the maximum message length) got me thinking. Currently metadata fields can hold at most 8192 bytes. Even with JSON objects, this can be reached rather fast, especially if we use to to actually represent a mutable state, e.g. controlled through a committee via an alias account. If we take a big step into the future and look at L1SCs, these fields are definitely not big enough.

The benefits of larger fields are pretty clear. If you want to store bigger amounts of data, you do not need to implement some sort of paging to load everything. This also reduces the overall disk storage (and such dust deposit) required, since we only need to store data like type, token amount and unlock conditions once. In addition, this should also lower node resources for fetching, since one large read is faster than multiple, spread out, reads.

There are a few minor cons. First of all, we’d need to change the data type to (uint32)ByteArray to allow >65.535 bytes in a metadata field, which would increase storage for all metadata by two bytes. Also I see a potential network (D)DoS on single nodes by requesting/submitting these large messages, since mana would only pay for the traffic inside the network itself, not when entering/pulling data from it via a node.

Do you see any additional risks on the protocol side?

3

I agree that variable size metadata should be as large as possible, provided it doesn’t overly burden transaction processing. Setting an higher upper limit might be beneficial. It’s essential to benchmark this under both regular transaction scenarios and spam attacks using current standard hardware.

4

According to Tip 19 the total database size is limited at an absolute maximum size. Not sure how this should work with storing large messages that are meant to stay in the ledger.

1 Like
5

Since the storage deposit does still apply, we do not change the maximum data size.

Of course this would increase the amount of space needed to store n days of history. An attacker would still need a lot of mana to attack this through, since bandwidth needs to be paid.