Data Flow in IOTA 2.0 explained

We recently made some important modifications to the data flow in IOTA 2.0 to make it simpler and to integrate the IOTA Congestion Control Algorithm with the current consensus. In this post, I will provide a high-level description of this improved data flow. Please note that the message creator and the rate setter will not be described in this post but are still part of the protocol :slight_smile:.

At each node, a new incoming message will undergo the following steps:

  • Parser checks: A sequence of checks needed to discard invalid messages; this includes removal of duplicated messages, syntactical validation and signature check (not fully comprehensive list).
  • Solidifier: When the new incoming message is not solid (i.e., its past cone is not entirely known), the solidifier module will send a solidification requests to the neighbors of the node trying to pull the missing message(s) from them; a message whose difference in timestamp with its parents is larger than a given threshold will be discarded (maxParentAge condition); further optimization to this module are possible.
  • Booker: Solid messages are added to the local version of the Tangle; the details of the parallel-reality ledger and how messages are discarded if they conflict with others are out of the scope of this post; in the current implementation, solidifier and booker are built as a single module.
  • Scheduling buffer: Solid, valid messages are enqueued onto the scheduling buffer (aka outbox buffer): the buffer is split in several queues depending on the node issuer; each message in this buffer is associated with a ready flag, which is set to TRUE if its timestamp is not in the future AND if the parents have already been scheduled or confirmed; non-ready messages that get confirmed are removed from such a buffer; moreover, if the buffer gets full, the first message from the largest aMana-scaled queue is dropped recursively, until the new message is appended to the buffer (please note that this drop policy is not per queue as described here).
  • Scheduler: A deficit round robin scheduler is used to choose the next message to be gossiped and added to the tip set; in order to be scheduled, messages must satisfies three conditions: (i) the deficit of the corresponding queue larger than the message weight (e.g., size); (ii) the message is ready; (iii) the message is not yet confirmed.
  • Tip selection algorithm: scheduled messages are added to the tip set and replaces their parents, if any; also, parents of confirmed not-yet-scheduled messages must be removed from the tip set; the tip manager selects tips using URTS (uniform random tip selection): however, if the most recent confirmed message in the past cone of the message (TSC = Time Since Confirmation) is far in the past, the message is discarded from the selection but temporarily kept in the tip set; on the other hand, messages older than a certain timestamp should be removed.
3 Likes

Thanks for the summary Luigi!

Could you though expand on this specific point please?

Why would you keep it around?

TSC condition may not valid at a given point, but may at a later time. When messages in the past cone of a tip get confirmed, the value of TSC gets lower. Messages are removed from the tip set if confirmed or old. The threshold for defining an old message is related to the thresholds for the TSC and for the MaxParentAge, currently set to 2/5 minutes.

Just to clarify a few things, which came up during the implementation in GoShimmer:

  • Scheduling buffer: (...) each message in this buffer is associated with a *ready* flag, which is set to TRUE if its timestamp is not in the future AND if the parents have already been scheduled or confirmed (...)

    • Should be: each message in this buffer is associated with a *ready* flag, which is set to TRUE if the parents have already been scheduled or confirmed, because timestamp is checked in the scheduler, which will not schedule messages with a timestamp in the future.
  • Scheduler: (...) (iii) the message is not yet confirmed.

    • If a message is already confirmed, then the scheduler simply skips it and takes the next message.
    • (iv) message timestamp is not in the future
  • Tip selection algorithm

    • Messages are not added to the tip-set if any of their approvers is confirmed or scheduled. This guarantees that messages are not added to the tip-set after their approvers.