Proposal: logical layout of the variable-sized transactions

🌌 Research Coordicide
1

It applies to the next version of Iota protocol, currently being developed on the experimental Goshimmer platform. This post follows ideas of Atomic transfers / transactions instead of bundles.

In short, current bundles of transactions to be replaced by variable size transactions, which are communicated as atomic pieces of data.
One of the main reasons is to avoid rather complicated way to reconstruct the bundle from independently gossiped transactions.
New transactions will basically take over functions of the bundle.

The below also takes into account requirements to the base protocol by the level 2 smart contracts (Qubic assemblies). See also “Signature scheme considerations” below.

EDIT: timestamp, sharding-related, mana-related and maybe some other fields were missed, must be placed somewhere into the layout
EDIT 2: also nonce and rate control related (sequence numer) fields must be added

Binary is assumed, but in general it can be ternary too.

Transaction
Transaction consist of header and any number of blocks:
[Transaction] ::= [header] [block 1]…[block N]
So, the shortest transaction consists of the header only.

Header
It contains Trunk, Branch, App ID and Tag fields.

[Header] ::= [trunk] [branch] [app id] [tag]

Trunk and Branch has usual interpretation, each of size of the hash (32 bytes).
App ID and Tag fields can contain any data. It is interpreted by applications, most likely as unique id of the application and type of the transaction respectively.
AppID and Tag fields are indexed and searchable by the node.
AppID has 32 bytes size. Tag has 16 or 32 byte size.

For example, for qubic assemblies App ID will be used as globally unique ID of the assembly (smart contract address), and Tag field will be used to recognize types of assembly transactions: epoch transaction (configurations), request transaction and state update transaction.

The reason to have 2 arbitrary tag fields is to provide fast search in the Tangle, because the rest of the transaction is not fixed or even not guaranteed.

Blocks of the transaction
Each block can have different size and can be of different types. There must be metadata to recognize type of the block. Block is identified by it’s index within the transaction.

There are several types of blocks. Each type defines own validation rules of the block.

Invalid block makes the whole transaction invalid. I.e. transaction is valid only if all blocks are valid.

The block of any type can be validated independently from other blocks in the transaction.
Therefore, validity of the transactions does not depend on the particular sequence of blocks within it.

There are 4 types of blocks:
(Note: it may be reduced to 3 types, see below)

Fund transfer block
It contains valid UTXO transfer data:

  • input address (EDIT. not needed. All inputs must point to outputs to the same address)
  • one or more inputs (indexed)
  • signature of inputs
  • one or more outputs (indexed)

Address is input account address.

Input is a reference to the (unspent) output in another block in another transaction.
[Input] ::= [transaction hash] [block index] [output index]
where:

transaction hash is hash of another transaction
block index is index of the transfer block in that transactions. The field can be 1 or 2 byte long
output index is index of the output in that block. The field can be 1 or 2 byte long

[output] ::= [address] [value]
where:

address is the target address
value is the number of iotas

The validation rules are usual for UTXO transfers: the transfer must preserve ledger balance and signature must be valid

Data block
The block contains any data of arbitrary length (maximum length must be capped by the protocol).
The data block content is not validated by the protocol.

Signed data digest
It contains arbitrary data of the size of hash (the digest) and signature.
[signed data digest] ::= [digest] [signature]

The block is validated by the node by checking validity of the signature.
It is up to the application what data to put into the digest section, but commonly it will contain hash of one or several data blocks of the same transaction.

Presence of such a block will enable signing and validation of the arbitrary combinations of the data of the transaction by the base protocol, i.e. base protocol won’t allow transactions with invalid signatures of data blocks to be confirmed.

The use-case specific semantics of the block is up to the application.

Chain link block
This is special type of block can be used to create “non-forkable” chains of transactions by applications on the layer 2.
Presence of this type of block ( or perhaps even more than one), makes the whole transaction the subject to the chain constraint.

Chain link block can be implemented the following two ways:

  1. “native” chain link. It will contain:

  • chain reference: a hash of another transaction and the block index
    [chain reference] ::= [tx hash][block index]
    The [tx hash] may be NULL hash (all 0s).

  • public key for the next chain block.

  • signature of the [chain reference]

Validation rules of the block:

  • if [tx hash] is not NULL, it must point to another transactions and the [block index] must point to the chain block within it.
  • signature must be verifiable with the public key, contained in the referenced chain link block
  • it is not allowed more than one valid chain link block with the same referenced chain block. I.e. two such chain blocks are considered conflicting.
  1. Chain link can also be implemented as UTXO transfer of 1 iota to the same address (a colored chain).
    This sequence of such transfers will effectively create a chain, because sequence of signed 1 iota transfer transactions cannot be forked.
    The block would be just an ordinary fund transfer block with usual validation rules. Functionally it will be equivalent to the “native” chain link block.
    It this case we don’t even need a special type of chain link block with “native” implementation of chain links.

Signature considerations
We assume the next version of Iota will use the following signature schemes and, respectively, addresses:

  • WOTS, for backward compatibility and quantum resistant accounts
  • ECDSA, currently being implemented as a part of Chrysalis upgrade.
  • BLS multi-signatures, which also allows Shamir secret sharing scheme and threshold signatures.

BLS signatures and addresses are needed for layer 2 smart contracts for the following purposes:

  • to enable control over iota accounts by the quorum of (shared) secret holders
  • to enable building the chain of consistent state updates on the tangle by the quorum of secret holders .
1 Like
2

We need also a (signed) timestamp and a shard marker (= geographical coordinates).

Also, for the Coordicide we need the issuing node ID + the ID of the node where the mana should go; these, however, need not be in the transaction itself (but they need to be gossiped together).

3

Yes, I missed this. Must be in the header, I think. Mana-related data can go to the transfer block

5

Apart from a signed timestamp, I would also suggest another field, a sequence number to keep track of transactions sent by a given node. This field will specifically be relevant for rate control.

2 Likes
6

thanks. Also nonce was forgotten…

7

this assumes the issuing node ID is part of the tx, right?

(In any case, we need to discuss if we include the issuing node stuff into the tx or not.)

8

Yes, I assume the sequence number is somehow linked to the node ID

9

I don’t think the App ID and Tag field should be in the header of the transaction, that’s like saying that the header of a TCP packet contains application specific information. (App ID is not analogous to a port number in TCP, because a node receives every message)

App ID and Tag fields can contain any data. It is interpreted by applications, most likely as unique id of the application and type of the transaction respectively.

This sentence basically directs that this fields should not be in the header.

For example, for qubic assemblies App ID will be used as globally unique ID of the assembly (smart contract address), and Tag field will be used to recognize types of assembly transactions: epoch transaction (configurations), request transaction and state update transaction .

So this is Qubic specific and therefore should all be handled via a module implementing Qubic “smart contracts”.

The reason to have 2 arbitrary tag fields is to provide fast search in the Tangle, because the rest of the transaction is not fixed or even not guaranteed.

It should be the module’s job to implement indexing of such fields and not the node’s, because this fields are arbitrary.

Transaction
Transaction consist of header and any number of blocks :
[Transaction] ::= [header] [block 1]…[block N]

Each block must define its size, therefore each block must be prefixed by a block header. I’d suggest adding a type field into the block header. I’d say 2 bytes is enough for the type field.

Here we must define what the maximum size of a transaction is, in order to know how much space the blocks are allowed to take.

Invalid block makes the whole transaction invalid. I.e. transaction is valid only if all blocks are valid.

Valid by which definition?

There are 4 types of blocks:

The types of blocks should not be fixed. The amount of types should be limited by the type field denoted in the block header.

In fact, nodes should be implemented in such way, that in a common sense we agree that blocks of type 0 for example contain transfer data. However, subsequent types of blocks should only be understood, respectively interpreted by specific modules listening for blocks with a given type.

This makes the entire protocol more adaptable to changes.

Regarding the things I’ve mentioned above: A Qubic module would listen for blocks within transactions with the specific type and validate them accordingly, while other nodes not caring about such data, will simply dismiss the block as not relevant for the node’s operation (not handle the data inside of it but still persist it, as it is needed to compute the transaction hash).

What block type corresponds to what module is an RFC related question.

In general, I would omit any application specific block types from this thread, until the actual skeleton of a transaction is made up.

10

@luca.moser Thank you for comments. I agree with almost all, except maybe the first one.
The two tag fields intended to be used by the node to filter each transaction and send it to the plugin which then will process transaction. Any node should be able to subscribe (globally) to transactions of applications (smart contracts) it is processing. The AppID is global address of the application instance, for example particular smart contract.

In SC case, qubic assembly or SC is a core plugin. It operates locally and it is only interested in specific transactions.
Not sticking to my preliminary ideas, I think it is essential that core protocol will know some important elements of its core extensions, like smart contracts. Just “listen to transaction with some specific blocks” doesn’t seem to me the right way.
Besides, address of the smart contracts is a good candidate to contain shard id as part of it, for example. I’d say [app id ] = [shard marker] + [app id within shard] have to be investigated. The very idea of the SC we are creating is to be local, so AppId is SC address and it may be routing-related and maybe even gossip can be somehow optimized for that.

So it is better for important properties to be in the fixed part of the transaction and hardwired into the node software.
SC contracts are not just an application, but must be very close to the core protocol. Btw, original idea of Iota is to be first of all a base protocol for L2 protocols, like qubic, not just a token ledger.
I’d say in the future there are good chances transfer-only transactions will be rather an exception and mixed (transfer+data) SC transaction will dominate.

In general, of course it all can be designed in a very polymorphic way, like you are proposing and similar to the meta-tx layout, with unlimited number of block types, layouts, validation types and so on. I am aware of that.

But here, my intention was to stick to known classes of use cases, which has to be as close to the core protocol as possible. The 4 (actually 3) types of blocks must be part of the base protocol, I think. There may be more of them later, yes, even if atm I cannot imagine what else…
From the other side, the very universal and polymorphic “lego” approach is nice but is a coin with two sides.

Anyway, I would agree with anything universal which provides at least equivalent functions.

You are right about many missing details. like field lengths, block types and other metadata. I skipped it intentionally as unimportant details for the “logical” layout. The final tx layout spec draft must be provided by the goshimmer team with all necessary details.

Here I provide rather abstract requirements we discussed with Wolfgang a week ago.
This proposal was created because soon I will need real transactions with real block types from the Goshimmer for the qnode plugin I am creating right now and they are not ready, so I will have to create abstract interfaces to mock it. So I initiated this.

Valid by which definition?

in a sense of ledger validation, which includes all validation filters before confirmation of the transaction.

11

In the ledger_state branch we already have these kind of atomic transactions which can contain different payloads. At the moment there is a “ValueTransfer” and a “Data” Payload, but we can easily add other payloads like “Qubic” as well.

The layout is not necessarily complete yet but it currently looks like this:

Transaction%20Layout%20(1)
Transaction%20Layout%20(1)681Ă—621 12.3 KB

The transaction id is hash(trunk + branch + payload id) so it supports merkle proofs right away.

Every payload then has its own structure again starting with a “Payload Type” that allows the plugins to “identify” the payloads they need to handle followed by all additional fields. Value transfers for example have the address, the inputs, the output addresses and so on …

I don’t think that an “AppId” makes much sense in the generic part of the header because if you want to filter certain things depending on the payload then that should be defined within the corresponding payload structure. If you want to “identify” relevant transactions then it makes much more sense to first filter by payload type as you might otherwise find a lot of unrelated transactions that just happen to have the same AppId but are not even qubic transactions for example.

Instead of having “a single tangle”, we have “multiple tangles” with different rules and even different entities as separate ontologies. The TransactionTangle just contains all transactions and takes care that they are solid regarding branch and trunk. As soon as a transaction is solid, it can for example be analyzed by the ValueTangle plugin that checks if it contains a ValueTransfer. If it does contain a value transfer payload, then the ValueTangle maintains a separate tangle based on the movements of funds, that is more or less independent from the “structure” in the TransactionTangle and that can have its own solidification rules and validation logic.

Using this “ontology” concept, we are able to define a very clean and data flow based approach for all the different transaction types. The TransactionTangle does not need to know anything about the ValueTangle and so on (see the API for the TransactionTangle and the ValueTangle below):

Transaction%20Parser%20(6)
Transaction%20Parser%20(6)1022Ă—1422 26.7 KB

The events of the ValueTangle deal with completely different “entities” than the underlying TransactionTangle. Instead of Transactions getting solid, here “Transfers” are solidified and emitted in the corresponding events (it waits for the transfer inputs to get solid).

A Qubic tangle could correspondingly emit Events like “StateTransitionSolid” and so on which are based on building “chains” of states “hidden in the TransactionTangle”.

This whole ontology concept with having different layers on top of the tangle is extremely powerful and we can even have sth like blockchains inside the tangle. Tbh. I am not really sure why IOTA was not defined like that from the start. You could have had a “TransactionTangle” that solidifes transactions which then gets passed on to a “BundleTangle” that waits for the bundles to be complete and then emits BundleEvents" and so on. Currently in IRI and any other node the Tangle class takes care of everything and is thousands of line of code because you need to deal with all kinds of “edge cases”.

The new Tangles using the described principles are extremely small and consist of less than 300 lines of code because the ValueTangle doesn’t even have to worry about waiting for transactions to become solid anymore.

1 Like
12

@hans.moog the above can accommodate the “branch like switch” (as per https://medium.com/@hans_94488/a-new-consensus-the-tangle-multiverse-part-4-59f70fbc19e6), correct?

13

The branch like switch would be an additional field in the “Tangle Section”.

14

The congestion control may need another field next to timestamp: the gap

15

Silly question, but should the “payload id” be renamed so its not confused with “payload section”? Perhaps Transaction Contents?

16

Last questions: the transaction hash would be the hash of everything, including the trunk and branch, right? Also, do we want to include a nonce?

17

Isn’t that something you calculate locally based on the timestamps and numbers?

I don’t mind the name as long as we have the ability to calculate the id using the 3 hashes

A transactionHash does not exist anymore. It gets replaced by the transaction id.

18

No its not. There is a gap score which is computed, but the gap is declared in the transaction so its objective.

I see. does it contain the trunk and branch id?

19

Yes its described earlier: hash(trunk + branch + payload id)

20

Oh sorry! Thanks for answering!!!

21

that is very much in line with what I imagine. Better wording I must admit :slight_smile:
List of blocks/payloads with different types is what I’m modeling right now. The smart contract program is actually building a resulting transaction block by block and it can build any.
Some types of payloads/blocks must be built-in and must be known by the Value Tangle to forward it the the qnode: the payload types which contain target address/id of the assembly.
(Forget the AppId, I immediately abandoned it after realizing that one output transaction (assembly state update) can be an input request for several assemblies. In this case it would contain several blocks (“payloads”) with several addresses/ids of target assemblies. I wanted AppId would be something IP address, routing transaction to the assembly but that’s too abstract and not clear what it means in the protocol).

Assemblies don’t need anything more than UTXO transfer blocks (value tx blocks), signed data blocks, and threshold signatures/addresses. It is enough to build chains and maintain consistent state of the assembly. Once transaction is solidified/approved (by the Value Tangle), it becomes a valid transaction potentially interesting for some assembly (particular set of nodes with qnode plugins and running particular assembly).
Qnode plugin will only need to be able to subscribe to confirmed transactions from the Value Tangle which contain blocks/payloads of certain type with certain assembly id in it: the event something like assemblyTransactionArrived. The rest will be done by the qnode plugin (which in the future may be detached from the node to separate device).

1 Like