Client libraries high level functionality proposal

Introduction

The current IOTA Client libraries have various ways of doing things, sometimes with a lot of overhead for trivial, repeating tasks that often don’t need most of the flexibility of the lower level API. This document proposes a first version of a common standard for high level functionality in the Client libraries that should work the same regardless of the language it’s implemented in or the specific implementation of the IOTA protocol now or in the (near) future. The idea behind this proposal is that the functions are generic enough for a longer term usage regardless of the underlying technology. The same function calls can be used pre- and post-coordicide even though they may do a lot of different things under the hood.

This is a work-in-progress design document that should be converted into a RFC once the initial model has been fleshed out. This proposal comes with another proposal for a generic, more future proof address format covered in a seperate topic.

Overall requirements

The higher level functionality should:

  • Work the same cross-implementation in terms of naming and arguments, regardless of the language of the client library (with the exception of language specific code style enforcing a certain way of naming functions).
  • Use as many sensible defaults as possible so that the least amount of configuration has to be done to work with it. For example depth can default to 3, MWM to 14.
  • Be implemented in a way that the basic functionality is always the same regardless of the protocol below it, so that it will still work in the exact same way even if we change the core protocol, transaction format or have coordicide implemented.
  • Have specific arguments well described in the documentation, for example “MWM stands for Minimum Weight Magnitude” is not enough, it should be obvious for any coder, regardless of IOTA experience, what a certain argument like MWM or Depth means.
  • Cover the most common use cases when working with IOTA as defined in the functionality below. If a certain functionality is not available for a given language (like for example MAM in Python) it should still implement the higher level function but with a NotImplemented error raised when called and a remark in the documentation.
  • Live next to the existing codebase, they are shortcuts that can be optionally used for faster development. The existing API can still be used for cases where you can’t use the higher level functionality.
  • Be as generic as possible, you should not even have to know about trytes in most cases in order to use it.
  • Have a seperate section in the documentation so people don’t get lost in all the details when starting out; Keeping it simple.
  • Have sensible error messages on exceptions describing exactly what’s wrong in a language any coder can understand or in case of specific IOTA related terms a reference to more information.
  • Provide a summary of the wrapped functionality so in case of custom solutions that don’t fit within the higher level functionality that can be used as a base instead. This can just be a small, maybe even hidden by default, remark in the docs.
  • Be extended once new functionality arrives, a good first additional candidate would be a higher level implementation for MAM once the new version is released.

Implementation proposal

Initializer

Initialization can be the same or very similar to the current API.

I prefer a very simple syntax for this that can easily be remembered so I prefer aliasing composeAPI in JS/Go to just IOTA and in Java

IotaAPI api = new IotaAPI.Builder().protocol("https").host("nodes.devnet.thetangle.org").port("443").build();

Can probably be a lot less bloated and easier to understand/remember if you can just do:

IotaAPI api = new IOTA(‘https://nodes.iota.org:443’);

Or even simpler with automated node selection and possible quorum:

IotaAPI api = new IOTA();

If we need to do value transactions we need to provide a seed:

IotaAPI api = new IOTA(‘SEED9999...’);

Optionally we can add additional default values for further calls here as well, for example default_mwm, since in most cases you always have the same MWM everywhere in your code, by using a default we can prevent code duplication in calling those functions.

We should add other optional arguments as well for state persistence when using the Account module internally, but those can have defaults like a iota.db state file or a path for state objects with the preferred adapter for storage. We need to make sure the state of the library is well documented with the higher level functionality so that people are very aware that they need both the state file and the seed in order to use it.

Seed is an optional argument for the initializer and is only used for value transactions, it’s optional for data only transactions.

Node quorum functionality is high on the wishlist. Provide some sensible defaults and if multiple nodes are given or no node at all it will pick X nodes and compares the results.

GetNodeInfo()

Already available in most libraries but useful for debugging purposes.

Only relevant for single node implementations without quorum.

Implementation as follows:

api = new IOTA()
print(api.getNodeInfo())

This returns an object with the node information as provided by the IRI API.

(private) NodeInSync([max_milestone_difference=0])

Returns True if the chosen node is in sync, False if not. It has one optional argument, max_milestone_difference with a default of 0, that is used to decide if a node is in sync or not. With the default value of 0 it is only marked as in sync if the milestone on the node is exactly the last issued milestone, but if you provide for example 2 it can be behind 2 milestones and still return True.

Mostly used internally for node quorum, maybe we should not even expose this in the public API.

Open question: What would be a sensible default for this, 0 might return out of sync if you call it at just the right time triggering a False without a good reason, but 1 or 2 is technically not fully in sync either.

SendValue(address, amount, [message], [tag])

A shortcut function to send a value transaction to a single address. Address can either be an address with checksum or a valid CDA (needs implemented checks that raise errors if invalid). The message is automatically converted to Trytes from a string, the tag only accepts a trytestring and raises an exception with a reference to what a valid tag can be if this is not passed.

Implementation as follows:

api = new IOTA()
bundle = api.sendValue(‘iota://ADDRESS9999...’, 1000, message=’Hi’)
print(bundle.hash)

In case of async this can work similar but with promises.

SendData(address, message, [tag], [encoding=’ascii’])

A shortcut function to send ASCII data to the tangle. In case the message is too long it will automatically split over several transactions in a bundle to the same address. The message is automatically converted to Trytes without the user having to know about this process.

Open question: This is what we currently support in the client libraries with asciiToTrytes; We currently do not have a built-in, generic way to send Unicode. This means that special characters (Japanese, Chinese, Russian, Emoji’s, etc) are not supported with this. Since we currently don’t have a generic way to send Unicode data I don’t think we should implement this yet in the higher level functions; But I do think we need a standardised way to do this a.s.a.p. Any thoughts on this? I’d suggest to just convert the UTF-8 bytes to trytes and use that as the standard. We might even want to think about replacing the default encoding method of AsciiToTrytes with this (even though it has overhead for just ascii data) so that messages are always in the same encoding by default regardless of characters/encoding used.

Implementation as follows:

api = new IOTA()
bundle = api.sendData(‘iota://ADDRESS9999...’, message=’Hey there’)
print(bundle.hash)

In case of async this can work similar but with promises.

GetData(address/bundle hash/transaction hash, [offset=0], [limit=100], [encoding=’ascii’])

A shortcut function to receive data from the tangle. This function will automatically stitch together multiple transactions from the same bundle in the right order to support longer messages and handle the automatically detected encoding if none is given. By default it returns the last 100 messages but limit and offset can be adjusted to your needs to get more or other sections.

Implementation as follows:

api = new IOTA()
messages = api.fetchData(‘iota://ADDRESS9999...’)
for message in messages:
    print(message) # This is just a string

In case of async this can work similar but with promises.

GenerateAddress([timeout_at], [multi_use], [expected_amount], [security_level=2], [legacy_format=False])

Returns a magnet link address, a new standard for sharing addresses with a checksum based on both address and optional arguments. This magnet link can be compatible with CDA’s or regular addresses making it forward compatible with a possible implementation of re-usable addresses.

Implementation as follows:

api = new IOTA(‘SEED9999...’);
addr = api.generateAddress()
# addr = ‘iota://ADDRESS99999...’

Or for a CDA:

api = new IOTA(‘SEED9999...’);
addr = api.generateAddress(timeout_at=’2020-01-01 00:00:00’, multi_use=False)
# addr = ‘iota://MBR….D/?timeout_at=1548337187&multi_use=0’

If you provide the legacy_format argument you’ll get the raw address with checksum (not the magnet link); this can be used in the transition period between old addresses and magnet links to still allow people to generate addresses for exchanges for example.

GetBalance()

Returns the balance in iota for the given seed, internally it checks the balances for addresses belonging to this seed and adds them up, based on the state.

Implementation as follows:

api = new IOTA(‘SEED9999...’);
balance = api.getBalance()
# balance = 1337

GetBalanceForAddress(address)

Returns the balance in iota for the given address.

Implementation as follows:

api = new IOTA();
balance = api.getBalanceForAddress(‘iota://ADDRESS9999...’)
# balance = 1337

Suggetions, thoughts and improvements to this proposal are very welcome!

3 Likes

We can use what is used by some of IRI’s functionality, where we consider the node to be in sync if it’s <= 5 milestones behind. This may be too lax for the certain user facing scenarios and may have to be weighed against the milestone issuance interval as well.

1 Like

Just my two cents…

Initializer

Having the seed passed into the initialization feels wrong to me. If you wanted to get the balance from multiple wallets you would have to create a whole new instance of the IOTA object, instead of reusing the existing one.
Also providing the seed at this point probably means the whole lib is more vulnerable to security issues.

GetNodeInfo

For this high level library do we really care about the node info, the response is very technical. Also for quorum does this make sense anyway.

NodeInSync

Not sure this is necessary for the public API, given the other methods exposed I am not sure why you would ever call this.

SendValue

I would pass the seed here instead of during initialization, it also reinforces that you need the seed to send value transactions.

Does it need to return the whole bundle, why not just the transaction hash (or a metadata structure). It also means that devs aren’t dependent on data within the structure if that changes in the future.

SendData

Maybe one option here is to change the last parameter to a converter method i.e. (input) => trytes. That way anybody could provide their own method for different data types.

We could supply some ready baked ones ASCII, UTF8, UTF16 etc
A default one could be a smart converter which tries to autodetect and use the best from the others.

Same applies about returning whole bundle here.

GetData

I think this is a bit confusing, as the inverse function of SendData I would only expect it to return 1 message. Which in turn means that you should only ever send it the transaction hash.

It also says fetchData in the example :slight_smile:

Maybe a separate function for getting multiple items with the paging support (paging would also require changes to IRI API as at the moment it doesn’t guarantee which transactions it will return), this might also need an epoch passing, as the items might change between page calls)

The same converter instead of encoding option as SendData could also be possible.

GenerateAddress

I would pass the seed here instead of during initialization, it reinforces that the seed is needed to generate addresses.

GetBalance

I would pass the seed here instead of during initialization.

GetBalanceForAddress

Should also support raw address format as input.

What’s Missing

How about a method for checking if a transaction/payment has confirmed.

PaymentIsConfirmed(transactionHash) => boolean

1 Like

Thank you very much Martyn, great feedback and I almost fully agree to all your points :slight_smile: A lot of this is about sensible default and there are multiple ways to achieve this, let’s make your suggestions part of the proposal since they all make a lot of sense. Here some comments where I have something to add:

Seed not in initializer

I did it like this because I think that in most cases just a single seed will be used, but you are totally right that it’s inefficient with multiple seeds. I choose this route initially because you have the least amount of repeating code (you only supply your seed once), but you are right that it’s more secure to not keep the seed somewhere in a class attribute.

GetNodeInfo and NodeInSync not high level enough

True, I’ve doubted about adding them due to these reasons, I left them mainly in the spec since they are internally used by the quorum and internal checks of the other functions but they don’t need to be part of the higher level documentation/spec to the public.

SendValue returns the whole bundle, not just the transaction hash

Valid point, a whole bundle contains more info but also exposes the lower level transaction format which might change which is undesired. I think your suggestion to just expose a metadata structure or bundle hash makes a lot of sense.

SendData conversion method instead of ascii/utf8

Good point, kind of a power feature for most cases but good to support it for sure, with the default of auto detecting the best suitable one that sounds reasonable to me. Maybe a hybrid where you can just input a string for the build in ones or a custom function if you want something special. I’d vouch for a default of UTF-8 encoding strings over the default ascii we use now since that covers the most common use cases keeping compatibility with all languages/character sets making it possibly a more sensible default.

GetData returns multiple messages

The idea is that the higher level sendData splits larger messages that don’t fit into a single TX over multiple and sends it in one bundle, GetData then turns it into 1 message again but from multiple transactions. If it just returns a single message what would it return if someone posts many messages to the same address? I think we need to brainstorm a bit more about what’s sensible when it comes to sending and receiving data since we currently have a lot of ways to do this. Addresses can have multiple transactions, those transactions can be just 1 per bundle or multiple per bundle, a single transaction alone might not be a full message, etc.

GetBalanceForAddress should support raw addresses

This proposal assumes the other proposal for changing the address format is implemented, but I agree it should support the raw address as well.

Add PaymentIsConfirmed

Sounds great!

I’ll change my doc to reflect these changes for a more final proposal, thanks again!

2 Likes

The high-level method looks good for applications. Functions in this proposal seem they are designed for applications instead of a library. As a library, we should keep APIs simple, have good performance and well documented.

  • Work the same cross-implementation in terms of naming and arguments, regardless of the language of the client library (with the exception of language specific code style enforcing a certain way of naming functions).

I agree to have a unified function name between languages but considering different features between languages, having the same arguments is odd.

Let me start by saying that I feel, that the root problem of low library utilization most probably stems from the exoticism of IOTA in regards to not having reusable addresses and using trinary.

If we don’t fix the two problems above, efforts to make current libraries easier to work with is almost a senseless efford, because it roots from the assumption, that easier to use function bring enough momentum to the libraries. I believe this assumption to be wrong, as the introduction of the account module in the libraries didn’t bring any gains in usage either, even though it abstracted the usage with IOTA extremely in comparison to the raw old API functions (at least for the Go lib this is the case, I won’t take responsibility for other libs).

Requirements

Be implemented in a way that the basic functionality is always the same regardless of the protocol below it, so that it will still work in the exact same way even if we change the core protocol, transaction format or have coordicide implemented.

I think it will be very hard to make sure that this actually stays true, especially when we right now do not have a clear picture how Coordicide will look like. Still good to have it on the back of your head somewhere.

Initializer

new IOTA():

  • What kind of errors does this initializer throw?
  • When not specifying any node URL, what are the quorum thresholds and what nodes are in the automated node selection?
  • Please list the optional arguments you mention so there’s a clear picture what those translate to in praxis. For example, how would specify a path for the to persisted database?
  • This initializer needs a possibility to add an object which will give the seed at runtime from any source, instead of having to pass the seed up on initialization. For example, I want that everything is signed on another local machine instead of within the program, so I need a way to be able to implement that.

GetNodeInfo()

  • What happens if the initializer was initialized with quorum?

NodeInSync([max_milestone_difference=0])

  • What will intern functions do when they encounter that the node is not in sync?
  • What happens in case of a quorum, how will it be handled there?

You’re right that this shouldn’t be a public call.

SendValue(address, amount, [message], [tag])

  • Please mention how as a library user I can check when the bundle is confirmed because with a very high chance, further application logic has to be executed when the transfer actually got confirmed. Given the current function signature, I see it only returns a bundle.
  • How does this function behave if the target address was spent, does it check that?
  • In your example code the initializer is not initialized without a seed, on what seed does this function now operate? Or should it return an error for this function, as no seed was supplied?

SendData(address, message, [tag], [encoding=’ascii’])

From my perspective functions which upload data to the Tangle by simply issuing transactions without any authentication are for real production applications useless. I believe the right approach to data handling on the Tangle is using MAM, so I feel that such function is a toy-function, because in praxis it is conceptually broken:

  • no authenticity who send this data
  • snapshotted away into oblivion as no one retains it

However, of course a developer can upload signed data which doesn’t use MAM.

Anyway, given this function, here are my concerns:

  • Like with SendValue(), perhaps there is a requirement to be able to know when the message was included into the confirmed part of the Tangle, how would I know about such fact using this API?

GetData(address/bundle hash/transaction hash, [offset=0], [limit=100], [encoding=’ascii’])

Like SendData() this function is also broken in the sense that the data just lies around the given address or within the bundle or transaction. How do we make sure that people understand that the data might be gone any minute and not be available for later retrieval?

  • Does this function remove duplicated messages?

GenerateAddress([timeout_at], [multi_use], [expected_amount], [security_level=2], [legacy_format=False])

  • How can I listen on the generated address for any kind of events like deposits? This is a must and very important for applications.
  • How do I know which addresses I’ve generated? Does the developer need to store them somewhere in order to be able to retrieve them later again?

GetBalance()

  • Will it give the total amount of balance, even if funds on certain addresses can not be used yet to fund a transfer, given the CDA is not yet usable? (This is the reason why there’s AvailableBalance()/TotalBalance() calls in the account module)

GetBalanceForAddress(address)

:+1:

1 Like

Converting utf-8 bytes to trytes sounds like a good idea for general encoding. It is experimental, but works in the python lib with TryteString.from_unicode() and TryteString.decode().

I like the idea to have simple and well documented methods that abstract away the complexity of the protocol for application developers. Probably they don’t have to know every little detail and can work with sensible defaults. To some extent, the so called “Extended API” methods move in this direction, but still kind of low level.

An open question:
In my opinion, in order to make these high level methods easy to use for developers, we need state management under the hood, therefore they will rely on the account module. Instead of adding another layer of abstraction, could we make the account module more user friendly by incorporating such methods?

@dave
Whilst adding another layer over the libraries right now will most definitely simplify things, i don’t think it is something we should do. This would mean yet another X codebases we need to manage, and another different API users could find when looking for code examples (As this high level will very likely not be sufficient for any advanced application, Account and Core would still need documentation).

I would rather work with you and the library team to add these features in some form or another to the account module, and simplify the usage of said module in a way you think is more understandable for the more basic usages.

For example; account module now has the following function for multiple cases: send(Recipient recipient)
We could split this in two with sendValue and sendData and make the contract easier to understand.

About the documentation, i do fully agree we need more and better. Although we expect people to read the docs on our documentation portal, they don’t always do…

Thank you all for the good feedback all! The whole reason for this proposal is to not bother the end user with all the exoticism of IOTA by just not exposing it to them by default. The idea is to make the libraries so easy to use that even without any specific IOTA knowledge any developer can work with it. For the ones familiar with Python and its ecosystem: What I’m trying to achieve is the usability of the requests library, while urllib2 worked just fine it provided so many valuable abstractions and sensible defaults it’s actually the defacto way to do requests in Python now. I think barely a handful of developers right now cares about Ternary at all when trying to use IOTA, single use addresses are annoying and make things difficult and all the conversions needed to push out a single transaction don’t help either.

While I really like the account module it has some flaws which I think contribute to its adoption:

  1. It uses a different address format by default which is incompatible with Trinity, making user interaction with Account module CDA’s hard and without the benefits of the extra checks of the CDA’s

  2. It’s seen/promoted as a separate ‘module’, something ‘optional’, not part of the core; Seperately documented in the docs instead of as a basic part of the client libs. While in my eyes it should actually be the default way to manage your addresses and transactions.

It should not be “another codebase to manage”, just a mere wrapper in the existing client libraries making use of the already existing code available; A entry point with sensible defaults for at least 90% of the uses of IOTA. My proposal isn’t complete; It should contain MAM related calls as well but that doesn’t make a lot of sense to implement while we are waiting for MAM 1.1. I don’t agree with @brord on that it won’t be sufficient for advanced applications, as long as the functions have sensible defaults but wrap the option to overwrite existing parameters I don’t see a reason why this can’t be the default API to use; Apart from MAM what other advanced use cases are we talking about here?

Taking the Account module as the basis for implementing this sounds great to me, we need it stateful regardless; but it needs to become a first class citizen both in the docs and in the code and examples. We do however need to resolve things like the inconsistent address format in that case. We also need to check out how relevant the account module still is with re-usable addresses. If we keep supporting the WOTS-non-re-usable addresses for a while it makes a lot of sense to actually use the Account module as a base and work from there.

Again, please note that I’m not trying to put a lot of work on your plates; Most of it is just creating simple, language-agnostic wrapper functions/shortcuts in the existing codebase with sensible defaults for the most common use cases. Many of the use-cases have a lot of duplicate, over-verbose code converting Ascii to Trytes, splitting and merging back transactions in a bundle, etc. Why not just abstract this away so people don’t have to look it up or think about it for almost all of their use-cases?

1 Like

The account module was a good step in the right direction, but it’s a bit useless if you can’t use it with Trinity and basically everything else out there at the moment because of the other address format. Of course you can convert them with additional custom code, but that’s far too cumbersome for most people and shouldn’t be necessary.

huhn started to create an iota payment module and asked me if I want to join. It should be an easy to use npm module, because the client lib isn’t really easy to use and you always need a lot code to get something with payments properly working (keep track of the key indexes, prepare transfers, send them, maybe store the txhash somewhere…). You can find the module here.
We havn’t really taken a close look at the account module because of the different address format and because we had never seen the account module in use anywhere (in JS). But over time I have noticed that many things from our payment module and the account module are quite similar, because they should solve the same problems. For example you don’t want to check an address for incoming txs forever, the account modul handles it with the CDAs but because we wanted to use the normal address format we checked the address only for a period of time which you can define yourself. It’s not as good as the CDA option, but it works with Trinity and everything else that was in use. We also wanted to store additional data next to created deposits/payments and withdrawals/payouts which can be used for an application and we wanted to have optional zmq and websockets. Maybe an official IOTA module shouldn’t have an option to store additional data, but for little projects it’s very useful because you don’t need an additional database, get the tx data and put it in there. An example application is this faucet bot for discord. The payment-module far from perfect, but it does what we wanted and found useful for nodejs applications.

I hope you can use it as input. I need to admit that I don’t know how much custom plugins can do with the account module (in JS it’s not possible to have custom plugins) but if they could do everything I can do with the client lib, it would be great. Then one could use the high level functionality for the most stuff and if that’s not enough one can just add a plugin, which can easily be shared with others.

Maybe it would also make sense to allow a CDA here, if you create a CDA and want to check if a payment to it has been confirmed you don’t want to query the transactions for the address first and then call it. And if you want to get the txHash you need findTransactions() which isn’t here or how do you get it?

1 Like

Thanks @dave, really appreciate the effort! Most methods and default values do make sense. But I also agree that most root problems are out of this proposal like reusable address and trytes. This proposal does mention to make trytes easier for end users to use but it’s always not that easy imo. The type of itself and methods/operations it can perform will always be somewhat different in everyone’s mind. The worst case scenario would be something like javascript’s holy trinity. To be honest I also don’t know what’s a good abstraction / generics for trytes either. We just take it as 3 trits per 1 byte encoding in bee team at the moment. Maybe we just need to let user worry about this when converting to other encodings like ascii or utf8.

Speaking of which, I do hope we can set utf8 as default when encoding data/messages. Because in many cases, users will usually want to encode lots of different characters that not supported by ascii. So yes, this is probably the issue we should do it asap. Perhaps it can already be an individual RFC for this. Anyway, I don’t have strong opinion other than these above until RFC is proposed.

1 Like