Coin Handler Base Classes

Submodules

BaseLoader

class payments.coin_handlers.base.BaseLoader.BaseLoader(symbols: list = None)[source]

Bases: abc.ABC

BaseLoader - Base class for Transaction loaders

A transaction loader loads incoming transactions from one or more cryptocurrencies or tokens, whether through a block explorer, or through a direct connection to a local RPC node such as steemd or bitcoind using connection settings set by the user in their Django settings.

Transaction loaders must be able to initialise themselves using the following data:

  • The coin symbols self.symbols passed to the constructor
  • The setting_xxx fields on self.coin payments.models.Coin
  • The Django settings from django.conf import settings
  • They should also use the logging instance settings.LOGGER_NAME

If your class requires anything to be added to the Coin object settings, or the Django settings file, you should write a comment listing which settings are required, which are optional, and their format/type.

e.g. (Optional) settings.STEEM_NODE - list of steem RPC nodes, or string of individual node, URL format

They must implement all of the methods in this class, as well as configure the provides list to display the tokens/coins that this loader handles.

list_txs(batch=100) → Generator[dict, None, None][source]

The list_txs function processes the transaction data from load(), as well as handling any pagination, if it’s required (e.g. only retrieve batch transactions at a time from the data source)

It should first check that load() has been ran if it’s required, if the data required has not been loaded, it should call self.load()

To prevent memory leaks, this must be a generator function.

Below is an example of a generator function body, it loads batch transactions from the full transaction list, pretends to processes them into txs, yields them, then loads another batch after the calling function has iterated over the current txs

>>> t = self.transactions   # All transactions
>>> b = batch
>>> finished = False
>>> offset = 0
>>> # To save memory, process 100 transactions per iteration, and yield them (generator)
>>> while not finished:
>>>     txs = []    # Processed transactions
>>>     # If there are less remaining TXs than batch size, get remaining txs and finish.
>>>     if (len(t) - offset) < batch:
>>>         finished = True
>>>     # Do some sort-of processing on the tx to make it conform to `Deposit`, then append to txs
>>>     for tx in t[offset:offset + batch]:
>>>         txs.append(tx)
>>>     offset += b
>>>     for tx in txs:
>>>         yield tx
>>>     # At this point, the current batch is exhausted. Destroy the tx array to save memory.
>>>     del txs
Parameters:batch (int) – Amount of transactions to process/load per each batch
Returns Generator:
 A generator returning dictionaries that can be imported into models.Deposit

Dict format:

{txid:str, coin:str (symbol), vout:int, tx_timestamp:datetime,
 address:str, from_account:str, to_account:str, memo:str, amount:Decimal}

vout is optional. One of either {from_account, to_account, memo} OR {address} must be included.

load(tx_count=1000)[source]

The load function should prepare your loader, by either importing all of the data required for filtering, or setting up a generator for the list_txs() method to load them paginated.

It does NOT return anything, it simply creates any connections required, sets up generator functions if required for paginating the data, and/or pre-loads the first batch of transaction data.

Parameters:tx_count – The total amount of transactions that should be loaded PER SYMBOL, most recent first.
Returns:None
provides = []

BaseManager

class payments.coin_handlers.base.BaseManager.BaseManager(symbol: str)[source]

Bases: abc.ABC

BaseManager - Base class for coin/token management

A coin manager handles balance checking, sending, and issuing of one or more cryptocurrencies or tokens, generally through a direct connection to a local/remote RPC node such as steemd or bitcoind using connection settings set by the user in their Django settings.

Coin managers must be able to initialise themselves using the following data:

  • The coin symbol self.symbol passed to the constructor
  • The setting_xxx fields on self.coin payments.models.Coin
  • The Django settings from django.conf import settings

If your class requires anything to be added to the Coin object settings, or the Django settings file, you should write a comment listing which settings are required, which are optional, and their format/type.

e.g. (Optional) settings.STEEM_NODE - list of steem RPC nodes, or string of individual node, URL format

They must implement all of the methods in this class, set the can_issue boolean for detecting if this manager can be used to issue (create/print) tokens/coins, as well as configure the provides list to display the tokens/coins that this manager handles.

address_valid(address) → bool[source]

A simple boolean method, allowing API requests to validate the destination address/account prior to giving the user deposit details.

Parameters:address – An address or account to send to
Return bool:Is the address valid? True if it is, False if it isn’t
balance(address: str = None, memo: str = None, memo_case: bool = False) → decimal.Decimal[source]

Return the balance of self.symbol for our “wallet”, or a given address/account, optionally filtered by memo

Parameters:
  • address – The address or account to get the balance for. If None, return our total wallet (or default account) balance.
  • memo – If not None (and coin supports memos), return the total balance of a given memo
  • memo_case – Whether or not to total memo’s case sensitive, or not. False = case-insensitive memo
Raises:

AccountNotFound – The requested account/address doesn’t exist
Return Decimal:

Decimal() balance of address/account, optionally balance (total received) of a given memo
can_issue = False

If this manager supports issuing (creating/printing) tokens/coins, set this to True

get_deposit() → tuple[source]
Return tuple:If the coin uses addresses, this method should return a tuple of (‘address’, coin_address)
Return tuple:If the coin uses accounts/memos, this method should return a tuple (‘account’, receiving_account) The memo will automatically be generated by the calling function.
health() → Tuple[str, tuple, tuple][source]

Return health data for the passed symbol, e.g. current block height, block time, wallet balance whether the daemon / API is accessible, etc.

It should return a tuple containing the manager name, the headings for a health table, and the health data for the passed symbol (Should include a symbol or coin name column)

You may use basic HTML tags in the health data result list, such as <b> <em> <u> and <span style=""></span>

Return tuple health_data:
 (manager_name:str, headings:list/tuple, health_data:list/tuple,)
health_test() → bool[source]

To reduce the risk of unhandled exceptions by sending code, this method should do some basic checks against the API to test whether the coin daemon / API is responding correctly.

This allows code which calls your send() or issue() method to detect the daemon / API is not working, and then delay sending/issuing until later, instead of marking a convert / withdrawal status to an error.

The method body should be wrapped in a try/except, ensuring there’s a non-targeted except which returns False

Return bool:True if the coin daemon / API appears to be working, False if it’s not
issue(amount: decimal.Decimal, address: str, memo: str = None, trigger_data: Union[dict, list] = None) → dict[source]

Issue (create/print) tokens to a given address/account, optionally specifying a memo if supported

Parameters:
  • amount (Decimal) – Amount of tokens to issue, as a Decimal()
  • address – Address or account to issue the tokens to
  • memo – Memo to issue tokens with (if supported)
  • trigger_data (dict) – Metadata related to this issue transaction (e.g. the deposit that triggered this)
Raises:
  • IssuerKeyError – Cannot issue because we don’t have authority to (missing key etc.)
  • IssueNotSupported – Class does not support issuing, or requested symbol cannot be issued.
  • AccountNotFound – The requested account/address doesn’t exist
Return dict:

Result Information

Format:

dict {
    txid:str - Transaction ID - None if not known,
    coin:str - Symbol that was sent,
    amount:Decimal - The amount that was sent (after fees),
    fee:Decimal    - TX Fee that was taken from the amount,
    from:str       - The account/address the coins were issued from.
                     If it's not possible to determine easily, set this to None.
    send_type:str  - Should be statically set to "issue"
}
orig_symbol = None

The original unique database symbol ID

provides = []

A list of token/coin symbols in uppercase that this loader supports e.g:

provides = ["LTC", "BTC", "BCH"]
send(amount: decimal.Decimal, address: str, from_address: str = None, memo: str = None, trigger_data: Union[dict, list] = None) → dict[source]

Send tokens to a given address/account, optionally specifying a memo and sender address/account if supported

Your send method should automatically subtract any blockchain transaction fees from the amount sent.

Parameters:
  • amount (Decimal) – Amount of coins/tokens to send, as a Decimal()
  • address – Address or account to send the coins/tokens to
  • memo – Memo to send coins/tokens with (if supported)
  • from_address – Address or account to send from (if required)
  • trigger_data (dict) – Metadata related to this send transaction (e.g. the deposit that triggered this)
Raises:
  • AuthorityMissing – Cannot send because we don’t have authority to (missing key etc.)
  • AccountNotFound – The requested account/address doesn’t exist
  • NotEnoughBalance – Sending account/address does not have enough balance to send
Return dict:

Result Information

Format:

dict {
  txid:str - Transaction ID - None if not known,
  coin:str - Symbol that was sent,
  amount:Decimal - The amount that was sent (after fees),
  fee:Decimal    - TX Fee that was taken from the amount,
  from:str       - The account(s)/address(es) the coins were sent from. if more than one, comma separated.
                   If it's not possible to determine easily, set this to None.
  send_type:str  - Should be statically set to "send"
}
send_or_issue(amount, address, memo=None, trigger_data: Union[dict, list] = None) → dict[source]

Attempt to send an amount to an address/account, if not enough balance, attempt to issue it instead. You may override this method if needed.

Parameters:
  • amount (Decimal) – Amount of coins/tokens to send/issue, as a Decimal()
  • address – Address or account to send/issue the coins/tokens to
  • memo – Memo to send/issue coins/tokens with (if supported)
  • trigger_data (dict) – Metadata related to this issue transaction (e.g. the deposit that triggered this)
Raises:
  • IssuerKeyError – Cannot issue because we don’t have authority to (missing key etc.)
  • IssueNotSupported – Class does not support issuing, or requested symbol cannot be issued.
  • AccountNotFound – The requested account/address doesn’t exist
Return dict:

Result Information

Format:

dict {
  txid:str       - Transaction ID - None if not known,
  coin:str       - Symbol that was sent,
  amount:Decimal - The amount that was sent (after fees),
  fee:Decimal    - TX Fee that was taken from the amount,
  from:str       - The account(s)/address(es) the coins were sent from. if more than one, comma separated.
                   If it's not possible to determine easily, set this to None.
  send_type:str  - Should be set to "send" if the coins were sent, or "issue" if the coins were issued.
}
symbol = None

The native coin symbol, e.g. BTC, LTC, etc. (non-unique)

BatchLoader

class payments.coin_handlers.base.BatchLoader.BatchLoader(symbols: list = None)[source]

Bases: payments.coin_handlers.base.BaseLoader.BaseLoader, abc.ABC

BatchLoader - An abstract sub-class of BaseLoader which comes with some pre-written batching/chunking functions

Copyright:

+===================================================+
|                 © 2019 Privex Inc.                |
|               https://www.privex.io               |
+===================================================+
|                                                   |
|        CryptoToken Converter                      |
|                                                   |
|        Core Developer(s):                         |
|                                                   |
|          (+)  Chris (@someguy123) [Privex]        |
|                                                   |
+===================================================+

This class is designed to save you time from re-writing your own “batching” / “chunking” functions.

Batching / chunking is a memory efficiency technique to prevent RAM leaks causing poor performance or crashes. Instead of loading all 1K - 10K transactions into memory, you load only a small amount of transactions, such as 100 transactions, then you use a Python generator (the yield keyword) to return individual transactions, quietly loading the next “batch” / “chunk” of 100 TXs after the first set has been processed, without interrupting the caller’s for loop or other iteration.

This allows other functions to iterate over the transactions and process them on the fly, instead of having to load the entire 1-10K transaction list into memory first.

The use of generators throughout this class helps to prevent the problem of RAM leaks due to constant duplication of the transaction list (e.g. self.transactions, self.filtered_txs, self.cleaned_txs), especially when the transaction lists contains thousands of transactions.

To use this class, simply extend it (instead of BaseLoader), and make sure to implement the two abstract methods:

  • load_batch - Loads and stores a small batch of raw (original format) transactions for a given coin
  • clean_txs - Filters the loaded TXs, yielding TXs (conformed to be compatible with models.Deposit)
    that were received by us (not sent), and various sanity checks depending on the type of coin.

If your Loader is for a coin which uses an account/memo system, set self.need_account = True before calling BatchLoader’s constructor, and it will remove coins in self.symbols/coins that do not have a non-empty/null our_account column.

You’re free to override any methods if you need to, just make sure to call this class’s constructor __init__ before/after your own constructor, otherwise some methods may break.

Flow of this class:

Transaction loading cron
   |
   V--> __init__(symbols:list)
   |--> load(tx_count:int)
   |--> list_txs(batch:int) -> _list_txs(coin:Coin, batch:int)
   V                              |--> load_batch(account, symbol, offset)
                                  V--> clean_txs(account, symbol, txs)
clean_txs(symbol: str, transactions: Iterable[dict], account: str = None) → Generator[dict, None, None][source]

Filters a list of transactions transactions as required, yields dict’s conforming with models.Deposit

Important things when implementing this function:

  • Make sure to filter out transactions that were sent from our own wallet/account - otherwise internal transfers will cause problems.
  • Make sure each transaction is destined to us
    • If your loader is account-based, make sure to only yield transactions where tx[“to_account”] == account.
    • If your loader is address-based, make sure that you only return transactions that are being received by our wallet, not being sent from it.
      • If account isn’t None, assume that you must yield TXs sent to the given crypto address account
  • If your loader deals with smart contract networks e.g. ETH, EOS, make sure that you only return transactions valid on the matching smart contract, don’t blindly trust the symbol!
  • Make sure that every dict that you yield conforms with the return standard shown for BaseLoader.list_txs()
  • While transactions is normally a list<dict> you should assume that it could potentially be a Generator, writing the code Generator-friendly will ensure it can handle both lists and Generator’s.

Example:

>>> def clean_txs(self, symbol: str, transactions: Iterable[dict],
>>>               account: str = None) -> Generator[dict, None, None]:
>>>     for tx in transactions:
>>>         try:
>>>             if tx['from'].lower() == 'tokens': continue       # Ignore token issues
>>>             if tx['from'].lower() == account: continue        # Ignore transfers from ourselves.
>>>             if tx['to'].lower() != account.lower(): continue  # If we aren't the receiver, we don't need it.
>>>             clean_tx = dict(
>>>                 txid=tx['txid'], coin=symbol, tx_timestamp=parse(tx['timestamp']),
>>>                 from_account=tx['from'], to_account=tx['to'], memo=tx['memo'],
>>>                 amount=Decimal(tx['quantity'])
>>>             )
>>>             yield clean_tx
>>>         except:
>>>             log.exception('Error parsing transaction data. Skipping this TX. tx = %s', tx)
>>>             continue
Parameters:
  • symbol – The symbol of the token being filtered
  • transactions – A list<dict> of transactions to filter
  • account – The ‘to’ account or crypto address to filter by (only required for account-based loaders)
Returns:

A generator yielding dict’s conforming to models.Deposit, check the PyDoc return info for coin_handlers.base.BaseLoader.list_txs() for current format.
list_txs(batch=100) → Generator[dict, None, None][source]

Yield transactions for all coins in self.coins as a generator, loads transactions in batches of batch and returns them seamlessly using a generator.

If load() hasn’t been ran already, it will automatically call self.load()

Parameters:batch – Amount of transactions to load per batch
Return Generator[dict, None, None]:
 Generator yielding dict’s that conform to models.Deposit
load(tx_count=1000)[source]

Simply imports tx_count into an instance variable, and then sets self.loaded to True.

If self.need_account is set to True by a child/parent class, this method will remove any coins from self.coins and self.symbols which have a blank/null our_account in the DB, ensuring that you can trust that all coins listed in symbols/coins have an our_account which isn’t empty or None.

Parameters:tx_count (int) – The amount of transactions to load per symbol specified in constructor
load_batch(symbol, limit=100, offset=0, account=None)[source]

This function should load limit transactions in their raw format from your data source, skipping the offset newest TXs efficiently, and store them in the instance var self.transactions

If you use the included decorator decorators.retry_on_err(), if any exceptions are thrown by your method, it will simply re-run it with the same arguments up to 3 tries by default.

Basic implementation:

>>> @retry_on_err()
>>> def load_batch(self, symbol, limit=100, offset=0, account=None):
>>>     self.transactions = self.my_rpc.get_tx_list(limit, offset)
Parameters:
  • symbol – The symbol to load a batch of transactions for
  • limit – The amount of transactions to load
  • offset – Skip this many transactions (most recent first)
  • account – An account name, or coin address to filter transactions using

SettingsMixin

Copyright:

+===================================================+
|                 © 2019 Privex Inc.                |
|               https://www.privex.io               |
+===================================================+
|                                                   |
|        CryptoToken Converter                      |
|                                                   |
|        Core Developer(s):                         |
|                                                   |
|          (+)  Chris (@someguy123) [Privex]        |
|                                                   |
+===================================================+
class payments.coin_handlers.base.SettingsMixin.SettingsMixin[source]

Bases: object

SettingsMixin - A mixin that can be used by coin loaders/managers for easy access to database/file settings, with handling of default settings

Copyright:

+===================================================+
|                 © 2019 Privex Inc.                |
|               https://www.privex.io               |
+===================================================+
|                                                   |
|        CryptoToken Converter                      |
|                                                   |
|        Core Developer(s):                         |
|                                                   |
|          (+)  Chris (@someguy123) [Privex]        |
|                                                   |
+===================================================+
all_coins

Since this is a Mixin, it may be self.coin: Coin, or self.coins: List[Coin]. This property detects whether we have a single coin, or multiple, and returns them as a dict.

Return dict coins:
 A dict<str,Coin> of supported coins, mapped by symbol
setting_defaults = {'host': '127.0.0.1', 'password': None, 'user': None}

If a setting isn’t specified, use this dict for defaults, include both RPC defaults and custom json defaults

settings

Get all settings, mapped by coin symbol (each coin symbol dict contains custom json settings merged)

Return dict settings:
 A dictionary mapping coin symbols to settings
use_coind_settings = True

If True, merges symbol settings from settings.COIND_RPC with precedence over database Coin settings

Override this to False in child classes to disable loading from the settings file

Base Decorators

payments.coin_handlers.base.decorators.retry_on_err(max_retries: int = 3, delay: int = 3, **retry_conf)[source]

Decorates a function or class method, wraps the function/method with a try/catch block, and will automatically re-run the function with the same arguments up to max_retries time after any exception is raised, with a delay second delay between re-tries.

If it still throws an exception after max_retries retries, it will log the exception details with fail_msg, and then re-raise it.

Usage (retry up to 5 times, 1 second between retries, stop immediately if IOError is detected):

>>> @retry_on_err(5, 1, fail_on=[IOError])
... def my_func(self, some=None, args=None):
...     if some == 'io': raise IOError()
...      raise FileExistsError()

This will be re-ran 5 times, 1 second apart after each exception is raised, before giving up:

>>> my_func()

Where-as this one will immediately re-raise the caught IOError on the first attempt, as it’s passed in fail_on:

>>> my_func('io')
Parameters:
  • max_retries (int) – Maximum total retry attempts before giving up
  • delay (int) – Amount of time in seconds to sleep before re-trying the wrapped function
  • retry_conf – Less frequently used arguments, pass in as keyword args:
  • (list) fail_on: A list() of Exception types that should result in immediate failure (don’t retry, raise)
  • (str) retry_msg: Override the log message used for retry attempts. First message param %s is func name, second message param %d is retry attempts remaining
  • (str) fail_msg: Override the log message used after all retry attempts are exhausted. First message param %s is func name, and second param %d is amount of times retried.

Base Exceptions

exception payments.coin_handlers.base.exceptions.AccountNotFound[source]

Bases: payments.coin_handlers.base.exceptions.CoinHandlerException

The sending or receiving account requested doesn’t exist

exception payments.coin_handlers.base.exceptions.AuthorityMissing[source]

Bases: payments.coin_handlers.base.exceptions.CoinHandlerException

Missing private key or other authorization for this operation

exception payments.coin_handlers.base.exceptions.CoinHandlerException[source]

Bases: Exception

Base exception for all Coin handler exceptions to inherit

exception payments.coin_handlers.base.exceptions.DeadAPIError[source]

Bases: payments.coin_handlers.base.exceptions.CoinHandlerException

A main API, e.g. a coin daemon or public node used by this coin handler is offline.

exception payments.coin_handlers.base.exceptions.IssueNotSupported[source]

Bases: payments.coin_handlers.base.exceptions.CoinHandlerException

This class does not support issuing, the token name cannot be issued, or other issue problems.

exception payments.coin_handlers.base.exceptions.IssuerKeyError[source]

Bases: payments.coin_handlers.base.exceptions.AuthorityMissing

Attempted to issue tokens you don’t have the issuer key for

exception payments.coin_handlers.base.exceptions.MissingTokenMetadata[source]

Bases: payments.coin_handlers.base.exceptions.CoinHandlerException

Could not process a transaction or run the requested Loader/Manager method as required coin metadata is missing, such as payments.models.Coin.our_account or a required key in the custom JSON settings.

exception payments.coin_handlers.base.exceptions.NotEnoughBalance[source]

Bases: payments.coin_handlers.base.exceptions.CoinHandlerException

The sending account does not have enough balance for this operation

exception payments.coin_handlers.base.exceptions.TokenNotFound[source]

Bases: payments.coin_handlers.base.exceptions.CoinHandlerException

The token/coin requested doesn’t exist

Module contents