Ripple: How to monitor multiple accounts reliably and effectively?

0

rpc = RippleNetwork(app.config['RIPPLE_SERVER'])
poll = RippleNetwork(app.config['RIPPLE_SERVER'])
last_ledger = Variables.get('ripple_last_ledger')
poll.subscribe(streams=['ledger'])

while True:
    trans = []
    try:
        rst = rpc.account_tx(account=withdraw_wallet, ledger_index_min=last_ledger, limit=1000)
        trans.extend(rst['transactions'])
        while rst.get('marker') is not None:
            rst = rpc.account_tx(account=withdraw_wallet, ledger_index_min=last_ledger, marker=rst['marker'], limit=1000)
            trans.extend(rst['transactions'])

    except RippleException as e:
        if e.resp['error'] == 'lgrIdxsInvalid':
            log.debug('ledger_index_min too new, sleeping for 10s')
            time.sleep(10)
            continue

        raise

    for rec in trans:
        tx = rec['tx']
        log.debug('Got tx %s', tx['hash'])

        if tx['TransactionType'] != 'Payment':
            continue

        if tx['Destination'] != withdraw_wallet:
            continue

        log.info('Queue tx %s', tx['hash'])
        tube.put(str(tx['hash']))

    last_ledger = rst['ledger_index_max'] + 1
    Variables.set('ripple_last_ledger', last_ledger)
    db.session.commit()
    # Wait for next(or several) ledger close here.

Code above is my current implementation, but this method does not scale to thousands of accounts.

Is there any 'best practice'?

Proton

Posted 2014-04-06T08:01:19.993

Reputation: 103

First, unless you're catching up on past transactions use subscribe instead of repeated account_tx calls. Second, for a large number of accounts it can be more efficient to just subscribe to the the full transaction stream and filter it down to the transactions you care about. – dchapes – 2014-04-06T08:54:47.840

dchapes: why don't you add that as an answer? I agree that if you're monitoring a significant number of accounts, it's best to get the full stream and filter. – Eric S – 2014-04-06T10:28:52.020

Answers

0

I'm not sure a large code paste is the best way to get your example across (I'm unfamiliar with the library/API you're using for example). Assuming it's correct to summarize that you are looping, repeatedly doing account_tx RPC calls with at most a ten second delay; then that's not efficient even for a small number of accounts.

For a small-ish number of accounts, I'd recommend doing something like your loop only on restart to catchup from a last known ledger you've processed. (For example, after restarting your monitoring after an outage or disconnect). Once caught up it's better to use the subscribe RPC with the set of Ripple accounts you're interested in. The server will send you a notification anytime any transaction touches any of the accounts you list. This avoids polling and is far more efficient, especially during periods where the accounts are idle. (I'm only familar with using subscribe over a websocket, and it works very well for this even over days; I don't know what library support for this is like).

Once you go beyond a certain threshold of accounts you are trying to watch, especially if they are very active accounts, it may be easier to use the subscribe RPC to subscribe to all transactions and filter those for just the ones effecting the accounts you're interested in. Also, at that point it may become a good idea to do such queries against your own locally running rippled rather than rely on a public Ripple server.

The point at which it's best to switch from subscribing to a list of accounts versus subscribing to all transactions is unclear to me. For starters, there may be a limit to the number of accounts you can specify in a subscribe call; if so perhaps modify a locally running rippled to remove any such limit and have it do the filtering may be appropriate.

dchapes

Posted 2014-04-06T08:01:19.993

Reputation: 1 731

Thanks for replying. The RippleNetwork thing is just a thin wrapper around rippled websocket API. I aware the subscribe method, but how do you recover to the point where the monitor daemon fails? Repeated transactions are acceptable, while missing several is not. – Proton – 2014-04-07T07:12:55.470

Subscribing all transactions and do filtering is also acceptable. But there is no "ledger_index_min" like account_tx provided, only live transaction streams. – Proton – 2014-04-07T07:18:11.970

@Proton "but how do you recover to the point where the monitor daemon fails". I covered that in the second paragraph. That's where it would make sense to do a loop like the one you gave. You'd keep track of the highest ledger number you've synced with (either via the ledger field of each tx or if you happen to also want/need the ledger close stream you could use that). On restart you'd do the account_tx from that ledger forward, and once caught up with the current ledger you'd switch over to the subscribe stream for efficiency and lower latency rather than needing to poll for changes. – dchapes – 2014-04-07T14:06:21.233

At least that's how I'd approach it, there are probably other ways. – dchapes – 2014-04-07T14:07:18.687

Asked: 2014-04-06T08:01:19.993

Viewed: 527 times

Active: 2014-04-06T13:37:08.983