Synchronous and asynchronous UDF execution

This notebook demonstrates the following features that were introduced in release 0.7.0:

Execute several UDFs in one pass
Obtain intermediate results from each merge step by executing UDFs as an iterator
Execute UDFs asynchronously

Please see example live-plotting.ipynb for the related live plotting feature!

[1]:

import os
import pprint
import asyncio
import copy

import numpy as np

import libertem.api as lt
from libertem.udf.sum import SumUDF
from libertem.udf.sumsigudf import SumSigUDF

[2]:

ctx = lt.Context()

[3]:

data_base_path = os.environ.get("TESTDATA_BASE_PATH", "/home/alex/Data/")

[4]:

ds = ctx.load("auto", path=os.path.join(data_base_path, "20200518 165148/default.hdr"))

[5]:

udfs = [SumUDF(), SumSigUDF()]

Synchronous execution, only result

Note that both UDFs are executed in a single pass!

[6]:

res = ctx.run_udf(dataset=ds, udf=udfs)

The result is a tuple with one entry per UDF:

[7]:

pprint.pprint(res)

({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})

The previous API when passing a single UDF is not changed, i.e. it doesn’t return a tuple but a single UDF result

[8]:

res = ctx.run_udf(dataset=ds, udf=udfs[0])

[9]:

pprint.pprint(res)

{'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>}

Asynchronous execution, only result

By setting sync=False, the result is awaitable:

[10]:

async_res = ctx.run_udf(dataset=ds, udf=udfs, sync=False)
print("Do something else while UDFs are running in the background")
res = await async_res
print("Finished")

Do something else while UDFs are running in the background
Finished

[11]:

pprint.pprint(res)

({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})

Just like in the synchronous case, running a single UDF returns the UDF result directly, not a tuple:

[12]:

async_res = ctx.run_udf(dataset=ds, udf=udfs[0], sync=False)
print("Do something else while UDF is running in the background")
res = await async_res
print("Finished")

Do something else while UDF is running in the background
Finished

[13]:

pprint.pprint(res)

{'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>}

Synchronous execution as an iterator

This returns UDFResults objects with attributes buffers and damage. buffers is a tuple with the results per UDF, and damage is a BufferWrapper with kind='nav' and dtype=bool that indicates which parts of the navigation space have been merged already.

[14]:

# NBVAL_IGNORE_OUTPUT
# (output is ignored in nbval run because the number of nav positions can be different)
for res in ctx.run_udf_iter(dataset=ds, udf=udfs):
    print(np.count_nonzero(res.damage.data), "nav positions processed")
    pprint.pprint(res.buffers)

682 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
1364 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
2047 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
2730 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
3413 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
4096 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
4778 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
5461 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
6144 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
6827 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
7509 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
8192 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
8874 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
9557 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
10240 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
10922 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
11605 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
12288 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
12971 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
13654 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
14337 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
15019 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
15702 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
16384 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})

Asynchronous execution as an iterator

This allows several iterators to proceed asynchronously in parallel. This approach is used in the backend for the web GUI of LiberTEM to run several analyses concurrently. It could also be useful to implement live feedback to instrument control from UDF results if the control solution works asynchronously.

Note that the UDFs are copied here so that different instances are executed in parallel. Executing the same UDF instances concurrently can lead to undefined behavior.

[15]:

# NBVAL_IGNORE_OUTPUT
# (output is ignored in nbval run because the number of nav positions can be different)
async def doit(label, udfs):
    async for res in ctx.run_udf_iter(dataset=ds, udf=udfs, sync=False):
        print(label, np.count_nonzero(res.damage.data), "nav positions processed")
        pprint.pprint((label, res.buffers))
    return res

p1 = doit("one", copy.deepcopy(udfs))
p2 = doit("two", copy.deepcopy(udfs))
print("Do something else while UDFs are running in the background")
await asyncio.gather(p1, p2)

Do something else while UDFs are running in the background
one 682 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 1364 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 2047 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 2729 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 3412 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 4095 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 4778 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 5460 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 6143 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 6826 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 7509 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 8192 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 8875 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 9557 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 10239 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 10922 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 11605 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 12288 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 12971 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 13653 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 14335 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 15018 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 15701 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 683 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 1366 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 2048 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 2731 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 3414 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 16384 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 4097 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 4780 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 5463 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 6145 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 6827 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 7510 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 8193 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 8875 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 9557 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 10240 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 10923 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 11606 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 12288 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 12971 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 13653 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 14336 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 15019 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 15701 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 16384 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))

[15]:

[<libertem.udf.base.UDFResults at 0x7f030858f590>,
 <libertem.udf.base.UDFResults at 0x7f03087a4150>]

[ ]: