Synchronous and asynchronous UDF execution

This notebook demonstrates the following features that were introduced in release 0.7.0:

Execute several UDFs in one pass
Obtain intermediate results from each merge step by executing UDFs as an iterator
Execute UDFs asynchronously

Please see example live-plotting.ipynb for the related live plotting feature!

[1]:

import os
import pprint
import asyncio
import copy

import numpy as np

import libertem.api as lt
from libertem.udf.sum import SumUDF
from libertem.udf.sumsigudf import SumSigUDF

[2]:

ctx = lt.Context()

[3]:

data_base_path = os.environ.get("TESTDATA_BASE_PATH", "/home/alex/Data/")

[4]:

ds = ctx.load("auto", path=os.path.join(data_base_path, "20200518 165148/default.hdr"))

[5]:

udfs = [SumUDF(), SumSigUDF()]

Synchronous execution, only result

Note that both UDFs are executed in a single pass!

[6]:

res = ctx.run_udf(dataset=ds, udf=udfs)

The result is a tuple with one entry per UDF:

[7]:

pprint.pprint(res)

({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})

The previous API when passing a single UDF is not changed, i.e. it doesn’t return a tuple but a single UDF result

[8]:

res = ctx.run_udf(dataset=ds, udf=udfs[0])

[9]:

pprint.pprint(res)

{'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>}

Asynchronous execution, only result

By setting sync=False, the result is awaitable:

[10]:

async_res = ctx.run_udf(dataset=ds, udf=udfs, sync=False)
print("Do something else while UDFs are running in the background")
res = await async_res
print("Finished")

Do something else while UDFs are running in the background
Finished

[11]:

pprint.pprint(res)

({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})

Just like in the synchronous case, running a single UDF returns the UDF result directly, not a tuple:

[12]:

async_res = ctx.run_udf(dataset=ds, udf=udfs[0], sync=False)
print("Do something else while UDF is running in the background")
res = await async_res
print("Finished")

Do something else while UDF is running in the background
Finished

[13]:

pprint.pprint(res)

{'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>}

Synchronous execution as an iterator

This returns UDFResults objects with attributes buffers and damage. buffers is a tuple with the results per UDF, and damage is a BufferWrapper with kind='nav' and dtype=bool that indicates which parts of the navigation space have been merged already.

[14]:

# NBVAL_IGNORE_OUTPUT
# (output is ignored in nbval run because the number of nav positions can be different)
for res in ctx.run_udf_iter(dataset=ds, udf=udfs):
    print(np.count_nonzero(res.damage.data), "nav positions processed")
    pprint.pprint(res.buffers)

683 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
1366 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
2049 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
2732 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
3414 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
4097 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
4780 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
5463 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
6145 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
6828 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
7510 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
8193 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
8876 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
9559 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
10242 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
10924 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
11606 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
12289 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
12971 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
13654 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
14337 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
15019 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
15701 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
16384 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})

Asynchronous execution as an iterator

This allows several iterators to proceed asynchronously in parallel. This approach is used in the backend for the web GUI of LiberTEM to run several analyses concurrently. It could also be useful to implement live feedback to instrument control from UDF results if the control solution works asynchronously.

Note that the UDFs are copied here so that different instances are executed in parallel. Executing the same UDF instances concurrently can lead to undefined behavior.

[15]:

# NBVAL_IGNORE_OUTPUT
# (output is ignored in nbval run because the number of nav positions can be different)
async def doit(label, udfs):
    async for res in ctx.run_udf_iter(dataset=ds, udf=udfs, sync=False):
        print(label, np.count_nonzero(res.damage.data), "nav positions processed")
        pprint.pprint((label, res.buffers))
    return res

p1 = doit("one", copy.deepcopy(udfs))
p2 = doit("two", copy.deepcopy(udfs))
print("Do something else while UDFs are running in the background")
await asyncio.gather(p1, p2)

Do something else while UDFs are running in the background
one 683 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 1365 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 2047 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 2730 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 3413 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 4096 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 4779 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 5461 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 6144 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 6827 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 7509 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 8191 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 8874 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 9557 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 10240 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 10923 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 11605 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 12288 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 12970 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 13653 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 14336 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 15019 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 15701 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 16384 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 683 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 1366 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 2048 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 2730 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 3413 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 4096 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 4779 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 5462 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 6145 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 6828 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 7510 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 8192 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 8875 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 9558 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 10240 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 10922 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 11605 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 12288 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 12971 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 13653 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 14336 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 15018 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 15701 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 16384 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))

[15]:

[<libertem.udf.base.UDFResults at 0x7ff489e8af20>,
 <libertem.udf.base.UDFResults at 0x7ff48a0b2230>]

[ ]: