Synchronous and asynchronous UDF execution

This notebook demonstrates the following features that were introduced in release 0.7.0:

  • Execute several UDFs in one pass

  • Obtain intermediate results from each merge step by executing UDFs as an iterator

  • Execute UDFs asynchronously

Please see example live-plotting.ipynb for the related live plotting feature!

[5]:
import os
import pprint
import asyncio
import copy

import numpy as np

import libertem.api as lt
from libertem.udf.sum import SumUDF
from libertem.udf.sumsigudf import SumSigUDF
[6]:
ctx = lt.Context()
[7]:
data_base_path = os.environ.get("TESTDATA_BASE_PATH", "/home/alex/Data/")
[8]:
ds = ctx.load("auto", path=os.path.join(data_base_path, "20200518 165148/default.hdr"))
[9]:
udfs = [SumUDF(), SumSigUDF()]

Synchronous execution, only result

Note that both UDFs are executed in a single pass!

[10]:
res = ctx.run_udf(dataset=ds, udf=udfs)

The result is a tuple with one entry per UDF:

[11]:
pprint.pprint(res)
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})

The previous API when passing a single UDF is not changed, i.e. it doesn’t return a tuple but a single UDF result

[12]:
res = ctx.run_udf(dataset=ds, udf=udfs[0])
[13]:
pprint.pprint(res)
{'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>}

Asynchronous execution, only result

By setting sync=False, the result is awaitable:

[14]:
async_res = ctx.run_udf(dataset=ds, udf=udfs, sync=False)
print("Do something else while UDFs are running in the background")
res = await async_res
print("Finished")
Do something else while UDFs are running in the background
Finished
[15]:
pprint.pprint(res)
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})

Just like in the synchronous case, running a single UDF returns the UDF result directly, not a tuple:

[16]:
async_res = ctx.run_udf(dataset=ds, udf=udfs[0], sync=False)
print("Do something else while UDF is running in the background")
res = await async_res
print("Finished")
Do something else while UDF is running in the background
Finished
[17]:
pprint.pprint(res)
{'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>}

Synchronous execution as an iterator

This returns UDFResults objects with attributes buffers and damage. buffers is a tuple with the results per UDF, and damage is a BufferWrapper with kind='nav' and dtype=bool that indicates which parts of the navigation space have been merged already.

[18]:
# NBVAL_IGNORE_OUTPUT
# (output is ignored in nbval run because the number of nav positions can be different)
for res in ctx.run_udf_iter(dataset=ds, udf=udfs):
    print(np.count_nonzero(res.damage.data), "nav positions processed")
    pprint.pprint(res.buffers)
630 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
1260 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
1890 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
2520 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
3150 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
3780 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
4410 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
5040 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
5670 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
6300 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
6930 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
7560 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
7564 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
8194 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
8824 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
9454 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
10084 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
10714 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
11344 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
11974 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
12604 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
13234 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
13864 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
14494 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
15124 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
15754 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})
16384 nav positions processed
({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
 {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>})

Asynchronous execution as an iterator

This allows several iterators to proceed asynchronously in parallel. This approach is used in the backend for the web GUI of LiberTEM to run several analyses concurrently. It could also be useful to implement live feedback to instrument control from UDF results if the control solution works asynchronously.

Note that the UDFs are copied here so that different instances are executed in parallel. Executing the same UDF instances concurrently can lead to undefined behavior.

[19]:
# NBVAL_IGNORE_OUTPUT
# (output is ignored in nbval run because the number of nav positions can be different)
async def doit(label, udfs):
    async for res in ctx.run_udf_iter(dataset=ds, udf=udfs, sync=False):
        print(label, np.count_nonzero(res.damage.data), "nav positions processed")
        pprint.pprint((label, res.buffers))
    return res

p1 = doit("one", copy.deepcopy(udfs))
p2 = doit("two", copy.deepcopy(udfs))
print("Do something else while UDFs are running in the background")
await asyncio.gather(p1, p2)
Do something else while UDFs are running in the background
two 630 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 630 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 1260 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 1260 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 1890 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 1890 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 2520 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 2520 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 3150 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 3150 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 3780 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 3780 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 4410 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 4410 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 5040 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 5670 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 5040 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 5670 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 6300 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 6300 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 6930 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 6930 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 7560 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 8190 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 8820 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 9450 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 7560 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 10080 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 8190 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 10710 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 8820 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 11340 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 9450 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 10080 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 11970 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 12600 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 10710 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 13230 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 11340 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 11970 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 12600 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 13860 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 14490 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 13230 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 15120 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 13860 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 15750 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 14490 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 15754 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 14494 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
two 16384 nav positions processed
('two',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 15124 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 15754 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
one 16384 nav positions processed
('one',
 ({'intensity': <BufferWrapper kind=sig dtype=float32 extra_shape=()>},
  {'intensity': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}))
[19]:
[<libertem.udf.base.UDFResults at 0x7fda300af820>,
 <libertem.udf.base.UDFResults at 0x7fdab0058be0>]
[ ]:

[ ]:

[ ]: