Live plotting for LiberTEM UDFs

This example demonstrates the live plotting feature of LiberTEM that is introduced in release 0.7.0. It can update a plot with new results while LiberTEM processing is running.

The data can be downloaded at https://zenodo.org/record/5113449.

This notebook requires additional packages bqplot, bqplot_image_gl and ipywidgets. See also https://ipywidgets.readthedocs.io/en/stable/user_install.html#installing-in-classic-jupyter-notebook in case the plots don’t show.

%matplotlib nbagg

import os
import matplotlib.pyplot as plt
import matplotlib.colors
import ipywidgets
import IPython
import numpy as np

import libertem.api as lt
from libertem.udf import UDF
from libertem.udf.raw import PickUDF
from libertem.viz.mpl import MPLLive2DPlot
from libertem.viz.bqp import BQLive2DPlot

Plot classes

Currently, LiberTEM implements three back-ends for live plotting:

• libertem.viz.mpl.MPLLive2DPlot with a matplotlib back-end. It is the default since matplotlib is the most popular and mature plotting package for Python. It requires about 100-200 ms to display or update a plot, which only allows a few updates per second.

• libertem.viz.bqp.BQLive2DPlot with a bqplot_image_gl back-end. It is much faster than matplotlib and can easily keep up with the full update rate of LiberTEM for a smoother and more responsive display.

• libertem.viz.gms.GMSLive2DPlot for plotting within Digital Micrograph, Gatan Microscopy Suite. This only works using the Python scripting of recent GMS releases and is not demonstrated here.

Additional plotting back-ends can be implemented by deriving from libertem.viz.base.Live2DPlot.

Here we change the default class for live plotting to BQLive2DPlot and then change it back right away by assigning to the plot_class property of the LiberTEM Context object.

# We set the Context to use BQLive2DPlot for live plotting
ctx = lt.Context(plot_class=BQLive2DPlot)

# We change it right back to the default MPLLive2DPlot
ctx.plot_class = MPLLive2DPlot

/home/weber/miniconda3/envs/libertem39/lib/python3.9/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 43253 instead
  warnings.warn(
2023-01-12 16:25:24,233 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-zqr7hwaf', purging
2023-01-12 16:25:24,234 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-5c010dto', purging
2023-01-12 16:25:24,234 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-jnsu9o2k', purging
2023-01-12 16:25:24,234 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-2a_xuttz', purging
2023-01-12 16:25:24,234 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-ur64txas', purging
2023-01-12 16:25:24,235 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-2_u1wsmk', purging
2023-01-12 16:25:24,235 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-l_yvjlql', purging
2023-01-12 16:25:24,235 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-edtxk3wh', purging
2023-01-12 16:25:24,235 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-r7o624n2', purging
2023-01-12 16:25:24,236 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-0g279rwd', purging
2023-01-12 16:25:24,236 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-0rbn83j_', purging
2023-01-12 16:25:24,236 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-n3j0wat2', purging
2023-01-12 16:25:24,236 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-n4e61of8', purging
2023-01-12 16:25:24,237 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-e4pc8cup', purging
2023-01-12 16:25:24,237 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-ghuu0p1n', purging
2023-01-12 16:25:24,237 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-ffjl_lg_', purging
2023-01-12 16:25:24,237 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CUDA", device=0)
2023-01-12 16:25:24,238 - distributed.utils - INFO - Reload module tmp3f8w1616 from .py file
2023-01-12 16:25:24,254 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=7)
2023-01-12 16:25:24,255 - distributed.utils - INFO - Reload module tmphpu43j4z from .py file
2023-01-12 16:25:24,267 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=3)
2023-01-12 16:25:24,267 - distributed.utils - INFO - Reload module tmpl7frtq47 from .py file
2023-01-12 16:25:24,275 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=16)
2023-01-12 16:25:24,275 - distributed.utils - INFO - Reload module tmpkov02hsu from .py file
2023-01-12 16:25:24,286 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=22)
2023-01-12 16:25:24,286 - distributed.utils - INFO - Reload module tmpz9kwmpcs from .py file
2023-01-12 16:25:24,315 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=13)
2023-01-12 16:25:24,315 - distributed.utils - INFO - Reload module tmp71fiolfj from .py file
2023-01-12 16:25:24,316 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=20)
2023-01-12 16:25:24,317 - distributed.utils - INFO - Reload module tmp1gkgontn from .py file
2023-01-12 16:25:24,327 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=18)
2023-01-12 16:25:24,328 - distributed.utils - INFO - Reload module tmpr5x2hgfg from .py file
2023-01-12 16:25:24,350 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=9)
2023-01-12 16:25:24,351 - distributed.utils - INFO - Reload module tmp0m8lxktt from .py file
2023-01-12 16:25:24,394 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=12)
2023-01-12 16:25:24,395 - distributed.utils - INFO - Reload module tmpgkfkifd0 from .py file
2023-01-12 16:25:24,398 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=23)
2023-01-12 16:25:24,399 - distributed.utils - INFO - Reload module tmplhj1wnfb from .py file
2023-01-12 16:25:24,410 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=5)
2023-01-12 16:25:24,410 - distributed.utils - INFO - Reload module tmpmyh60x9e from .py file
2023-01-12 16:25:24,426 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-service-0', service_id='0')
2023-01-12 16:25:24,427 - distributed.utils - INFO - Reload module tmpup6klqgy from .py file
2023-01-12 16:25:24,439 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=8)
2023-01-12 16:25:24,440 - distributed.utils - INFO - Reload module tmpu4mjstu4 from .py file
2023-01-12 16:25:24,454 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=1)
2023-01-12 16:25:24,455 - distributed.utils - INFO - Reload module tmp1iuyri_s from .py file
2023-01-12 16:25:24,477 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=15)
2023-01-12 16:25:24,478 - distributed.utils - INFO - Reload module tmppvlgvbg9 from .py file
2023-01-12 16:25:24,483 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=17)
2023-01-12 16:25:24,483 - distributed.utils - INFO - Reload module tmpv09py0t_ from .py file
2023-01-12 16:25:24,496 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=2)
2023-01-12 16:25:24,496 - distributed.utils - INFO - Reload module tmpszuwwvjv from .py file
2023-01-12 16:25:24,530 - distributed.preloading - INFO - Import preload module: /tmp/tmp3f8w1616.py
2023-01-12 16:25:24,530 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cuda-0', service_id='0')
2023-01-12 16:25:24,530 - distributed.utils - INFO - Reload module tmpaug68oyd from .py file
2023-01-12 16:25:24,531 - distributed.preloading - INFO - Import preload module: /tmp/tmpaug68oyd.py
2023-01-12 16:25:24,531 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,538 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=21)
2023-01-12 16:25:24,539 - distributed.utils - INFO - Reload module tmp6wals7j1 from .py file
2023-01-12 16:25:24,544 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=10)
2023-01-12 16:25:24,544 - distributed.utils - INFO - Reload module tmpp3rrhqm9 from .py file
2023-01-12 16:25:24,568 - distributed.preloading - INFO - Import preload module: /tmp/tmpl7frtq47.py
2023-01-12 16:25:24,568 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-3', service_id='3')
2023-01-12 16:25:24,569 - distributed.utils - INFO - Reload module tmpo_4ac6jp from .py file
2023-01-12 16:25:24,569 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=6)
2023-01-12 16:25:24,569 - distributed.preloading - INFO - Import preload module: /tmp/tmpo_4ac6jp.py
2023-01-12 16:25:24,569 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,569 - distributed.utils - INFO - Reload module tmpwz9cxz8h from .py file
2023-01-12 16:25:24,570 - distributed.preloading - INFO - Import preload module: /tmp/tmpkov02hsu.py
2023-01-12 16:25:24,570 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-16', service_id='16')
2023-01-12 16:25:24,571 - distributed.utils - INFO - Reload module tmp3qdbrznz from .py file
2023-01-12 16:25:24,571 - distributed.preloading - INFO - Import preload module: /tmp/tmp3qdbrznz.py
2023-01-12 16:25:24,571 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,571 - distributed.preloading - INFO - Import preload module: /tmp/tmphpu43j4z.py
2023-01-12 16:25:24,571 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-7', service_id='7')
2023-01-12 16:25:24,572 - distributed.utils - INFO - Reload module tmppil73x3q from .py file
2023-01-12 16:25:24,572 - distributed.preloading - INFO - Import preload module: /tmp/tmppil73x3q.py
2023-01-12 16:25:24,572 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,572 - distributed.preloading - INFO - Import preload module: /tmp/tmpz9kwmpcs.py
2023-01-12 16:25:24,572 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-22', service_id='22')
2023-01-12 16:25:24,573 - distributed.utils - INFO - Reload module tmpo59m2bon from .py file
2023-01-12 16:25:24,573 - distributed.preloading - INFO - Import preload module: /tmp/tmpo59m2bon.py
2023-01-12 16:25:24,573 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,576 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,581 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=11)
2023-01-12 16:25:24,582 - distributed.utils - INFO - Reload module tmpix_ky9sy from .py file
2023-01-12 16:25:24,594 - distributed.preloading - INFO - Import preload module: /tmp/tmp1gkgontn.py
2023-01-12 16:25:24,595 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-20', service_id='20')
2023-01-12 16:25:24,595 - distributed.utils - INFO - Reload module tmpurnf7le2 from .py file
2023-01-12 16:25:24,595 - distributed.preloading - INFO - Import preload module: /tmp/tmpurnf7le2.py
2023-01-12 16:25:24,595 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,602 - distributed.preloading - INFO - Import preload module: /tmp/tmpr5x2hgfg.py
2023-01-12 16:25:24,602 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-18', service_id='18')
2023-01-12 16:25:24,602 - distributed.preloading - INFO - Import preload module: /tmp/tmp71fiolfj.py
2023-01-12 16:25:24,602 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-13', service_id='13')
2023-01-12 16:25:24,603 - distributed.utils - INFO - Reload module tmp3t2e65w4 from .py file
2023-01-12 16:25:24,603 - distributed.utils - INFO - Reload module tmpe39iu6jb from .py file
2023-01-12 16:25:24,603 - distributed.preloading - INFO - Import preload module: /tmp/tmp3t2e65w4.py
2023-01-12 16:25:24,603 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,603 - distributed.preloading - INFO - Import preload module: /tmp/tmpe39iu6jb.py
2023-01-12 16:25:24,603 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,616 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,617 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,617 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,619 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,624 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=4)
2023-01-12 16:25:24,624 - distributed.utils - INFO - Reload module tmp4h6rbkyt from .py file
2023-01-12 16:25:24,626 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=14)
2023-01-12 16:25:24,627 - distributed.utils - INFO - Reload module tmp72q61mef from .py file
2023-01-12 16:25:24,631 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=19)
2023-01-12 16:25:24,631 - distributed.utils - INFO - Reload module tmpai7uu018 from .py file
2023-01-12 16:25:24,639 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=0)
2023-01-12 16:25:24,639 - distributed.utils - INFO - Reload module tmpql54zi1g from .py file
2023-01-12 16:25:24,641 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,649 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,651 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,678 - distributed.preloading - INFO - Import preload module: /tmp/tmpgkfkifd0.py
2023-01-12 16:25:24,679 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-12', service_id='12')
2023-01-12 16:25:24,679 - distributed.utils - INFO - Reload module tmppfm28_kd from .py file
2023-01-12 16:25:24,679 - distributed.preloading - INFO - Import preload module: /tmp/tmppfm28_kd.py
2023-01-12 16:25:24,679 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,698 - distributed.preloading - INFO - Import preload module: /tmp/tmplhj1wnfb.py
2023-01-12 16:25:24,699 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-23', service_id='23')
2023-01-12 16:25:24,699 - distributed.utils - INFO - Reload module tmpsjbppemr from .py file
2023-01-12 16:25:24,700 - distributed.preloading - INFO - Import preload module: /tmp/tmpsjbppemr.py
2023-01-12 16:25:24,700 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,700 - distributed.preloading - INFO - Import preload module: /tmp/tmpup6klqgy.py
2023-01-12 16:25:24,700 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,706 - distributed.preloading - INFO - Import preload module: /tmp/tmpmyh60x9e.py
2023-01-12 16:25:24,706 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-5', service_id='5')
2023-01-12 16:25:24,706 - distributed.utils - INFO - Reload module tmpxxp71o_o from .py file
2023-01-12 16:25:24,707 - distributed.preloading - INFO - Import preload module: /tmp/tmpxxp71o_o.py
2023-01-12 16:25:24,707 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,710 - distributed.preloading - INFO - Import preload module: /tmp/tmp0m8lxktt.py
2023-01-12 16:25:24,710 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-9', service_id='9')
2023-01-12 16:25:24,710 - distributed.utils - INFO - Reload module tmpe0e2jzwn from .py file
2023-01-12 16:25:24,711 - distributed.preloading - INFO - Import preload module: /tmp/tmpe0e2jzwn.py
2023-01-12 16:25:24,711 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,725 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,731 - distributed.preloading - INFO - Import preload module: /tmp/tmpu4mjstu4.py
2023-01-12 16:25:24,731 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-8', service_id='8')
2023-01-12 16:25:24,732 - distributed.utils - INFO - Reload module tmpv39xr13i from .py file
2023-01-12 16:25:24,732 - distributed.preloading - INFO - Import preload module: /tmp/tmpv39xr13i.py
2023-01-12 16:25:24,732 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,744 - distributed.preloading - INFO - Import preload module: /tmp/tmppvlgvbg9.py
2023-01-12 16:25:24,744 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-15', service_id='15')
2023-01-12 16:25:24,744 - distributed.utils - INFO - Reload module tmpvlf0hedx from .py file
2023-01-12 16:25:24,744 - distributed.preloading - INFO - Import preload module: /tmp/tmpvlf0hedx.py
2023-01-12 16:25:24,744 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,746 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,746 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,752 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,756 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,765 - distributed.preloading - INFO - Import preload module: /tmp/tmpv09py0t_.py
2023-01-12 16:25:24,766 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-17', service_id='17')
2023-01-12 16:25:24,766 - distributed.utils - INFO - Reload module tmpb5_h042v from .py file
2023-01-12 16:25:24,767 - distributed.preloading - INFO - Import preload module: /tmp/tmpb5_h042v.py
2023-01-12 16:25:24,767 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,776 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,790 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,813 - distributed.preloading - INFO - Import preload module: /tmp/tmp1iuyri_s.py
2023-01-12 16:25:24,813 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-1', service_id='1')
2023-01-12 16:25:24,813 - distributed.utils - INFO - Reload module tmp717y0j27 from .py file
2023-01-12 16:25:24,814 - distributed.preloading - INFO - Import preload module: /tmp/tmp717y0j27.py
2023-01-12 16:25:24,814 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,814 - distributed.preloading - INFO - Import preload module: /tmp/tmpp3rrhqm9.py
2023-01-12 16:25:24,814 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-10', service_id='10')
2023-01-12 16:25:24,814 - distributed.utils - INFO - Reload module tmpl5r1nafi from .py file
2023-01-12 16:25:24,815 - distributed.preloading - INFO - Import preload module: /tmp/tmpl5r1nafi.py
2023-01-12 16:25:24,815 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,821 - distributed.preloading - INFO - Import preload module: /tmp/tmp6wals7j1.py
2023-01-12 16:25:24,822 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-21', service_id='21')
2023-01-12 16:25:24,822 - distributed.utils - INFO - Reload module tmp9hl4orip from .py file
2023-01-12 16:25:24,822 - distributed.preloading - INFO - Import preload module: /tmp/tmp9hl4orip.py
2023-01-12 16:25:24,822 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,829 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,858 - distributed.preloading - INFO - Import preload module: /tmp/tmpszuwwvjv.py
2023-01-12 16:25:24,858 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-2', service_id='2')
2023-01-12 16:25:24,858 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,858 - distributed.utils - INFO - Reload module tmp8_mbmg5v from .py file
2023-01-12 16:25:24,859 - distributed.preloading - INFO - Import preload module: /tmp/tmp8_mbmg5v.py
2023-01-12 16:25:24,859 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,859 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,866 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,894 - distributed.preloading - INFO - Import preload module: /tmp/tmp4h6rbkyt.py
2023-01-12 16:25:24,894 - distributed.preloading - INFO - Import preload module: /tmp/tmp72q61mef.py
2023-01-12 16:25:24,894 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-4', service_id='4')
2023-01-12 16:25:24,894 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-14', service_id='14')
2023-01-12 16:25:24,894 - distributed.utils - INFO - Reload module tmpp1axw22y from .py file
2023-01-12 16:25:24,894 - distributed.utils - INFO - Reload module tmpa7y_ti4x from .py file
2023-01-12 16:25:24,895 - distributed.preloading - INFO - Import preload module: /tmp/tmpp1axw22y.py
2023-01-12 16:25:24,895 - distributed.preloading - INFO - Import preload module: /tmp/tmpa7y_ti4x.py
2023-01-12 16:25:24,895 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,895 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,901 - distributed.preloading - INFO - Import preload module: /tmp/tmpai7uu018.py
2023-01-12 16:25:24,901 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-19', service_id='19')
2023-01-12 16:25:24,901 - distributed.utils - INFO - Reload module tmpoynbj8h2 from .py file
2023-01-12 16:25:24,902 - distributed.preloading - INFO - Import preload module: /tmp/tmpoynbj8h2.py
2023-01-12 16:25:24,902 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,903 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,906 - distributed.preloading - INFO - Import preload module: /tmp/tmpwz9cxz8h.py
2023-01-12 16:25:24,906 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-6', service_id='6')
2023-01-12 16:25:24,907 - distributed.utils - INFO - Reload module tmpdgqjvtv2 from .py file
2023-01-12 16:25:24,907 - distributed.preloading - INFO - Import preload module: /tmp/tmpdgqjvtv2.py
2023-01-12 16:25:24,907 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,909 - distributed.preloading - INFO - Import preload module: /tmp/tmpql54zi1g.py
2023-01-12 16:25:24,909 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-0', service_id='0')
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Import preload module: /tmp/tmpix_ky9sy.py
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-11', service_id='11')
2023-01-12 16:25:24,910 - distributed.utils - INFO - Reload module tmpp3uwrt5g from .py file
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Import preload module: /tmp/tmpp3uwrt5g.py
2023-01-12 16:25:24,910 - distributed.utils - INFO - Reload module tmpgz7yxmln from .py file
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Import preload module: /tmp/tmpgz7yxmln.py
2023-01-12 16:25:24,911 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,938 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,939 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,945 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,951 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,954 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,955 - distributed.preloading - INFO - Import preload module: libertem.preload

data_base_path = os.environ.get("TESTDATA_BASE_PATH", "/home/alex/Data/")
ds = ctx.load("auto", path=os.path.join(data_base_path, "20200518 165148/default.hdr"))

# This internal method is used here to artificially create more partitions than necessary
# for better demonstration of live plotting on machines with few cores.
# This reduces performance, should NOT be used in production and can change without
# notice in future releases.
ds.set_num_cores(32)

2023-01-12 16:25:25,468 - distributed.worker - WARNING - Compute Failed
Key:       _do_detect-4a5d47dc-598f-491d-b7f0-04e219a5ed91
Function:  _do_detect
args:      ()
kwargs:    {}
Exception: 'DataSetException("OSError(\'Unable to open file (file signature not found)\')")'

Demo UDFs

We implement three simple user-defined functions (UDFs) to demonstrate the various features and options for live plotting. They calculate a map of the navigation space with sum and maximum, a map of the signal space with sum and maximum, and a global maximum.

See https://libertem.github.io/LiberTEM/udf.html for more information on UDFs.

class DemoNavUDF(UDF):
    def get_result_buffers(self):
        return {
            'nav_sum': self.buffer(kind='nav', dtype='float32'),
            'nav_max': self.buffer(kind='nav', dtype='float32'),
        }

    def process_frame(self, frame):
        self.results.nav_sum[:] = np.sum(frame)
        self.results.nav_max[:] = np.max(frame)


class DemoSigUDF(UDF):
    def get_result_buffers(self):
        return {
            'sig_sum': self.buffer(kind='sig', dtype='float32'),
            'sig_max': self.buffer(kind='sig', dtype='float32'),
        }

    def process_frame(self, frame):
        self.results.sig_sum += frame
        np.maximum(self.results.sig_max, frame, out=self.results.sig_max)

    def merge(self, dest, src):
        dest.sig_sum += src.sig_sum
        np.maximum(dest.sig_max, src.sig_max, out=dest.sig_max)


class DemoSingleUDF(UDF):
    def get_result_buffers(self):
        return {
            'maximum': self.buffer(kind='single', dtype='float32'),
        }

    def process_frame(self, frame):
        self.results.maximum[:] = np.maximum(self.results.maximum, np.max(frame))

    def merge(self, dest, src):
        dest.maximum[:] = np.maximum(dest.maximum, src.maximum)

The UDFs are instantiated and stored in a list for subsequent use.

udfs = [DemoNavUDF(), DemoSigUDF(), DemoSingleUDF()]

Plot all plottable channels

Note how Context.run_udf() can execute several UDFs in one pass since release 0.7.0 by passing a list or tuple of UDFs.

By setting plots=True you can plot all channels that have a 2D shape after applying np.squeeze. The DemoSingleUDF is not plotted since its only channel maximum is a single value. This triggers a warning.

res = ctx.run_udf(dataset=ds, udf=udfs, plots=True)

/home/weber/LiberTEM/LiberTEM/src/libertem/api.py:1212: UserWarning: No plottable channels found for UDF #2: DemoSingleUDF, not plotting.
  warnings.warn(

Select specific channels

We can specify which channels should be plotted by passing a nested list with channel names for each of the UDFs as the plots parameter.

res = ctx.run_udf(dataset=ds, udf=udfs, plots=[['nav_sum'], ['sig_sum', 'sig_max']])