LiberTEM UDFs

[1]:

%matplotlib inline

[2]:

import os
import matplotlib.pyplot as plt
import libertem.api as lt
import numpy as np

[3]:

ctx = lt.Context()

/home/weber/miniconda3/envs/libertem39/lib/python3.9/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 36095 instead
  warnings.warn(
2023-01-12 17:14:25,996 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-8f213pwu', purging
2023-01-12 17:14:25,997 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-n3ws_9ic', purging
2023-01-12 17:14:25,997 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-1xqd091z', purging
2023-01-12 17:14:25,998 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-wi3ns33a', purging
2023-01-12 17:14:25,998 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-03g78l_e', purging
2023-01-12 17:14:25,998 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-48u41lm4', purging
2023-01-12 17:14:25,998 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-a7fdkrne', purging
2023-01-12 17:14:25,998 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-awkh6gob', purging
2023-01-12 17:14:25,999 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-xfijlddh', purging
2023-01-12 17:14:25,999 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-_6p8_h1o', purging
2023-01-12 17:14:25,999 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-r2f3v0j4', purging
2023-01-12 17:14:26,000 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-g0wt60ni', purging
2023-01-12 17:14:26,000 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-dwz_4fyo', purging
2023-01-12 17:14:26,000 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-9876_ixm', purging
2023-01-12 17:14:26,000 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-gyul3rfp', purging
2023-01-12 17:14:26,001 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-fcq1wgw8', purging
2023-01-12 17:14:26,001 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-s39k6xg6', purging
2023-01-12 17:14:26,001 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-3dgbcb5n', purging
2023-01-12 17:14:26,002 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=0)
2023-01-12 17:14:26,002 - distributed.utils - INFO - Reload module tmprcqajvtw from .py file
2023-01-12 17:14:26,012 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=12)
2023-01-12 17:14:26,013 - distributed.utils - INFO - Reload module tmp0w0h9c0g from .py file
2023-01-12 17:14:26,024 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=7)
2023-01-12 17:14:26,024 - distributed.utils - INFO - Reload module tmpo1k1d_mb from .py file
2023-01-12 17:14:26,035 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=21)
2023-01-12 17:14:26,035 - distributed.utils - INFO - Reload module tmpe9_gyt72 from .py file
2023-01-12 17:14:26,050 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=15)
2023-01-12 17:14:26,050 - distributed.utils - INFO - Reload module tmp5cvsbgqm from .py file
2023-01-12 17:14:26,063 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=8)
2023-01-12 17:14:26,063 - distributed.utils - INFO - Reload module tmp66wygbjn from .py file
2023-01-12 17:14:26,070 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=20)
2023-01-12 17:14:26,071 - distributed.utils - INFO - Reload module tmpd8erz6s_ from .py file
2023-01-12 17:14:26,126 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=5)
2023-01-12 17:14:26,127 - distributed.utils - INFO - Reload module tmpq3k7q2ok from .py file
2023-01-12 17:14:26,136 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=19)
2023-01-12 17:14:26,136 - distributed.utils - INFO - Reload module tmpv6prmkwl from .py file
2023-01-12 17:14:26,144 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=1)
2023-01-12 17:14:26,145 - distributed.utils - INFO - Reload module tmpj04fhm9v from .py file
2023-01-12 17:14:26,158 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CUDA", device=0)
2023-01-12 17:14:26,158 - distributed.utils - INFO - Reload module tmplp1hhi6y from .py file
2023-01-12 17:14:26,167 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=3)
2023-01-12 17:14:26,167 - distributed.utils - INFO - Reload module tmpjaub_lqj from .py file
2023-01-12 17:14:26,182 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=22)
2023-01-12 17:14:26,182 - distributed.utils - INFO - Reload module tmpdhg2bn4z from .py file
2023-01-12 17:14:26,197 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=11)
2023-01-12 17:14:26,198 - distributed.utils - INFO - Reload module tmpmvvk4ah4 from .py file
2023-01-12 17:14:26,209 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=17)
2023-01-12 17:14:26,209 - distributed.utils - INFO - Reload module tmp4ojhow71 from .py file
2023-01-12 17:14:26,229 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=4)
2023-01-12 17:14:26,230 - distributed.utils - INFO - Reload module tmpyxjg20xm from .py file
2023-01-12 17:14:26,236 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-service-0', service_id='0')
2023-01-12 17:14:26,237 - distributed.utils - INFO - Reload module tmp7f9zuyi5 from .py file
2023-01-12 17:14:26,254 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=18)
2023-01-12 17:14:26,255 - distributed.utils - INFO - Reload module tmp3ers66bd from .py file
2023-01-12 17:14:26,263 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=10)
2023-01-12 17:14:26,264 - distributed.utils - INFO - Reload module tmpkrzer_o1 from .py file
2023-01-12 17:14:26,277 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=13)
2023-01-12 17:14:26,278 - distributed.utils - INFO - Reload module tmpeitammja from .py file
2023-01-12 17:14:26,290 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=9)
2023-01-12 17:14:26,291 - distributed.utils - INFO - Reload module tmpk7bn699x from .py file
2023-01-12 17:14:26,306 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=16)
2023-01-12 17:14:26,307 - distributed.utils - INFO - Reload module tmpqpgvkr1c from .py file
2023-01-12 17:14:26,354 - distributed.preloading - INFO - Import preload module: /tmp/tmp0w0h9c0g.py
2023-01-12 17:14:26,354 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-12', service_id='12')
2023-01-12 17:14:26,354 - distributed.utils - INFO - Reload module tmpm8pts2sm from .py file
2023-01-12 17:14:26,355 - distributed.preloading - INFO - Import preload module: /tmp/tmpm8pts2sm.py
2023-01-12 17:14:26,355 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,362 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=14)
2023-01-12 17:14:26,363 - distributed.utils - INFO - Reload module tmp3a8iwk7j from .py file
2023-01-12 17:14:26,367 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=2)
2023-01-12 17:14:26,367 - distributed.utils - INFO - Reload module tmprgr7wy9x from .py file
2023-01-12 17:14:26,371 - distributed.preloading - INFO - Import preload module: /tmp/tmpe9_gyt72.py
2023-01-12 17:14:26,372 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-21', service_id='21')
2023-01-12 17:14:26,372 - distributed.utils - INFO - Reload module tmpqwoh3xyq from .py file
2023-01-12 17:14:26,372 - distributed.preloading - INFO - Import preload module: /tmp/tmpqwoh3xyq.py
2023-01-12 17:14:26,373 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,385 - distributed.preloading - INFO - Import preload module: /tmp/tmpd8erz6s_.py
2023-01-12 17:14:26,386 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-20', service_id='20')
2023-01-12 17:14:26,386 - distributed.utils - INFO - Reload module tmpb963prju from .py file
2023-01-12 17:14:26,386 - distributed.preloading - INFO - Import preload module: /tmp/tmpb963prju.py
2023-01-12 17:14:26,386 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,387 - distributed.preloading - INFO - Import preload module: /tmp/tmp5cvsbgqm.py
2023-01-12 17:14:26,388 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-15', service_id='15')
2023-01-12 17:14:26,388 - distributed.utils - INFO - Reload module tmp61lej30i from .py file
2023-01-12 17:14:26,388 - distributed.preloading - INFO - Import preload module: /tmp/tmp61lej30i.py
2023-01-12 17:14:26,389 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,397 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=23)
2023-01-12 17:14:26,398 - distributed.utils - INFO - Reload module tmp1vmjsx6k from .py file
2023-01-12 17:14:26,404 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,409 - distributed.preloading - INFO - Import preload module: /tmp/tmp66wygbjn.py
2023-01-12 17:14:26,409 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-8', service_id='8')
2023-01-12 17:14:26,410 - distributed.utils - INFO - Reload module tmpfu2i0ci2 from .py file
2023-01-12 17:14:26,410 - distributed.preloading - INFO - Import preload module: /tmp/tmpfu2i0ci2.py
2023-01-12 17:14:26,410 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,411 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=6)
2023-01-12 17:14:26,412 - distributed.utils - INFO - Reload module tmp0n4uvr59 from .py file
2023-01-12 17:14:26,421 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,436 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,438 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,449 - distributed.preloading - INFO - Import preload module: /tmp/tmpv6prmkwl.py
2023-01-12 17:14:26,450 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-19', service_id='19')
2023-01-12 17:14:26,450 - distributed.utils - INFO - Reload module tmphg7bn6vs from .py file
2023-01-12 17:14:26,450 - distributed.preloading - INFO - Import preload module: /tmp/tmphg7bn6vs.py
2023-01-12 17:14:26,450 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,456 - distributed.preloading - INFO - Import preload module: /tmp/tmpo1k1d_mb.py
2023-01-12 17:14:26,456 - distributed.preloading - INFO - Import preload module: /tmp/tmprcqajvtw.py
2023-01-12 17:14:26,457 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-0', service_id='0')
2023-01-12 17:14:26,457 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-7', service_id='7')
2023-01-12 17:14:26,457 - distributed.utils - INFO - Reload module tmpd0mpyda_ from .py file
2023-01-12 17:14:26,457 - distributed.utils - INFO - Reload module tmp5ruukbik from .py file
2023-01-12 17:14:26,457 - distributed.preloading - INFO - Import preload module: /tmp/tmpd0mpyda_.py
2023-01-12 17:14:26,457 - distributed.preloading - INFO - Import preload module: /tmp/tmp5ruukbik.py
2023-01-12 17:14:26,457 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,457 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,457 - distributed.preloading - INFO - Import preload module: /tmp/tmpq3k7q2ok.py
2023-01-12 17:14:26,458 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-5', service_id='5')
2023-01-12 17:14:26,458 - distributed.utils - INFO - Reload module tmp3s8m5qcq from .py file
2023-01-12 17:14:26,458 - distributed.preloading - INFO - Import preload module: /tmp/tmp3s8m5qcq.py
2023-01-12 17:14:26,459 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,471 - distributed.preloading - INFO - Import preload module: /tmp/tmpj04fhm9v.py
2023-01-12 17:14:26,471 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-1', service_id='1')
2023-01-12 17:14:26,471 - distributed.utils - INFO - Reload module tmpdwr3jqm8 from .py file
2023-01-12 17:14:26,471 - distributed.preloading - INFO - Import preload module: /tmp/tmpdwr3jqm8.py
2023-01-12 17:14:26,472 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,479 - distributed.preloading - INFO - Import preload module: /tmp/tmpjaub_lqj.py
2023-01-12 17:14:26,479 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-3', service_id='3')
2023-01-12 17:14:26,480 - distributed.utils - INFO - Reload module tmp1v7trw7e from .py file
2023-01-12 17:14:26,480 - distributed.preloading - INFO - Import preload module: /tmp/tmp1v7trw7e.py
2023-01-12 17:14:26,480 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,481 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,497 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,505 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,505 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,505 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,509 - distributed.preloading - INFO - Import preload module: /tmp/tmp4ojhow71.py
2023-01-12 17:14:26,509 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-17', service_id='17')
2023-01-12 17:14:26,509 - distributed.utils - INFO - Reload module tmpbdpw_rm7 from .py file
2023-01-12 17:14:26,510 - distributed.preloading - INFO - Import preload module: /tmp/tmpbdpw_rm7.py
2023-01-12 17:14:26,510 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,510 - distributed.preloading - INFO - Import preload module: /tmp/tmpmvvk4ah4.py
2023-01-12 17:14:26,510 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-11', service_id='11')
2023-01-12 17:14:26,511 - distributed.utils - INFO - Reload module tmp8cmzlxi8 from .py file
2023-01-12 17:14:26,511 - distributed.preloading - INFO - Import preload module: /tmp/tmp8cmzlxi8.py
2023-01-12 17:14:26,511 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,516 - distributed.preloading - INFO - Import preload module: /tmp/tmp7f9zuyi5.py
2023-01-12 17:14:26,517 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,518 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,527 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,561 - distributed.preloading - INFO - Import preload module: /tmp/tmpyxjg20xm.py
2023-01-12 17:14:26,562 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-4', service_id='4')
2023-01-12 17:14:26,562 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,562 - distributed.utils - INFO - Reload module tmpuzx184vp from .py file
2023-01-12 17:14:26,562 - distributed.preloading - INFO - Import preload module: /tmp/tmpuzx184vp.py
2023-01-12 17:14:26,562 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,566 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,570 - distributed.preloading - INFO - Import preload module: /tmp/tmp3ers66bd.py
2023-01-12 17:14:26,570 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-18', service_id='18')
2023-01-12 17:14:26,571 - distributed.utils - INFO - Reload module tmp2c59zmp_ from .py file
2023-01-12 17:14:26,571 - distributed.preloading - INFO - Import preload module: /tmp/tmp2c59zmp_.py
2023-01-12 17:14:26,571 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,573 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,582 - distributed.preloading - INFO - Import preload module: /tmp/tmplp1hhi6y.py
2023-01-12 17:14:26,582 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cuda-0', service_id='0')
2023-01-12 17:14:26,582 - distributed.utils - INFO - Reload module tmp3o85bw1y from .py file
2023-01-12 17:14:26,583 - distributed.preloading - INFO - Import preload module: /tmp/tmp3o85bw1y.py
2023-01-12 17:14:26,583 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,608 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,609 - distributed.preloading - INFO - Import preload module: /tmp/tmpdhg2bn4z.py
2023-01-12 17:14:26,610 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-22', service_id='22')
2023-01-12 17:14:26,610 - distributed.utils - INFO - Reload module tmpwsn459cm from .py file
2023-01-12 17:14:26,610 - distributed.preloading - INFO - Import preload module: /tmp/tmpwsn459cm.py
2023-01-12 17:14:26,610 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,618 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,621 - distributed.preloading - INFO - Import preload module: /tmp/tmpkrzer_o1.py
2023-01-12 17:14:26,622 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-10', service_id='10')
2023-01-12 17:14:26,622 - distributed.utils - INFO - Reload module tmpmlywld0o from .py file
2023-01-12 17:14:26,622 - distributed.preloading - INFO - Import preload module: /tmp/tmpmlywld0o.py
2023-01-12 17:14:26,622 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,624 - distributed.preloading - INFO - Import preload module: /tmp/tmpqpgvkr1c.py
2023-01-12 17:14:26,624 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-16', service_id='16')
2023-01-12 17:14:26,625 - distributed.utils - INFO - Reload module tmpacg0gsvq from .py file
2023-01-12 17:14:26,625 - distributed.preloading - INFO - Import preload module: /tmp/tmpacg0gsvq.py
2023-01-12 17:14:26,625 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,629 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,657 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,659 - distributed.preloading - INFO - Import preload module: /tmp/tmpk7bn699x.py
2023-01-12 17:14:26,660 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-9', service_id='9')
2023-01-12 17:14:26,660 - distributed.utils - INFO - Reload module tmp07xp19wc from .py file
2023-01-12 17:14:26,661 - distributed.preloading - INFO - Import preload module: /tmp/tmp07xp19wc.py
2023-01-12 17:14:26,661 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,672 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,673 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,679 - distributed.preloading - INFO - Import preload module: /tmp/tmpeitammja.py
2023-01-12 17:14:26,679 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-13', service_id='13')
2023-01-12 17:14:26,679 - distributed.utils - INFO - Reload module tmpmuk01c0i from .py file
2023-01-12 17:14:26,680 - distributed.preloading - INFO - Import preload module: /tmp/tmpmuk01c0i.py
2023-01-12 17:14:26,680 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,694 - distributed.preloading - INFO - Import preload module: /tmp/tmprgr7wy9x.py
2023-01-12 17:14:26,694 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-2', service_id='2')
2023-01-12 17:14:26,695 - distributed.utils - INFO - Reload module tmp0r8w2hih from .py file
2023-01-12 17:14:26,695 - distributed.preloading - INFO - Import preload module: /tmp/tmp0r8w2hih.py
2023-01-12 17:14:26,695 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,700 - distributed.preloading - INFO - Import preload module: /tmp/tmp3a8iwk7j.py
2023-01-12 17:14:26,700 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-14', service_id='14')
2023-01-12 17:14:26,700 - distributed.utils - INFO - Reload module tmpl_pepip3 from .py file
2023-01-12 17:14:26,701 - distributed.preloading - INFO - Import preload module: /tmp/tmpl_pepip3.py
2023-01-12 17:14:26,701 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,706 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,716 - distributed.preloading - INFO - Import preload module: /tmp/tmp0n4uvr59.py
2023-01-12 17:14:26,716 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-6', service_id='6')
2023-01-12 17:14:26,716 - distributed.utils - INFO - Reload module tmpcd9324xa from .py file
2023-01-12 17:14:26,717 - distributed.preloading - INFO - Import preload module: /tmp/tmpcd9324xa.py
2023-01-12 17:14:26,717 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,721 - distributed.preloading - INFO - Import preload module: /tmp/tmp1vmjsx6k.py
2023-01-12 17:14:26,721 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-23', service_id='23')
2023-01-12 17:14:26,721 - distributed.utils - INFO - Reload module tmp4ssjomqn from .py file
2023-01-12 17:14:26,722 - distributed.preloading - INFO - Import preload module: /tmp/tmp4ssjomqn.py
2023-01-12 17:14:26,722 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 17:14:26,731 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,742 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,747 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,763 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 17:14:26,766 - distributed.preloading - INFO - Import preload module: libertem.preload

Specifying the dataset

Most formats can be loaded using the "auto" type, but some may need additional parameters.

See the loading data section of the LiberTEM docs for details.

[4]:

data_base_path = os.environ.get("TESTDATA_BASE_PATH", "/home/alex/Data/")

[5]:

ds = ctx.load("auto", path=os.path.join(data_base_path, "01_ms1_3p3gK.hdr"))

2023-01-12 17:14:27,239 - distributed.worker - WARNING - Compute Failed
Key:       _do_detect-576a164e-3c4f-4994-917a-c7de21bc56d5
Function:  _do_detect
args:      ()
kwargs:    {}
Exception: 'DataSetException("OSError(\'Unable to open file (file signature not found)\')")'

After loading, some information is available in the diagnostics attribute:

[6]:

ds.diagnostics

[6]:

[{'name': 'Bits per pixel', 'value': '12'},
 {'name': 'Data kind', 'value': 'u'},
 {'name': 'Layout', 'value': '(1, 1)'},
 {'name': 'Partition shape', 'value': '(2075, 256, 256)'},
 {'name': 'Number of partitions', 'value': '32'},
 {'name': 'Number of frames skipped at the beginning', 'value': 0},
 {'name': 'Number of frames ignored at the end', 'value': 0},
 {'name': 'Number of blank frames inserted at the beginning', 'value': 0},
 {'name': 'Number of blank frames inserted at the end', 'value': 0}]

Standard analyses: virtual detector

A standard analysis to run on 4D STEM data is to apply a virtual detector. Here, we define a ring detector, with radii in pixels:

[7]:

ring = ctx.create_ring_analysis(dataset=ds, ri=60, ro=70)

The analysis can be run with the Context.run method:

[8]:

ring_res = ctx.run(ring, progress=True)
ring_res

[8]:

[<AnalysisResult: intensity>, <AnalysisResult: intensity_log>]

As the analysis mirrors what the web GUI does, we have to access the data using the raw_data attribute, as we would get a viusalized result otherwise. Here we do the visualization ourselves using matplotlib:

[9]:

plt.figure()
plt.imshow(ring_res.intensity.raw_data)

[9]:

<matplotlib.image.AxesImage at 0x7fb29825a4f0>
../_images/udf_introduction_15_1.png

Simple UDF definition

User-defined funtions provide a way for you to implement your own data processing functionality. As a very simple example, we define a function that just sums up the pixels of each frame:

[10]:

def sum_of_pixels(frame):
    return np.sum(frame)

The easiest way to run this on the data is to use the Context.map function:

[11]:

res_pixelsum_1 = ctx.map(dataset=ds, f=sum_of_pixels, progress=True)
res_pixelsum_1

[11]:

<BufferWrapper kind=nav dtype=float32 extra_shape=()>

The result is of type BufferWrapper, but can be used by any function that expects a numpy array, for example for plotting it:

[12]:

plt.figure()
plt.imshow(res_pixelsum_1)

[12]:

<matplotlib.image.AxesImage at 0x7fb2909a1130>
../_images/udf_introduction_21_1.png

The Context.map function is a shortcut for implementing very easy mapping over data, in a frame-by-frame fashion. The longer way of writing this would be as follows:

[13]:

from libertem.udf import UDF


class SumOfPixels(UDF):
    def get_result_buffers(self):
        return {
            'sum_of_pixels': self.buffer(kind='nav', dtype='float32')
        }

    def process_frame(self, frame):
        self.results.sum_of_pixels[:] = np.sum(frame)

This can now be run using the Context.run_udf method:

[14]:

res_pixelsum_2 = ctx.run_udf(dataset=ds, udf=SumOfPixels(), progress=True)
res_pixelsum_2

[14]:

{'sum_of_pixels': <BufferWrapper kind=nav dtype=float32 extra_shape=()>}

The result is now a dict, which maps buffer names, as defined in get_result_buffers, to the BufferWrapper result, so we can use the following to plot the results:

[15]:

plt.figure()
plt.imshow(res_pixelsum_2['sum_of_pixels'])

[15]:

<matplotlib.image.AxesImage at 0x7fb290867a00>
../_images/udf_introduction_27_1.png

extra_shape: more than one result per scan position

[16]:

class StatsUDF(UDF):
    def get_result_buffers(self):
        return {
            'all_stats': self.buffer(kind='nav', dtype='float32', extra_shape=(4,)),
        }

    def process_frame(self, frame):
        self.results.all_stats[:] = (np.mean(frame), np.min(frame), np.max(frame), np.std(frame))

[17]:

res_stats = ctx.run_udf(dataset=ds, udf=StatsUDF(), progress=True)

Result now has an extra dimension, as specified by extra_shape above:

[18]:

res_stats['all_stats'].data.shape

[18]:

(186, 357, 4)

Let’s plot the stddev of each frame:

[19]:

plt.figure()
plt.imshow(res_stats['all_stats'].data[..., 3])

[19]:

<matplotlib.image.AxesImage at 0x7fb2904c1370>
../_images/udf_introduction_34_1.png

kind=“sig” buffers, merge functions

  • Previously: one result for each scan position

  • Now: result buffer shaped like the diffraction patterns

  • We need a merge function to merge the result of one partition into the final result

  • Different buffer kinds can be combined in a single UDF, so you can combine different operations in a single pass over the data

[20]:

class MaxFrameUDF(UDF):
    def get_result_buffers(self):
        return {
            'maxframe': self.buffer(kind='sig', dtype='float32')
        }

    def process_frame(self, frame):
        # element-wise maximum:
        self.results.maxframe[:] = np.maximum(self.results.maxframe, frame)

    def merge(self, dest, src):
        # src: the maximum observed in the current partition
        # dest: the maximum observed in all partitions that were already merged together
        dest.maxframe[:] = np.maximum(dest.maxframe, src.maxframe)

[21]:

res_max = ctx.run_udf(dataset=ds, udf=MaxFrameUDF(), progress=True)

[22]:

plt.figure()
plt.imshow(np.log1p(res_max['maxframe']))

[22]:

<matplotlib.image.AxesImage at 0x7fb2904b9bb0>
../_images/udf_introduction_38_1.png

Region of interest

  • work on a subset of the navigation axes

  • can be used with all UDFs

  • useful for working selectively on data, or just reducing the I/O and computational load when implementing a new UDF

  • defined as a binary mask

Let’s create a mask based on the previously calculated pixel-sum:

[23]:

from skimage.morphology import opening, closing

[24]:

np.min(res_pixelsum_1),  np.max(res_pixelsum_1), np.mean(res_pixelsum_1)

[24]:

(7762.0, 12367.0, 10277.385)

[25]:

mask = res_pixelsum_1.data < np.mean(res_pixelsum_1)
# mask = opening(mask)
mask = closing(opening(mask))

plt.figure()
plt.imshow(mask.astype("float32"))
plt.colorbar()

[25]:

<matplotlib.colorbar.Colorbar at 0x7fb28dd2deb0>
../_images/udf_introduction_43_1.png 
[26]:

res_roi = ctx.run_udf(dataset=ds, udf=StatsUDF(), roi=mask, progress=True)

[27]:

plt.figure()
plt.imshow(np.log1p(res_roi['all_stats'].data[..., 2]))

[27]:

<matplotlib.image.AxesImage at 0x7fb28d9cbee0>
../_images/udf_introduction_45_1.png

Results with ROI applied

One has to take some care when handling results where a roi was applied; just using the result as a numpy array or accessing the data attribute will give you a result that keeps the whole dataset shape, where the deselected parts are filled with NaN values:

[28]:

res_roi['all_stats'].data.shape

[28]:

(186, 357, 4)

There is a second attribute, raw_data, which will give you a flattenned array of just the results, like numpy would give you for fancy indexing:

[29]:

res_roi['all_stats'].raw_data.shape

[29]:

(45358, 4)

Parameters

Keyword arguments passed to the UDF are made available in self.params on the UDF. A good convention to document your parameters is to put a docstring into your __init__ method and just pass on the values to super().__init__. You can also validate the arguments here.

[30]:

class PixelPicker(UDF):
    def __init__(self, coords, *args, **kwargs):
        """
        Parameters
        ----------
        coords : Tuple[int]
            The coordinates to look at in each frame
        """
        if len(coords) != 2:
            raise ValueError("invalid coordinates")
        super().__init__(*args, coords=coords, **kwargs)

    def get_result_buffers(self):
        return {
            'value_of_pixel': self.buffer(kind='nav', dtype=np.float32)
        }

    def process_frame(self, frame):
        self.results.value_of_pixel[:] = frame[self.params.coords]

[31]:

res_pixel_picker = ctx.run_udf(dataset=ds, udf=PixelPicker(coords=(128, 128)), progress=True)

[32]:

plt.figure()
plt.imshow(res_pixel_picker['value_of_pixel'])

[32]:

<matplotlib.image.AxesImage at 0x7fb28d8adb50>
../_images/udf_introduction_55_1.png 
[ ]: