Internal API

MaskContainer

libertem.common.container.MaskContainer helps to implement highly efficient mask application operations, such as virtual detector, center of mass or feature vector calculations.

Changed in version 0.4.0: Moved from libertem.job.masks to libertem.common.container to use it in UDFs and prepare deprecation of the Job interface.

class libertem.common.container.MaskContainer(mask_factories, dtype=None, use_sparse=None, count=None, backend=None)[source]

Container for mask stacks that are created from factory functions.

It allows stacking, cached slicing, transposing and conversion to condition the masks for high-performance dot products.

__init__(mask_factories, dtype=None, use_sparse=None, count=None, backend=None)[source]

use_sparse can be None, ‘scipy.sparse’, ‘scipy.sparse.csc’ or ‘sparse.pydata’

Shapes and slices

These classes help to manipulate shapes and slices of n-dimensional binary data to facilitate the MapReduce-like processing of LiberTEM. See Concepts for a high-level introduction.

class libertem.common.shape.Shape(shape, sig_dims)[source]
__init__(shape, sig_dims)[source]

Create a Shape that knows how many dimensions are part of navigation/signal. It is assumed that the signal is in the last sig_dims dimensions.

Parameters
  • shape (tuple of int) – the shape we want to work with, as n-tuple (like numpy array shapes)

  • sig_dims (int) – the number of dimensions that are considered part of the signal

property dims

Number of dimensions

Examples

>>> from libertem.common import Shape
>>> s = Shape((5, 5, 16, 16), sig_dims=2)
>>> s.dims
4
>>> s.nav.dims  # creates a new temporary Shape and accesses .dims on it
2
>>> s.sig.dims
2
flatten_nav()[source]

Returns a new Shape that is flat in the navigation dimensions

Examples

>>> from libertem.common import Shape
>>> s = Shape((5, 5, 16, 16), sig_dims=2)
>>> s.flatten_nav()
(25, 16, 16)
flatten_sig()[source]

Flatten in the signal dimensions

Examples

>>> from libertem.common import Shape
>>> s = Shape((5, 5, 16, 16), sig_dims=2)
>>> s.flatten_sig()
(5, 5, 256)
property nav

Crop to navigation dimensions

Returns

shape – like this shape, but without the signal dimensions

Return type

Shape

Examples

>>> from libertem.common import Shape
>>> s = Shape((5, 5, 16, 16), sig_dims=2)
>>> s.nav
(5, 5)
property sig

Crop to signal dimensions

Returns

shape – like this shape, but without the navigation dimensions

Return type

Shape

Examples

>>> from libertem.common import Shape
>>> s = Shape((5, 5, 16, 16), sig_dims=2)
>>> s.sig
(16, 16)
property size

Number of elements covered by this shape

Examples

>>> from libertem.common import Shape
>>> s = Shape((16, 16), sig_dims=2)
>>> s.size
256
to_tuple()[source]
class libertem.common.slice.Slice(origin, shape)[source]
__init__(origin, shape)[source]

A n-dimensional slice, defined by origin and shape

Parameters
  • origin (tuple of int) – global “top-left” coordinates of this slice

  • shape (Shape instance) – the size of this slice

adjust_for_roi(roi)[source]

Make a new slice that has origin and shape modified according to roi.

discard_nav()[source]

returns a copy with the origin/shape zeroed in the nav dimensions

this is used to create uniform cache keys

flatten_nav(containing_shape)[source]
get(arr=None, sig_only=False, nav_only=False)[source]

Get a standard python tuple-of-slice-object which can be used to slice any compatible numpy.ndarray

Parameters
  • arr – something implementing the slice interface. if given, returns arr[slice]

  • sig_only (bool) – get a signal-only slice for frames/masks

  • nav_only (bool) – get a nav-only slice, for example for indexing something that is shaped like the navigation dimensions of this Slice.

Returns

returns standard python slices computed from our origin+shape model or arr indexed with this slicing if arr is given

Return type

tuple of slice objects

Examples

>>> import numpy as np
>>> from libertem.common import Slice, Shape
>>> s = Slice(shape=Shape((16, 16, 4, 4), sig_dims=2), origin=(0, 0, 12, 12))
>>> data = np.ones((16, 16))
>>> data[s.get(sig_only=True)]
array([[1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.]])
intersection_with(other)[source]

Calculate the intersection between this slice and other. May result in dimensions that are zero, which means that there is no intersection.

Returns

slice – the intersection between this and the other slice

Return type

Slice

is_null()[source]

If any part of our shape is zero, this slice doesn’t span any data and is null / empty.

origin
shape
shift(other)[source]

make a new Slice with origin relative to other.origin and the same shape as this Slice

useful for translating to the local coordinate system of other

subslices(shape)[source]

Generator for all subslices of this slice with dimensions specified by shape.

Parameters

shape (tuple of int or Shape) – the shape of each sub-slice

Yields

Slice – all subslices, in fast-access order

CPU and CUDA devices

These methods get and set information that controls on which devices a computation runs.

libertem.common.backend.get_device_class()[source]

New in version 0.6.0.

Returns

class – Device class to use. Can be ‘cpu’ or ‘cuda’. Default is ‘cpu’ if no settings were applied before.

Return type

str

libertem.common.backend.get_use_cpu()[source]

New in version 0.6.0.

Returns

id – CPU device ID to use. Currently there is no pinning, i.e. the value itself is ignored. None means “don’t use CPU” and any integer means “use CPU”. Default is 0 if no settings were applied before.

Return type

int or None

libertem.common.backend.get_use_cuda()[source]

New in version 0.6.0.

Returns

id – CUDA device ID to use.

Return type

int or None

libertem.common.backend.set_use_cpu(cpu: int)[source]

This sets a CPU device ID and unsets any CUDA ID

New in version 0.6.0.

Parameters

cpu (int) – CPU to use. The value is currently ignored, i.e. any CPU is used without pinning

libertem.common.backend.set_use_cuda(cuda_device: int)[source]

This sets a CUDA device ID and unsets any CPU ID

New in version 0.6.0.

Parameters

cuda_device (int) – CUDA device ID to use

libertem.utils.devices.detect()[source]

Detect which devices are present

New in version 0.6.0.

Returns

Dictionary with keys 'cpus' and 'cudas' Each containing a list of devices. Only physical CPU cores are counted, i.e. no hyperthreading. Additionally it has the key 'has_cupy', which signals if cupy is installed and available.

Return type

dict

libertem.utils.devices.has_cupy()[source]

Probe if cupy was loaded successfully.

New in version 0.6.0.

CuPy is an optional dependency with special integration for UDFs. See CuPy support for details.