0.8.0 / 2021-10-04
This release mainly contains improvements of center of mass / first moment analysis and support for starting the web GUI from JupyterHub or JupyterLab.
In the web API, support was added to re-run visualization only, without re-running UDFs for an analysis. This allows for almost instant feedback for some operations, like changing CoM parameters.
Added token-based authentication. For now, it is only usable via integrations like Jupyter. It will be extended to local/manual usage later (#1074, #1097). Please comment on #1097 if local/manual use would be beneficial for you so that it is prioritized accordingly.
SEQ dataset: Added support for loading excluded pixels from XML (#805, #1077). See
SEQDataSetfor more information. Also support both
*.seqas extension for the main SEQ file to find files with matching base name that contain correction data (#1120, #1121).
Link to two more public datasets: High-resolution 4D STEM dataset of SrTiO3 and Synthetic 4D STEM dataset based on a SrTiO3 supercell (#1073).
Make sure tasks are scheduled dynamically on available workers if they have uneven run time to benefit more from GPUs (#1107).
Many thanks to our new contributors Levente Puskás for the excluded pixel loading and to Matthew Bryan for figuring non-standard compression in HDF5 and improving DM input validation. Congratulations to Alex for closing the long-standing CoM issue #31 and for enabling easy and secure access to the web interface on shared IT infrastructure.
0.7.1 / 2021-07-08
This is a bugfix release that ensures compatibility with the upcoming numba 0.54 release.
Our custom numba caching makes some assumptions about numba internals, which have changed in numba 0.54. This fixes compatibility with numba 0.54, and also makes sure we fail gracefully for future changes (#1060, #1061).
0.7.0 / 2021-06-10
This release introduces features that are essential for live data processing, but can be used for offline processing as well: Live plotting, API for bundled execution of several UDFs in one run, iteration over partial UDF results, and asynchronous UDF execution. Features and infrastructure that are specific to live processing are included in the LiberTEM-live package, which will be released soon.
Support for postprocessing of results on the main node after merging partial results. This adds
buffer(). See Post-processing after merging for details (#994, #1003, #1001).
Live plotting using the new
run_udf_iter(), as well as live plotting classes documented in Visualization. Pass
plots=Truefor simple usage. See Live Plotting as well as an example for the various possibilities for advanced usage (#980, #1011).
Allow some UDF-internal threading. This is mostly interesting for ad-hoc parallelization on top of the
InlineJobExecutorand live processing that currently relies on the
InlineJobExecutorfor simplicity, but could also be used for hybrid multiprocess/multithreaded workloads. Threads for numba, pyfftw, OMP/MKL are automatically controlled. The executor makes the number of allowed threads available as
libertem.udf.base.UDFMeta.threads_per_workerfor other threading mechanisms that are not controlled automatically (#993).
K2IS: reshaping, sync offset and time series support. Users can now specify a
sync_offsetfor a K2IS data set, and load time series data (#1019, #911). Many thanks to @AnandBaburajan for implementing this feature!
UDF: Consistently use attribute access in
UDF.get_results()etc. instead of mixing it with
__getitem__()dict-like access. The previous method still works, but triggers a
Better choice of
kind='nav'buffer fill value outside ROI.
String : Was
bool : Was
integers : Was smallest possible value, now
objects : was
Improve performance for chunked HDF5 files, especially compressed HDF5 files which have a chunking in both navigation dimensions. They were causing excessive read amplification (#984).
Fix axes order in COM template: The components in the field are (x, y) while the template had them as (y, x) before (#1023).
Update Gatan Digital Micrograph (GMS) examples to work with the current GMS and LiberTEM releases and demonstrate the new features. (#999, #1002, #1004, #1011). Many thanks to Winnie from Gatan for helping to work around a number of issues!
Restructure UDF documentation (#1034).
Removed deprecated blobfinder and
FeatureVecMakerUDFas previously announced. Blobfinder is available as a separate package at https://github.com/liberTEM/LiberTEM-blobfinder. Instead of
FeatureVecMakerUDF, you can use a sparse matrix and
Jobinterface as previously announced. The functionality was ported to the more capable UDF interface #978.
0.6.0 / 2021-02-16
We are pleased to announce the latest LiberTEM release, with many improvements since 0.5. We would like to highlight the contributions of our GSoc 2020 students @AnandBaburajan (reshaping and sync offset correction) and @twentyse7en, (Code generation to replicate GUI analyses in Jupyter notebooks) who implemented significant improvements in the areas of I/O and the user interface.
Another highlight of this release is experimental support of NVidia GPUs, both via CuPy and via native libraries. The API is ready to be used, including support in the GUI. Performance optimization is still to be done (#946). GPU support is activated for all mask-based analyses (virtual detector and Radial Fourier) for testing purposes, but will not bring a noticeable improvement of performance yet. GPU-based processing did show significant benefits for computationally heavy applications like the SSB implementation in https://github.com/Ptychography-4-0/ptychography.
A lot of work was done to implement tiled reading, resulting in a new I/O system. This improves performance in many circumstances, especially when dealing with large detector frames. In addition, a correction module was integrated into the new I/O system, which can correct gain, subtract a dark reference, and patch pixel defects on the fly. See below for the full changelog!
Allow UDFs that implement
process_tileto influence the tile shape by overriding
libertem.udf.base.UDF.get_tiling_preferences()and make information about the tiling scheme available to the UDF through
libertem.udf.base.UDFMeta.tiling_scheme. (#554, #247, #635).
MemoryDataSetto allow testing with different tile shapes (#634).
Added I/O backend selection (#896), which allows users to select the best-performing backend for their circumstance when loading via the new
Context.load. This fixes a K2IS performance regression (#814) by disabling any readahead hints by default. Additionaly, this fixes a performance regression (#838) on slower media (like HDDs), by adding a buffered reading backend that tries its best to linearize I/O per-worker. GUI integration of backend selection is to be done.
For now, direct I/O is no longer supported, please let us know if this is an important use-case for you (#716)!
Support for specifying logging level from CLI (#758).
Support for loading stacks of 3D DM files (#877). GUI integration still to be done.
GUI: Filebrowser improvements: users can star directories in the file browser for easy navigation (#772).
Support for running multiple UDFs “at the same time”, not yet exposed in public APIs (#788).
GUI: Users can add or remove scan size dimensions according to the dataset’s shape (#779).
GUI: Shutdown button to stop server, useful for example for JupyterHub integration (#786).
Infrastructure for consistent coordinate transforms are added in
libertem.utils. See also a description of coordinate systems in Concepts.
Support for loading dark frame and gain map that are sometimes shipped with SEQ data sets.
Spinning out holography to a separate package is in progress: https://github.com/LiberTEM/LiberTEM-holo/
GUI: Allows the user to select the GPUs to use when creating a new local cluster (#812).
GUI: Support to download Jupyter notebook corresponding to an analysis made by a user in GUI (#801).
Allow reshaping datasets into a custom shape. The
DataSetimplementations (currently except HDF5 and K2IS) and GUI now allow specifying
sig_shapeparameters to set a different shape than the layout in the dataset (#441, #793).
DataSetimplementations (except HDF5 and K2IS) and GUI now allow specifying a
sync_offsetto handle synchronization/acquisition problems (#793).
Cache warmup when opening a data set: Precompiles jit-ed functions on a single process per node, in a controlled manner, preventing CPU oversubscription. This improves further through implementing caching for functions which capture other functions in their closure (#886, #798).
Allow selecting lin and log scaled visualization for sum, stddev, pick and single mask analyses to handle data with large dynamic range. This adds key
PickResultSetand the result of
SDAnalysis. It adds key
SingleMaskResultSet. The new keys are chosen to not affect existing keys (#925, #929).
Tuples can be added directly to
Shapeobjects. Right addition adds to the signal dimensions of the
Shapeobject while left addition adds to the navigation dimensions (#749)
Fix an off-by-one error in sync offset for K2IS data (drive-by change in #706).
Missing-directory error isn’t thrown if it’s due to last-recent-directory not being available (#748).
GUI: when cluster connection fails, reopen form with parameters user submitted (#735).
GUI: Fixed the glitch in file opening dialogue by disallowing parallel browsing before loading is concluded (#752).
Handle empty ROI and extra_shape with zero. Empty result buffers of the appropriate shape are returned if the ROI is empty or
extra_shapehas a zero (#765)
Libertem-server can now be started from Bash on Windows (#731)
Fix reading without a copy from multi-file datasets. The start offset of the file was not taken account when indexing into the memory maps (#903).
Improve performance and reduce memory consumption of point analysis. Custom right hand side matrix product to reduce memory consumption and improve performance of sparse masks, such as point analysis. See also scipy/13211 (#917, #920).
Fix stability issue with multiple dask clients.
dd.as_completedneeds to specify the
loopto work with multiple
Switched to the readthedocs sphinx theme, improving the overall documentation structure. The developer documentation is now in a separate section from the user documentation.
Command line options can also be accessed with shorter alternatives (#757).
libertem-server: Ask for confirmation if the user press ctrl+c. Can immediately stop using another ctrl+c (#781).
The X and Y components for the color wheel visualization in Center of Mass and Radial Fourier Analysis are swapped to match the axis convention in empyre. This just changes the color encoding in the visualization and not the result (#851).
Remove color wheel code from
libertem.vizand replace with imports from empyre. Note that these functions expect three vector components instead of two (#851).
The new and consistent
sig_shapeparameters should be used when loading data. The old
detector_sizeparameters, where they existed, are still recognized (#793).
0.5.1 / 2020-08-12
Allow installation with latest dask distributed on Python 3.6 and 3.7
0.5.0 / 2020-04-23
Progress bar support based on
tqdmthat can be enabled by passing
libertem.api.Context.map(): Running UDFs. (#613, #670, #655)
Include explicit support for Direct Electron’s DE5 format based on HDF5. (#704)
Relocatable GUI: Allow LiberTEM to run from different URL prefixes, allowing integration into, for example, JupyterLab. (#697)
No need to set thread count environment variables anymore since the thread count for OpenBLAS, OpenMP, Intel MKL and pyFFTW is now set on the workers at run-time. Numba support will be added as soon as Numba 0.49 is released. (#685).
libertem.io.dataset.raw.RawFileDataSetare deprecated and will be removed after 0.6.0. Please specify
detector_sizeinstead or use a specialized DataSet, for example for EMPAD.
libertem.udf.feature_vector_maker.FeatureVecMakerUDFis deprecated and will be removed in 0.6.0. Use
ApplyMasksUDFwith a sparse stack of single pixel masks or a stack generated by
LiberTEM works with Python 3.8 for experimental use. A context using a remote Dask.Distributed cluster can lead to lock-ups or errors with Python 3.8. The default local Dask.Distributed context works.
Improve performance with large tiles. (#649)
0.4.1 / 2020-02-18
This is a bugfix release, mainly constraining the
as distributed is not compatible to version 1.0 yet. It also contains
important fixes in the HDF5 dataset.
0.4.0 / 2020-02-13
Dismiss error messages via keyboard: allows pressing the escape key to close all currently open error messages (#437)
ROI doesn’t have any effect if in pick mode, so we hide the dropdown in that case (#511)
Make tileshape parameter of HDF5 DataSet optional (#578)
Fix FRMS6 in a distributed setting. We now make sure to only do I/O in methods that are running on worker nodes (#531).
DaskJobExecutor.run_each_host. Need to pass
pure=Falseto ensure multiple runs of the function (#528).
Job API deprecation
The original Job API of LiberTEM is superseded by the new User-defined functions (UDFs) API with release 0.4.0. See #549 for a detailed overview of the changes. The UDF API brings the following advantages:
Support for regions of interest (ROIs).
Easier to implement, extend and re-use UDFs compared to Jobs.
Clean separation between back-end implementation details and application-specific code.
Facilities to implement non-trivial operations, see User-defined functions: advanced topics.
Performance is at least on par.
For that reason, the Job API has become obsolete. The existing public
libertem.api.Context.create_pick_job(), will be supported in LiberTEM for
two more releases after 0.4.0, i.e. including 0.6.0. Using the Job API will
trigger deprecation warnings starting with this release. The new
The Analysis classes that relied on the Job API as a back-end are already ported
to the corresponding UDF back-end. The new back-end may lead to minor
differences in behavior, such as a change of returned dtype. The legacy code for
using a Job back-end will remain until 0.6.0 and can be activated during the
transition period by setting
analysis.TYPE = 'JOB' before running.
ApplyMasksUDFreturns the result with the first axes being the dataset’s navigation axes. The last dimension is the mask index.
ApplyMasksJobused to return transposed data with flattened navigation dimension.
ROIs are supported now, like in all UDFs.
# Deprecated! mask_job = ctx.create_mask_job( factories=[all_ones, single_pixel], dataset=dataset ) mask_job_result = ctx.run(mask_job) plt.imshow(mask_job_result.reshape(dataset.shape.nav))
mask_udf = libertem.udf.masks.ApplyMasksUDF( mask_factories=[all_ones, single_pixel] ) mask_udf_result = ctx.run_udf(dataset=dataset, udf=mask_udf) plt.imshow(mask_udf_result['intensity'].data[..., 0])
PickFrameJob allowed to pick arbitrary contiguous
slices in both navigation and signal dimension. In practice, however, it was
mostly used to extract single complete frames.
PickUDF allows to pick the complete signal
dimension from an arbitrary non-contiguous region of interest in navigation
space by specifying a ROI.
If necessary, more complex subsets of a dataset can be extracted by constructing a suitable subset of an identity matrix for the signal dimension and using it with ApplyMasksUDF and the appropriate ROI for the navigation dimension. Alternatively, it is now easily possible to implement a custom UDF for this purpose. Performing the complete processing through an UDF on the worker nodes instead of loading the data to the central node may be a viable alternative as well.
PickUDF now returns data in the native
of the dataset. Previously,
PickFrameJob converted to
libertem.api.Context.create_pick_analysis() continues to be the
recommended convenience function to pick single frames.
Restructuring into sub-packages
We are currently restructuring LiberTEM into packages that can be installed and used independently, see #261. This will be a longer process and changes the import locations.
Blobfinder is the first module separated in 0.4.0.
See Package overview for a current overview of sub-packages.
For a transition period, importing from the previous locations is supported but
will trigger a
FutureWarning. See Show deprecation warnings on how to
activate deprecation warning messages, which is strongly recommended while the
restructuring is ongoing.
0.3.0 / 2019-12-12
Make OOP based composition and subclassing easier for
Introduce plain circular match pattern
Adds a new
map()executor primitive. Used to concurrently read the metadata for DM3/DM4 files on initialization.
Note: no support for the web GUI yet, as the naming patterns for DM file series varies wildly. Needs changes in the file dialog.
Speed up of up to 150x for correlation-based peak refinement in
libertem.udf.blobfinder.correlationwith a Numba-based pipeline (#468)
FullFrameCorrelationUDFwhich correlates a large number (several hundred) of small peaks (10x10) on small frames (256x256) faster than
Resolve shape mismatch issue and simplify dominant order calculation in Radial Fourier Analysis (#502)
Actually pass the
enable_directparameter from web API to the DataSet
The Job interface is planned to be replaced with an implementation based on UDFs in one of the upcoming releases.
Split up the blobfinder code between several files to reduce file size (#468)
0.2.2 / 2019-10-14
Point release to fix a number of minor issues, most notably PR #439 that should have been merged for version 0.2.
Slightly improved error and validation handling when opening files with GUI (ec74c13)
Recognize BLO file type (#432)
Fixed a glitch where negative peak elevations were possible (#446)
Update examples to match 0.2 release (#439)
0.2.1 / 2019-10-07
Point release to fix a bug in the Zenodo upload for production releases.
0.2.0 / 2019-10-07
This release constitutes a major update after almost a year of development. Systematic change management starts with this release.
This is the release message:
LiberTEM 0.2 offers a new API to define a wide range of user-defined reduction functions (UDFs) on distributed data. The interface and implementation offers a number of unique features:
Reductions are defined as functions that are executed on subsets of the data. That means they are equally suitable for distributed computing, for interactive display of results from a progressing calculation, and for handling live data¹.
Interfaces adapted to both simple and complex use cases: From a simple map() functionality to complex multi-stage reductions.
Rich options to define input and output data for the reduction functions, which helps to implement non-trivial operations efficiently within a single pass over the input data.
Composition and extension through object oriented programming
Interfaces that allow highly efficient processing: locality of reference, cache efficiency, memory handling
Advanced features: https://libertem.github.io/LiberTEM/udf/advanced.html
A big shoutout to Alex (@sk1p) who developed it! 🏆
¹User-defined functions will work on live data without modification as soon as LiberTEM implements back-end support for live data, expected in 2020.
Support for 4D STEM applications
In parallel to the UDF interface, we have implemented a number of applications that make use of the new facilities:
Correlation-based peak finding and refinement for CBED (credit: Karina Ruzaeva @kruzaeva)
Radial Fourier Series (advanced Fluctuation EM)
More details and examples: https://libertem.github.io/LiberTEM/applications.html
We have greatly improved the coverage of our documentation: https://libertem.github.io/LiberTEM/index.html#documentation
Fully automated release pipeline
Alex (@sk1p) invested a great deal of effort into fully automating our release process. From now on, we will be able to release more often, including service releases. 🚀
Basic dask.distributed array integration
LiberTEM can generate efficient dask.distributed arrays from all supported dataset types with this release. That means it should be possible to use our high-performance file readers in applications outside of LiberTEM.
Support for various file formats has improved. More details: https://libertem.github.io/LiberTEM/formats.html
0.1.0 / 2018-11-06
Initial release of a minimum viable product and proof of concept.
Support for applying masks with high throughput on distributed systems with interactive web GUI display and scripting capability.