- [Bugfix] Addition between Shape and tuple
- [Bugfix] Restore user parameter on error
- [Bugfix] Handle empty ROI and extra_shape with zero
- [Bugfix] Avoid error for missing last-recent-directory
- [Bugfix] Fixed the parallel file browsing glitch
- [Feature] Coordinate transforms for center of mass analysis
- [Feature] Coordinate transforms
- [Feature] configurable corrections
- [Feature] Transparent GPU processing with CuPy
- [Feature] Support for loading stacks of 3D DM files
- [Feature] Flexible scan size
- [Feature] Add GPU configuration to GUI
- [Feature] CuPy support for holography UDF
- [Feature] I/O Overhaul
- [Feature] Change logging level from CLI
- [Feature] MRC file format support
- [Feature] Support for running multiple UDFs in one go through the data
- [Feature] Copy Jupyter notebook cell
- [Feature] Jupyter notebook download
- [Feature] Support for Norpix SEQ files
- [Feature] Shutdown button to stop server
- [Feature] Starred places in the browser
- [Misc] Benchmarks with pytest-benchmarks
- [Misc] Added alias for cmd line options
- X and Y component for colorwheel visualization swapped
- [Misc] Confirm exit message for ctrl+c
- [Misc] Threading control for Numba
- [Deprecation] Tileshape parameters are deprecated
- Import colormaps from empyre
0.5.0 / 2020-04-23¶
Progress bar support based on
tqdmthat can be enabled by passing
libertem.api.Context.map(): Running UDFs. (#613, #670, #655)
Include explicit support for Direct Electron’s DE5 format based on HDF5. (#704)
Relocatable GUI: Allow LiberTEM to run from different URL prefixes, allowing integration into, for example, JupyterLab. (#697)
No need to set thread count environment variables anymore since the thread count for OpenBLAS, OpenMP, Intel MKL and pyFFTW is now set on the workers at run-time. Numba support will be added as soon as Numba 0.49 is released. (#685).
libertem.io.dataset.raw.RawFileDataSetare deprecated and will be removed after 0.6.0. Please specify
detector_sizeinstead or use a specialized DataSet, for example for EMPAD.
libertem.udf.feature_vector_maker.FeatureVecMakerUDFis deprecated and will be removed in 0.6.0. Use
ApplyMasksUDFwith a sparse stack of single pixel masks or a stack generated by
LiberTEM works with Python 3.8 for experimental use. A context using a remote Dask.Distributed cluster can lead to lock-ups or errors with Python 3.8. The default local Dask.Distributed context works.
Improve performance with large tiles. (#649)
0.4.1 / 2020-02-18¶
This is a bugfix release, mainly constraining the
as distributed is not compatible to version 1.0 yet. It also contains
important fixes in the HDF5 dataset.
0.4.0 / 2020-02-13¶
Dismiss error messages via keyboard: allows pressing the escape key to close all currently open error messages (#437)
ROI doesn’t have any effect if in pick mode, so we hide the dropdown in that case (#511)
Make tileshape parameter of HDF5 DataSet optional (#578)
Fix FRMS6 in a distributed setting. We now make sure to only do I/O in methods that are running on worker nodes (#531).
DaskJobExecutor.run_each_host. Need to pass
pure=Falseto ensure multiple runs of the function (#528).
Job API deprecation¶
Support for regions of interest (ROIs).
Easier to implement, extend and re-use UDFs compared to Jobs.
Clean separation between back-end implementation details and application-specific code.
Facilities to implement non-trivial operations, see User-defined functions: advanced topics.
Performance is at least on par.
For that reason, the Job API has become obsolete. The existing public
libertem.api.Context.create_pick_job(), will be supported in LiberTEM for
two more releases after 0.4.0, i.e. including 0.6.0. Using the Job API will
trigger deprecation warnings starting with this release. The new
The Analysis classes that relied on the Job API as a back-end are already ported
to the corresponding UDF back-end. The new back-end may lead to minor
differences in behavior, such as a change of returned dtype. The legacy code for
using a Job back-end will remain until 0.6.0 and can be activated during the
transition period by setting
analysis.TYPE = 'JOB' before running.
ApplyMasksUDFreturns the result with the first axes being the dataset’s navigation axes. The last dimension is the mask index.
ApplyMasksJobused to return transposed data with flattened navigation dimension.
ROIs are supported now, like in all UDFs.
# Deprecated! mask_job = ctx.create_mask_job( factories=[all_ones, single_pixel], dataset=dataset ) mask_job_result = ctx.run(mask_job) plt.imshow(mask_job_result.reshape(dataset.shape.nav))
mask_udf = libertem.udf.masks.ApplyMasksUDF( mask_factories=[all_ones, single_pixel] ) mask_udf_result = ctx.run_udf(dataset=dataset, udf=mask_udf) plt.imshow(mask_udf_result['intensity'].data[..., 0])
PickFrameJob allowed to pick arbitrary contiguous
slices in both navigation and signal dimension. In practice, however, it was
mostly used to extract single complete frames.
PickUDF allows to pick the complete signal
dimension from an arbitrary non-contiguous region of interest in navigation
space by specifying a ROI.
If necessary, more complex subsets of a dataset can be extracted by constructing a suitable subset of an identity matrix for the signal dimension and using it with ApplyMasksUDF and the appropriate ROI for the navigation dimension. Alternatively, it is now easily possible to implement a custom UDF for this purpose. Performing the complete processing through an UDF on the worker nodes instead of loading the data to the central node may be a viable alternative as well.
PickUDF now returns data in the native
of the dataset. Previously,
PickFrameJob converted to
libertem.api.Context.create_pick_analysis() continues to be the
recommended convenience function to pick single frames.
Restructuring into sub-packages¶
We are currently restructuring LiberTEM into packages that can be installed and used independently, see #261. This will be a longer process and changes the import locations.
Blobfinder is the first module separated in 0.4.0.
See Package overview for a current overview of sub-packages.
For a transition period, importing from the previous locations is supported but
will trigger a
FutureWarning. See Show deprecation warnings on how to
activate deprecation warning messages, which is strongly recommended while the
restructuring is ongoing.
0.3.0 / 2019-12-12¶
Make OOP based composition and subclassing easier for
Introduce plain circular match pattern
libertem.io.dataset.dm.DMDataSetimplementation based on ncempy (#497)
Adds a new
map()executor primitive. Used to concurrently read the metadata for DM3/DM4 files on initialization.
Note: no support for the web GUI yet, as the naming patterns for DM file series varies wildly. Needs changes in the file dialog.
Speed up of up to 150x for correlation-based peak refinement in
libertem.udf.blobfinder.correlationwith a Numba-based pipeline (#468)
FullFrameCorrelationUDFwhich correlates a large number (several hundred) of small peaks (10x10) on small frames (256x256) faster than
Resolve shape mismatch issue and simplify dominant order calculation in Radial Fourier Analysis (#502)
Actually pass the
enable_directparameter from web API to the DataSet
The Job interface is planned to be replaced with an implementation based on UDFs in one of the upcoming releases.
0.2.2 / 2019-10-14¶
Point release to fix a number of minor issues, most notably PR #439 that should have been merged for version 0.2.
Slightly improved error and validation handling when opening files with GUI (ec74c13)
Recognize BLO file type (#432)
Fixed a glitch where negative peak elevations were possible (#446)
Update examples to match 0.2 release (#439)
0.2.1 / 2019-10-07¶
Point release to fix a bug in the Zenodo upload for production releases.
0.2.0 / 2019-10-07¶
This release constitutes a major update after almost a year of development. Systematic change management starts with this release.
This is the release message:
LiberTEM 0.2 offers a new API to define a wide range of user-defined reduction functions (UDFs) on distributed data. The interface and implementation offers a number of unique features:
Reductions are defined as functions that are executed on subsets of the data. That means they are equally suitable for distributed computing, for interactive display of results from a progressing calculation, and for handling live data¹.
Interfaces adapted to both simple and complex use cases: From a simple map() functionality to complex multi-stage reductions.
Rich options to define input and output data for the reduction functions, which helps to implement non-trivial operations efficiently within a single pass over the input data.
Composition and extension through object oriented programming
Interfaces that allow highly efficient processing: locality of reference, cache efficiency, memory handling
Advanced features: https://libertem.github.io/LiberTEM/udf/advanced.html
A big shoutout to Alex (@sk1p) who developed it! 🏆
¹User-defined functions will work on live data without modification as soon as LiberTEM implements back-end support for live data, expected in 2020.
Support for 4D STEM applications¶
In parallel to the UDF interface, we have implemented a number of applications that make use of the new facilities:
Correlation-based peak finding and refinement for CBED (credit: Karina Ruzaeva @kruzaeva)
Radial Fourier Series (advanced Fluctuation EM)
More details and examples: https://libertem.github.io/LiberTEM/applications.html
We have greatly improved the coverage of our documentation: https://libertem.github.io/LiberTEM/index.html#documentation
Fully automated release pipeline¶
Alex (@sk1p) invested a great deal of effort into fully automating our release process. From now on, we will be able to release more often, including service releases. 🚀
Basic dask.distributed array integration¶
LiberTEM can generate efficient dask.distributed arrays from all supported dataset types with this release. That means it should be possible to use our high-performance file readers in applications outside of LiberTEM.
0.1.0 / 2018-11-06¶
Initial release of a minimum viable product and proof of concept.
Support for applying masks with high throughput on distributed systems with interactive web GUI display and scripting capability.