Why Python? =========== Python is well established in the scientific community. It is usable both for developers, but also for many microscopists. Its high-level construct allow for fast iteration and prototyping. There is an extensive ecosystem of packages for scientific computation leveraging existing native libraries. Python has good interoperability with low-level languages like C, which means it is well suited as a glue language for existing low-level routines, without introducing inefficiencies such as copies of large buffers on each interaction. Isn't Python slow? ------------------ Yes, it can be slow, but we mostly use Python for setting up the computation, creating buffers, setting parameters, etc. We use it as a glue language for native parts (NumPy/OpenBLAS, ...) or use `Numba `_ for critical code sections. See for example this profile, visualized as a flamegraph: .. image:: ./images/read_from_hdfs_profile.png The workload is similar to what LiberTEM will later do: it reads files from HDFS and does matrix multiplication on the data. Most of the time is spent reading the file (block on the left: :code:`sys_read`) or actually performing the matrix multiplication (center block: anything containing :code:`dgemm`). The Python parts are mostly in the narrow (= little time) but high (= deep call stacks) pillar on the right. The dask scheduler is also visible in the profile, but takes up less than 2% of the total samples. Note the :code:`swapper` part on the right: this was a full-system profile, so unrelated things like :code:`swapper` or :code:`intel_idle` are also included. But what about (multicore) scaling? ----------------------------------- ``NumPy`` releases the GIL, so multiple threads can work at the same time. Even if this were not the case, we can still use the multiprocessing workers of ``dask.distributed`` and scale to multiple cores. See also the :doc:`performance` section.