The Impact of Hardware Variability on Applications Packaged with Docker and Guix: a Case Study in Neuroimaging

Very interesting.

Results show that hardware, software, and numerical variability lead to perturbations of similar magnitudes — albeit uncorrelated — suggesting that these three types of variability act as independent sources of numerical noise with similar magnitude.

We define “hardware variability" as the output differences caused by using CPU micro-architectures with or without AVX2 support, “software variability" the variability resulting from different compilation environments in Docker and Guix, mostly influenced by the FSL, gcc, libm, and OpenBlas versions, and “numerical variability" the variability resulting from random rounding.

AVX2 (Advanced Vector Extensions 2) is a CPU extension related to x86/AMD floating point & integer vector processing.

I only skimmed the article, and I’m not sure what to make of the “random rounding” variability.

Random rounding is a type of Monte Carlo Arithmetic, a stochastic arithmetic technique to empirically evaluate numerical stability by injecting noise into floating point operations and quantifying the resulting error at a given virtual precision. While Monte-Carlo Arithmetic provides different noise injection modes, we only used random rounding (RR).

Perhaps they’re saying that the precision of floating point computations varies slightly depending on the numbers themselves?

This was an interesting bit about Guix and Docker.

Docker and Guix, the two packaging solutions used in our experiments are known to mitigate software variability. From a computational bit-wise reproducibility point of view, experiments conducted in this study show that the two packaging solutions lead to similar conclusions: results are bit-wise reproducible when using the same packaged FLIRT executable on equivalent micro-architectures. We note however that, despite using the same version of the FLIRT source code, the two solutions yielded different outputs for most of the input files (but not for all). This is due to the software variability resulting from different compilation environments, mostly influenced by the gcc, libm and OpenBlas versions. The Docker image is a black box providing little or no information on how the executable was built (both on the compilation process, and on software dependency stack). In contrast, the Guix solution enforces full transparency on both compiling and runtime environments, requesting for the description and availability of all dependencies. The Guix package is thus more complex to produce than a Docker image, but, once available, variations can be easily built by modifying compiler options or by using other versions of dependent packages.

1 comments

Very interesting.
Results show that hardware, software, and numerical variability lead to perturbations of similar magnitudes — albeit uncorrelated — suggesting that these three types of variability act as independent sources of numerical noise with similar magnitude.
We define “hardware variability" as the output differences caused by using CPU micro-architectures with or without AVX2 support, “software variability" the variability resulting from different compilation environments in Docker and Guix, mostly influenced by the FSL, gcc, libm, and OpenBlas versions, and “numerical variability" the variability resulting from random rounding.
AVX2 (Advanced Vector Extensions 2) is a CPU extension related to x86/AMD floating point & integer vector processing.
I only skimmed the article, and I’m not sure what to make of the “random rounding” variability.
Random rounding is a type of Monte Carlo Arithmetic, a stochastic arithmetic technique to empirically evaluate numerical stability by injecting noise into floating point operations and quantifying the resulting error at a given virtual precision. While Monte-Carlo Arithmetic provides different noise injection modes, we only used random rounding (RR).
Perhaps they’re saying that the precision of floating point computations varies slightly depending on the numbers themselves?
This was an interesting bit about Guix and Docker.
Docker and Guix, the two packaging solutions used in our experiments are known to mitigate software variability. From a computational bit-wise reproducibility point of view, experiments conducted in this study show that the two packaging solutions lead to similar conclusions: results are bit-wise reproducible when using the same packaged FLIRT executable on equivalent micro-architectures. We note however that, despite using the same version of the FLIRT source code, the two solutions yielded different outputs for most of the input files (but not for all). This is due to the software variability resulting from different compilation environments, mostly influenced by the gcc, libm and OpenBlas versions. The Docker image is a black box providing little or no information on how the executable was built (both on the compilation process, and on software dependency stack). In contrast, the Guix solution enforces full transparency on both compiling and runtime environments, requesting for the description and availability of all dependencies. The Guix package is thus more complex to produce than a Docker image, but, once available, variations can be easily built by modifying compiler options or by using other versions of dependent packages.