From hours to minutes: moving measurement-data analysis to an HPC cluster
How we moved MF4 measurement analysis from engineering workstations to the HPC cluster, and why data staging and integrity checks mattered more than raw compute.
E-motor and inverter development produces a lot of measurement data. Test benches record MF4 files (ASAM MDF) with hundreds of signals at high sample rates, and engineers analyze them with MATLAB scripts. For a long time that analysis ran on the engineers' own workstations, so a large evaluation could block a machine for hours.
The problem
The analysis itself was sound. The bottleneck was where it ran:
- Large measurement campaigns took hours to process on a single workstation.
- Engineers waited for results, or started runs overnight and hoped nothing failed.
- Every engineer had a slightly different local setup, so results were hard to reproduce.
The company already had an HPC cluster and central NAS storage. The goal of AMD2Matlab was to let engineers use them without having to become cluster experts.
The approach
1. Convert once, in a format built for large data. An automated extraction step converts ASAM MDF files into MAT v7.3, the HDF5-based MATLAB format that supports large files and partial loading. Analysis code no longer parses MF4 itself.
2. Compiled MATLAB jobs, orchestrated on the cluster. The existing analysis scripts are compiled and submitted as parameterized jobs through JGen, the cluster's job generation tooling. Engineers keep writing MATLAB; the cluster runs it in parallel.
3. Treat data movement as the main design problem. Computing was rarely the slow part; moving gigabytes was. Input data is staged from the NAS to the compute nodes, processed there, and only the results (MAT files, PNG and JPG plots) are written back.
4. Prove it before rolling it out. A proof of concept and a formal acceptance test compared cluster results with workstation results, including SHA256 checksum validation of transferred data. Speed means nothing if the numbers might be different.
The result
- Processing time went from hours to minutes.
- Workstations stay free for interactive work while the cluster does the heavy lifting.
- Runs are reproducible, because every job uses the same compiled code and the same environment.
What I learned
- I/O is the real architecture. Where data lives and how often it moves decided performance far more than the number of cores.
- Keep the engineer's workflow familiar. Adoption came from engineers keeping their MATLAB code, not from a new tool they had to learn.
- Verify integrity explicitly. Checksums and side-by-side comparisons made the switch easy to trust.
- Align early. HPC, storage and analysis teams, plus external partners, all had a stake; agreeing on interfaces up front avoided rework.