Deployment, Validation, and Optimization of the Scientific and AI Software Stack on an HPC cluster

Context and objectives

This PFE project focuses on deploying and validating a comprehensive scientific and AI software ecosystem on an HPC cluster. The stack includes programming compilers, parallel programming libraries, mathematical libraries, GPU computing frameworks, scientific applications and AI workloads. The project will establish reproducible deployment and validation procedures to support future software-stack maintenance and evolution.

Project goals

• Deploy the latest stable versions of compiler suites and toolchains, including GCC and Intel oneAPI.
• Install and validate MPI, OpenMP, BLAS/LAPACK, FFTW, HDF5 and related mathematical and scientific libraries.
• Deploy and validate CUDA, cuDNN and relevant GPU-accelerated libraries and frameworks.
• Install and validate applications such as GROMACS, LAMMPS, CP2K, VASP and Quantum ESPRESSO.
• Deploy and validate PyTorch, TensorFlow and associated GPU-acceleration components.
• Develop and maintain reproducible EasyBuild recipes and Lmod modulefiles.
• Perform functional and performance validation across CPU and GPU resources.
• Establish automated deployment, testing and validation procedures.

Scope and methodology

The project covers:

• Compiler and development toolchains, with support for C, C++ and Fortran applications.
• MPI implementations including OpenMPI and Intel MPI, OpenMP, BLAS/LAPACK, optimized mathematical libraries, FFTW, HDF5, NetCDF, cuBLAS, cuFFT and cuDNN.
• Scientific and engineering applications including GROMACS, LAMMPS, CP2K, Quantum ESPRESSO, VASP and ParaView.
• AI and machine-learning frameworks including PyTorch, TensorFlow, CUDA-enabled dependencies, NumPy, SciPy and scikit-learn.
• EasyBuild recipes, Lmod modulefiles and automated validation tests.

Validation will cover installation and dependency resolution, compiler and MPI compatibility, CPU and GPU functionality, single-node and multi-node execution, reference performance benchmarks and reproducibility across software versions and deployment environments. Representative benchmarks include HPL, OSU Micro-Benchmarks, STREAM and application-specific test cases.

Deliverables

• Comprehensive documented HPC software stack covering compilers, libraries, scientific applications and AI frameworks.
• Reproducible EasyBuild recipes and associated Lmod modulefiles.
• Validation reports covering installation status, compatibility, functionality and performance.
• Software compatibility matrix covering compiler versions, MPI implementations, libraries and application dependencies.
• Automated deployment and validation scripts.
• Technical documentation describing deployment procedures, configurations, limitations and maintenance recommendations.

Required skills

• Strong problem-solving skills, curiosity and willingness to learn new technologies.
• Familiarity with Linux environments and command-line operations.
• Programming skills in C/C++ or Python.
• Basic understanding of software compilation, libraries and dependency management.
• Familiarity with Git and version control.
• Interest in HPC, scientific computing or AI.
• Good command of English.
• Ability to work independently and collaboratively.

Training and conditions

• Linux fundamentals through the “Complete Linux Training Course” on Udemy.
• Introduction to HPC systems and parallel programming, including MPI, OpenMP and GPU computing fundamentals.
• Hands-on training on EasyBuild and Lmod.
• Introduction to software compilation, toolchains, dependency management and environment modules.
• Practical training on deploying and benchmarking scientific and AI applications.
• Introduction to automated testing and validation using scripting and benchmarking tools.
• Mentorship through 1:1 sessions with ReDX engineers and HPC domain experts.
• Recommended period: 6 months.
• Monthly stipend, with the possibility of an end-of-internship performance bonus.
• Possibility to work with end customers.
• Hands-on experience with a production HPC environment and a broad portfolio of scientific and AI applications.