Performance Analysis and Optimization of Scientific Python Environments on HPC Shared Filesystems

Context and objectives

This PFE project analyzes how software deployment strategies and filesystem placement affect the performance of scientific Python workloads on High-Performance Computing (HPC) clusters. Scientific environments managed with Conda, EasyBuild and Singularity/Apptainer may contain thousands of small files. When deployed on shared filesystems such as Lustre or NFS, simultaneous initialization across multiple compute nodes can generate substantial metadata activity, filesystem contention and startup latency.

The project will deploy equivalent PyTorch environments using Conda, EasyBuild and Apptainer across NFS, Lustre and node-local storage.

Project goals

• Evaluate the impact of scientific Python environment placement on shared-filesystem metadata activity and application startup performance.
• Investigate Singularity/Apptainer containers as a way to reduce filesystem metadata overhead.
• Assess the benefits and costs of staging software environments and container images on node-local storage.
• Characterize the influence of deployment strategies on PyTorch initialization, GPU training throughput and multi-node scalability.

Experimental work

• Measure installation time, Python library import latency, CUDA initialization and metadata-related operations.
• Execute PyTorch benchmarks, including ResNet-50 and DistributedDataParallel (DDP), using different deployment configurations and process-concurrency levels.
• Analyze startup time, GPU utilization, training throughput and multi-node scalability.
• Evaluate packaging environments into Apptainer images and staging them onto node-local storage.
• Develop a lightweight framework to automate experiments and recommend deployment strategies according to workload characteristics and measured overheads.
• Distinguish environment-related filesystem activity from training-dataset I/O while accounting for staging costs and filesystem caching effects.

Deliverables

• Reproducible benchmarking framework for Conda and containerized PyTorch environments.
• Comparative performance analysis across Lustre, NFS and node-local storage.
• Evaluation of filesystem metadata overhead and its impact on large-scale Python application initialization.
• Practical guidelines for efficient scientific Python environment deployment on HPC systems.

Required skills

• Strong problem-solving skills, curiosity and willingness to learn new technologies.
• Familiarity with Linux environments.
• Programming skills in C/C++ or Python.
• Familiarity with Git and version control.
• Good command of English.
• Ability to work independently and collaboratively.

Training and conditions

• Linux fundamentals through the “Complete Linux Training Course” on Udemy.
• Introduction to HPC systems and parallel programming, including MPI, OpenMP and GPU basics.
• Training on HPC software management tools such as EasyBuild.
• Mentorship through 1:1 sessions with ReDX engineers and domain experts.
• Recommended period: 6 months.
• Monthly stipend, with the possibility of an end-of-internship performance bonus.
• Possibility to work with end customers.