Workload-Aware HPC Storage Architecture Recommendation

Project overview

A new HPC storage infrastructure must be designed before its applications are deployed. Users can estimate capacity, budget and reliability but often cannot describe future input/output behavior. This internship extends an existing Lustre architecture recommender with patterns learned from historical HPC I/O logs. The project will discover representative workload profiles, match them to simple user descriptions and use them to refine feasible storage recommendations without requiring application code or logs from users.

ReDX has developed a Lustre storage recommender that collects numeric requirements, preferences and constraints through a conversation, then performs sizing, hardware selection and physical architecture validation. The current system does not yet use patterns learned from real application I/O behavior. Public Darshan logs provide historical I/O statistics. If profiling access is approved on Toubkal, pre-written workload descriptions may be paired with measured I/O behavior.

Project goals

Historical I/O analysis and workload profiling

• Collect and analyze public logs to study read/write activity, access sizes and other recorded I/O characteristics.
• Use data analysis and simple clustering methods to identify representative workload profiles and document their variability.

Learning from application descriptions

• Where permissions and tools allow, create paired examples on Toubkal by writing a short description before executing a workload and associating it with its measured I/O profile.
• Start with expert-reviewed rules and a simple model such as logistic regression.
• Evaluate the approach on unseen applications.
• Consider more advanced language models only if they demonstrate an improvement.
• If reliable paired examples are unavailable, use transparent scenario matching and report uncertainty rather than claiming supervised prediction.

Integration and evaluation

• Reuse the existing requirement and preference extractors and add optional questions about expected application usage.
• Use plausible workload profiles to compare feasible Lustre architectures without changing mandatory capacity, budget, compatibility or reliability constraints.
• Evaluate profile quality, classification performance and generalization to unseen applications.
• Compare recommendations with and without workload information and check feasibility and uncertainty.
• Support performance claims with appropriate measurements.

Required skills

• Good Python programming, basic Linux skills and familiarity with Git.
• Basic knowledge of data analysis or machine learning.
• Willingness to learn HPC storage concepts.
• Good analytical skills, technical English and interest in experimental research.

Training

• Introduction to HPC storage, Lustre and Darshan I/O profiling.
• Guided training in data preparation, basic machine learning and research evaluation.
• Regular mentoring and technical reviews with ReDX engineers and research supervisors.

Deliverables

• Reproducible I/O dataset preparation scripts and documented workload-profile library.
• Tested description-to-profile matcher integrated into the existing Lustre recommender.
• Comparative experiments, source code, documentation and final PFE report.

Conditions

• Recommended period: 4 to 6 months.
• Collaborative research toward a joint journal paper with ReDX and academic supervisors. The student may contribute to experiments and manuscript preparation; publication and authorship depend on results and contributions.
• Opportunity to work with real HPC I/O data and, subject to authorization, conduct profiling experiments on Toubkal.