Multimodal Knowledge Assistant

Track

Software Engineering — reference SE-02

Project

Build a pipeline that indexes text, scanned PDFs, tables, charts, images and audio in one shared retrieval space.

Objectives

• Answer questions that mix several media types, with citations back to the exact page, image region or timestamp.
• Benchmark retrieval and answer quality per media type, and tune the system for speed and cost.

Candidate profile

Student in AI or software engineering, with solid Python skills and an interest in vision-language models and search systems.

Skills and tools

• Multimodal embeddings and late interaction
• Whisper and document layout parsing
• LLM guardrails and evaluation

The project is part of Astrolab’s PFE Book 2027. Projects have an estimated duration of up to six months, confirmed with the supervisor.