Abstract
End-to-end archival video question answering (QA) is expensive because relevant evidence is dispersed across long recordings in degraded imagery, speech, on-screen text, and incomplete metadata. Tool orchestration instead converts these streams into compact annotations and retrieves only question-relevant evidence.
This dissertation presents CLAMS-Agent, a locally deployable agent built on the Computational Linguistics Applications for Multimedia Services (CLAMS) platform. A policy selects tools for speech recognition, on-screen-text extraction, visual captioning, speaker identification, and entity extraction; an answerer uses that evidence. The evaluation compares simulated on-demand processing with retrieval over a pre-processed archive and direct whole-video models. Its reusable annotations also support cataloging, search, description, and review.
The work contributes an archival-video QA benchmark, a tool-selected evidence framework, and an evaluation of accuracy and deployment cost. On 232 cleaned held-out free-text questions, CLAMS-Agent reaches 65.9% accuracy and single-hop RAG reaches 65.5%. An end-to-end audio-visual model reaches 66.7% on the 204 cleaned questions for which predictions are available, so that smaller-coverage result is not treated as a matched ranking. CLAMS-Agent provides about 2,000 selected text tokens per query rather than roughly 34,000 multimodal tokens, while peak memory falls from 33 to 18 GB. This 17-fold reduction in query representation and 45% lower peak memory support lower latency at comparable measured accuracy.