Abstract
This paper presents our latest work on Computational Linguistics Applications for Multimedia Services (CLAMS), a open-source Artificial Intelligence (AI) and machine learning (ML) platform for various cultural institutions in the GLAM sector. CLAMS provides a framework for developing and implementing ML-based computational multimedia analysis tools, and optimizes the processing of audiovisual archival material by seamlessly integrating tools across various media types, including text, audio, video, and images. CLAMS’s primary function, automated content analysis and information extraction, provides archivists with an AI-assisted environment for metadata refinement. This will enable the cataloging of extensive audiovisual collections, which would be impossible to complete manually, thus ultimately increasing the usability of the audiovisual archives and allowing library patrons and media researchers to discover and search the archives more easily.
At the core of CLAMS interoperability is the Multi-Media Interchange Format (MMIF), a structured, JSON-based data abstraction that supports a consistent data exchange layer between different computational analysis tools, including AI and ML applications. This allows annotations from one tool to be easily used by others, enabling complex automated content analysis workflows.
The paper describes specifics of MMIF, the CLAMS platform and ecosystem, and case studies of CLAMS workflows and evaluation schemes using data from the American Archive of Public Broadcasting (AAPB). These use cases illustrate how CLAMS can enhance metadata for mass-digitized multimedia collections, that is often only implicitly available within the digitized media and are largely unsearchable and held in archives and libraries.