AVAnnotate grew from a longstanding effort to make audiovisual cultural heritage more accessible, discoverable, and usable for research, teaching, and public engagement. Its history is rooted in a deceptively simple problem: although libraries, archives, and museums hold extraordinary collections of recorded sound and moving images, working closely with those materials has often remained much more difficult than working with text. Audio and video are time-based, difficult to search, frequently under-described, and often locked within systems that make annotation, contextualization, and publication challenging.
AVAnnotate emerged from years of research led by Tanya Clement at The University of Texas at Austin into how scholars listen to, analyze, describe, and share audiovisual collections. This work developed through the High Performance Sound Technologies for Access and Scholarship (HiPSTAS) project and through collaborations with scholars, libraries, archives, technologists, and cultural heritage institutions interested in creating better ways to work with sound.
From HiPSTAS to AudiAnnotate
In 2019, these efforts developed into The AudiAnnotate Project, led by Tanya Clement in collaboration with Ben Brumfield and Sara Brumfield of Brumfield Labs. With support from an American Council of Learned Societies Digital Extension Grant, AudiAnnotate began developing an open workflow that would allow scholars to annotate audiovisual materials and publish those annotations on the web using open standards.
At the center of this work was the International Image Interoperability Framework, or IIIF. Already widely used by libraries and museums to share images and cultural heritage materials, IIIF was expanding to support audio and video. AudiAnnotate explored what those emerging standards could make possible for humanities scholarship.
Rather than separating a recording from the scholarly work surrounding it, AudiAnnotate made it possible to bring media, time-stamped annotations, contextual information, metadata, and interpretation together in a web-based edition or exhibit. Researchers could move through a recording while encountering commentary attached to precise moments in the audio or video. In doing so, annotation became not simply a technical feature but a means of making audiovisual collections more legible, searchable, contextualized, and useful.
Building an Audiovisual Extensible Workflow
The project entered a major new phase in 2020 with a generous grant from the Mellon Foundation, then The Andrew W. Mellon Foundation, for the AudiAnnotate Audiovisual Extensible Workflow (AWE) project.
This support allowed the team to move beyond an initial proof of concept and build a broader open-source workflow for audiovisual scholarship. AWE connected annotation tools, IIIF standards, GitHub, archival repositories, and web publishing into an extensible process through which audiovisual materials could be accessed, annotated, organized, and shared.
Mellon’s support was crucial not only to the development of the technology but also to the larger scholarly ecosystem surrounding it. The project worked with cultural heritage organizations and research partners, developed documentation and workshops, created proof-of-concept projects, and investigated how audiovisual annotation could support researchers, teachers, students, archivists, and members of the public.
Partners during this period included institutions such as the Harry Ransom Center at The University of Texas at Austin, the Library of Congress, the Furious Flower Poetry Center at James Madison University, the Fortunoff Video Archive for Holocaust Testimonies at Yale University, the Louie B. Nunn Center for Oral History at the University of Kentucky, the Woodberry Poetry Room at Harvard University, the IIIF community, and the SpokenWeb research network.
This phase demonstrated something fundamental to the future of the project: providing access to audiovisual heritage requires more than digitizing a recording. Recordings become substantially more useful when people can identify what they contain, describe them, place them in context, teach with them, interpret them, and share that work with others.
Becoming AVAnnotate
In 2023, the project took another major step forward. With a new grant from the Mellon Foundation, AudiAnnotate evolved into AVAnnotate.
The new name reflected the project’s expanding ambitions. While AudiAnnotate had begun primarily through work with sound and audio collections, AVAnnotate was designed from the outset to provide stronger support for both audio and video while making the process of building annotated digital projects more interactive and accessible.
AVAnnotate retained the core commitments established through AudiAnnotate: free and open-source infrastructure, the use of open standards, sustainable web publishing, and collaboration among scholars and cultural heritage institutions. At the same time, the new phase expanded the project’s focus beyond the software itself.
AVAnnotate has worked to support the responsible use of audiovisual materials across research, teaching, archives, and public scholarship. This means not only making it easier to create annotations and digital exhibits, but also developing pedagogical resources, supporting scholars and students as they create projects, working with archives and collections, and confronting the ethical and political questions that accompany access to audiovisual cultural heritage.
These questions matter. Audiovisual collections frequently contain voices, bodies, performances, testimonies, memories, and histories whose circulation cannot be treated as technologically neutral. Increasing access therefore requires attention to context, permissions, representation, description, and the communities connected to the materials themselves. AVAnnotate’s development has increasingly placed these concerns alongside technological innovation.
An Open Infrastructure for Audiovisual Scholarship
Today, AVAnnotate allows researchers, teachers, students, archivists, and other users to create digital exhibits and editions centered on audio and video. Projects can combine audiovisual materials with time-stamped annotations, transcripts, essays, metadata, tags, and other forms of scholarly interpretation while using open technologies such as IIIF, GitHub, and static web publishing.
The history of AVAnnotate is therefore not simply the history of a software platform. It is the history of an evolving collaboration among humanities researchers, technologists, librarians, archivists, students, educators, and cultural heritage organizations asking a shared question: How can we make audiovisual collections easier to encounter, understand, interpret, teach with, and preserve for future scholarship?
The sustained support of the Mellon Foundation has been indispensable to pursuing that question. Across successive phases of the project, Mellon funding enabled AudiAnnotate and AVAnnotate to grow from an experimental scholarly workflow into a broader open infrastructure for audiovisual research, teaching, and public engagement. That support has made possible not only technological development, but partnerships, student and researcher involvement, workshops, fellowships, educational resources, and continued experimentation with more ethical and sustainable forms of access to audiovisual cultural heritage.
AVAnnotate continues that work today at The University of Texas at Austin, building on the foundation established by HiPSTAS and AudiAnnotate while developing new possibilities for scholarship with sound and moving images.