Take control: Build your own privacy-focused meeting transcription & summarization tool with open-source LLMs
This whitepaper outlines the process of building a local, privacy-focused meeting transcription and summarization tool using open-source large language models (LLMs). The main goal is to create a solution that offers enhanced privacy, customization, and user trust by keeping all data on-premises, thereby avoiding the risks associated with third-party services.
Tl;dr
-
Traditional meeting notetaking is manual, time-consuming, and often biased. Automated solutions like Microsoft’s Copilot can handle these tasks but raise privacy concerns.
-
A local solution addresses these issues by ensuring all data is controlled by the user and is not stored or analyzed externally.
-
The tool is designed to run efficiently on consumer-grade hardware and effectively supports the local processing of the transcripts and summaries, making it accessible for users who don’t have access to high-end, enterprise-grade hardware setups.
-
The Whisper.cpp model by OpenAI is used for multilingual speech-to-text conversion via the Faster Whisper framework. This model is chosen for its accuracy, especially in English and German, and its ability to run efficiently on constrained hardware.
-
Self-Hosting an LLM: The tool uses the Mixtral 8x7B model for local language processing tasks. To host this model, the article discusses various frameworks for hosting LLMs were considered, including HuggingFace’s Text Generation Inference, Llama.cpp, and ExLlamaV2. Each has strengths and limitations, such as ease of setup, hardware requirements, and support for model formats.
-
The summarization process uses a map-reduce approach to manage the limited context window of LLMs. Transcripts are divided into smaller chunks for initial summarization, and then these summaries are combined to create a final comprehensive summary.
Conclusion
Despite the advantages of commercially available models like GPT-4, the article emphasizes the benefits of self-hosting, including data privacy and the ability to customize solutions. The tool was integrated into internal workflows, significantly reducing note-taking time and improving the quality of meeting summaries, although human oversight remains essential to ensure accuracy.
Overall, the article highlights the practical steps and considerations involved in developing a privacy-respecting, locallyhosted meeting transcription and summarization tool using open-source AI technologies.
Do you have questions about the topic, LLMs in general or are you curious about how you can use AI in your company in the future?
Contact our colleague at joao.tozato@feld-m.de – our team will be happy to assist you with advice and implementation expertise!
Do you have any experience with meeting transcript or summarisation tools – either in a private or professional context? We are keen to hear your opinion and look forward to discussing it with you.
Share this