VS Code Extension for DevOps : Using Local LLM(like Mistral, Llama2 etc.),

Summary

A VS Code extension has been developed to leverage local large language models (LLMs) like Mistral and Llama2, as well as cloud-based APIs from Azure OpenAI and OpenAI, offering DevOps professionals tools for code generation, summarization, and more, with a focus on flexibility and privacy.

Abstract

The article discusses the creation of a VS Code extension aimed at assisting DevOps professionals by integrating various large language models (LLMs). The extension supports three modes of operation: direct use of OpenAI model APIs, Azure-hosted OpenAI model APIs, and the ability to work with locally hosted LLMs such as Llama 2, Mistral, and Orca. This versatility allows users to choose the most suitable model for their needs, whether they are concerned with privacy and prefer local models or are willing to use cloud-based services for certain projects. The developer, who embarked on this as a personal weekend project, found the integration of open-source LLMs within the VS Code extension surprisingly straightforward, thanks to tools like ollama. Additionally, the extension includes a reporting feature to track user feedback and usage metrics, providing valuable insights for continuous improvement. The source code for the extension, named DevOps CoPilot, is available on GitHub, and the author encourages readers to try it out.

Opinions

The author values the flexibility of choosing between local and cloud-based LLMs, considering the sensitivity of the project code.
There is an emphasis on the ease of creating VS Code extensions, which facilitated the development of this tool.
The author sees the potential for open-source LLMs to perform comparably to proprietary models, hence the inclusion of local LLM support.
User feedback and metrics are considered important for evaluating the performance of the AI models used in the extension.
The author is enthusiastic about sharing the tool with the community and is open to feedback and further development.
A cost-effective AI service alternative to ChatGPT Plus (GPT-4) is recommended, highlighting the author's interest in providing value for money.

VS Code Extension for DevOps : Using Local LLM(like Mistral, Llama2 etc.), Azure Open AI and Open AI APIs

DevOps CoPilot

We have many VS Code extensions to help DevOps folks with code generation, summarization, problem finding, test generation, and more. Most of them are based on OpenAI models. I wanted to find out how we can use open-source LLMs and compare their performance locally based on user feedback. It would be helpful to have metrics and reporting for feedback. Writing a VS Code extension is easy, so I went with the VS Code extension experiment path. :-)

I was also thinking, what if I have a project code that I’m worried to share, and other projects that I don’t mind taking a risk with? So, I’m looking for flexibility while getting help from Gen AI.

This is another personal weekend GenAI project. I started by looking at open-source code and got a good idea of how to pursue it. I ended up creating a VS Code extension that can use all three modes of LLMs — OpenAI model APIs, Azure-hosted OpenAI model APIs, and most importantly, locally hosted LLMs such as Llama 2, Mistral, Orca, etc.

I thought it will be tricky to do Open source LLM integration within VS Code Extension but with ollama it turned out easy.

Also able to get simple reporting screen on VS Code extension to track feedback Metrics and usages.

This provide flexibility to choose model and more private/personal options with Local Open source LLMs.

Thanks for reading. Happy to share!!!