Kenji Kaneda - Chief Architect, CloudNatix

TL;DR

We would like to announce LLMariner. LLMariner is an open-source platform that brings enterprise-grade control to managing generative AI workloads, with an OpenAI-compatible API for easy integration.

Background

Large Language Models (LLMs) have been revolutionizing the computing industry, and its significance in the business space has been continuously growing. For example, a report from Mckinsey identified 63 generative AI use cases that could deliver total value in the range of $2.6 trillion to $4.4 trillion in economic benefits annually.

On the other hand, enterprises are still grappling with numerous challenges around deploying and managing LLMs despite this massive potential. In particular, enterprises have complex requirements on security, reliability and efficiency, and hosting LLMs while meeting these requirements is a very challenging task.

The mission of CloudNatix is to provide an efficient platform for enterprises and to foster innovation. Towards the mission, we have provided Autopilot and Co-pilot technologies for cloud cost optimization and cloud operation optimization for years. Given the increasing importance of LLMs in the enterprise space, it was very natural for CloudNatix to expand the scope and provide a technical solution to the problems that enterprises are having with LLMs.

What is LLMariner?

LLMariner makes LLMs ready for enterprise. You can host LLMs in your Kubernetes clusters and make them available instantly while keeping data protected and scaling directly in your hands. Imagine having complete authority over your infrastructure and the deployment of LLMs. This empowers enterprises to meet stringent security and compliance standards effortlessly.

The API provided by LLMariner is compatible with OpenAI API. As there is already an ecosystem built around OpenAI API, you can leverage it to quickly develop and deploy your LLM applications such as a support chat application or a coding assistant. LLMariner also keeps your infrastructure with optimized GPU usage and reduces the operational overhead of managing complex AI/ML infrastructure.

Why Open Source?

We decided to make LLMariner open source as we believe this technology should evolve in the open source community. The progress of AI/ML is rapid, and various open source technologies have emerged and will continue to emerge. It is important to leverage these technologies as much as possible, and being part of the open source community is the best way for that. 

For example, LLMariner provides an extensible framework for supporting multiple inference runtimes such as vLLM, Ollama, and Nvidia Triton Inference Server so that various models can be served with a runtime that is the best suited for them.By being open source, LLMariner not only offers transparency but also allows developers to contribute features, making it a collaborative effort to build the most adaptable solution for generative AI workloads (e.g, can be integrated with a new inference runtime information whenever it arises).

Conclusion

Ready to transform your AI infrastructure? LLMariner is available under Apache License 2.0. Dive into LLMariner today and see how easy it is to scale your generative AI models with enterprise-grade control. Visit our GitHub to get started, or catch our upcoming talk at CloudNative + Kubernetes AI Day to learn more!