Cost and Access Management for Enterprises Using the OpenAI API (or Other LLM APIs)

In the era of AI and machine learning, Large Language Models (LLMs) have garnered a massive amount of public attention. According to Reuters, OpenAI’s ChatGPT, which is based on their GPT 3.5 and GPT4 LLMs, amassed a record-breaking 100 million users in just two months (2). A plethora of businesses including Shopify, Snapchat and Instacart scrambled to create applications that use the OpenAI API (3). A 17B USD market for AIOps and Observability has emerged, with multi-billion dollar companies creating tools for LLM monitoring and evaluation (4).

There is a strong case for building both internal-facing (for employee use) and external-facing (consumer use) Enterprise applications that utilise the abilities of LLMs, for example by using OpenAI’s API endpoints. However, being an emerging technology, safely deploying these applications has proved to be notoriously difficult. Apart from their concerning tendency to ‘hallucinate’ (when the model outputs false information with a confident tone), LLMs are quite expensive when deployed at scale. In addition, current monitoring and usage tracking tools from LLM providers are lacking, for example the OpenAI API dashboard only shows usage information per organisation API key. This level of monitoring granularity does not allow for a business to track (and control) the LLM usage of each team or user of their LLM-powered application.

The Need for User Cost and Access Monitoring and Control

For internal-facing applications, using an LLM API without an LLM API gateway means a single team may deplete your entire organisational monthly budget. For external-facing applications, spammers can drain your monthly budget and the obscurity of OpenAI’s usage tracking dashboard makes it very challenging to accurately price your offering. Since LLM usage is priced per token, most of the traditional API gateways are not suitable for solving the aforementioned issues.

In both cases, the lack of an abstraction layer to the LLM API gives rise to the risk of exposing secret organisational API keys, and creates a lot of hassle to protect and monitor the usage of these keys. Furthermore, the inability to set usage and cost limits per team or per user makes it very difficult to safely deploy your LLM-powered application. It is clear that an LLM-specific API gateway is needed to allow for accurate budget and usage management on a user-level.

There are multiple solutions to the issue of cost and access management for LLM-powered applications. Of course, if you look at the URL of this blog you will notice that I am writing as part of LLMetrics, a hosted cost and access manager for LLM APIs with a purpose-built LLM-specific API gateway. Hence, a degree of bias is unavoidable when presenting the solutions; I encourage you to explore the links to the solutions presented to evaluate each option with a more balanced perspective.

Option 1: Modifying Existing API Gateways

One possible solution is to modify existing API gateways to handle LLM APIs. This involves creating custom plugins or extensions to add the necessary functionality for tracking and controlling LLM API usage. While this approach can provide a high degree of customization, it can be time-consuming and complex. It requires significant development effort and expertise, and may not provide the level of granularity and control that you need for effective budget and access management.

For example, you may choose to modify Kong, which is a popular open-source API gateway. Modifications may include tracking the response tokens of each API call to a separate database and calculating the cost spent for each user that way.

Option 2: Using LLM Observability Platforms

Another option is to use LLM Observability Platforms. These platforms provide tools for monitoring and evaluating LLM APIs. They can provide valuable insights into API usage and performance, but they may not offer the detailed cost tracking and access control features that you need. Additionally, these platforms can be expensive and may require a significant investment in infrastructure and resources. There are several popular LLM observability platforms out there, including Arize AI and Aporia. These place a heavy focus on LLM output accuracy and evaluation, compared to the more pragmatic cost and access management issues which a simpler LLM API gateway such as LLMetrics is suited for.

There are also a few open-source LLM monitoring platforms, such as Helicone. Helicone is great for logging purposes, and a simpler alternative to the enterprise-grade offerings above. However, it lacks the security features, user auth, and granular access management of a purpose built LLM API access manager.

Option 3: LLM API Cost and Access Managers, A Hassle-Free Solution

If you do not want to go through the trouble of creating custom plugins for traditional API gateways or building and maintaining your own LLM-specific API gateway, a hosted LLM API cost and access manager (such as LLMetrics) is your best bet.

This service provides a simple yet powerful solution for managing LLM APIs. It offers detailed cost tracking, robust access control, and an abstraction layer for the LLM API, all in a single, easy-to-use platform. LLMetrics is focused on controlling LLM costs and access on a user level, by automatically generating and handling child API keys for each user. This makes it much simpler to use, more robust and more scalable than applications that let you create child LLM API keys. LLMetrics eliminates the need for developing custom AI gateway infrastructure or using expensive observability platforms (often with many features that are not necessary for a simpler LLM API use-case), making it an ideal solution for businesses of all sizes.

Logging into the LLMetrics admin dashboard, you can define your organisation’s structure by adding users and allocating them to groups. For an internal-facing app, this could be each department you are planning to give LLM API access to as part of your application. For an external-facing app, this could be users in different tiers (i.e. free, simple paid, pro etc.). LLMetrics API keys are automatically generated for each user allowing them to communicate with our API (where their calls are tracked and controlled) as a proxy to the LLM provider (e.g. OpenAI). You can control LLM model access and set cost limits for each group. This reduces the risk of unauthorized use, which could lead to security breaches or unexpected costs. We handle user authentication and authorisation every time a user calls the LLMetrics API gateway.

LLMetrics provides granular tracking of API usage, allowing you to see exactly which users or teams are consuming your resources. This level of detail is crucial for effective budget management and can help you identify opportunities for cost savings. Our product, being an abstraction layer to the LLM APIs, eliminates the need for you to manage and monitor the usage of these organisational keys directly. This not only saves you time and effort but also enhances the security of your LLM-powered applications. Due to the user-focused nature of the platform, you do not have to worry about tracking (generated) user LLM API keys, we take care of this for you so you only have to worry about adding and removing users (or re-generating a user API key in the event that it has been exposed).

In conclusion, if you’re looking for a hassle-free solution to manage the costs and access of your LLM APIs, LLMetrics is an excellent choice. It provides all the essentially tools you need to effectively track and control your LLM API usage in a simple and easy-to-use offering, helping you to safely deploy your LLM-powered applications and manage your budget effectively. Sign up for the LLMetrics Beta here!

Bibliography:

(1) “TechCrunch Is Part of the Yahoo Family of Brands,” March 1, 2023.
(2) Hu, “ChatGPT sets record for fastest-growing user base – analyst note.”
(3) Gupta, “GPT-4 Takes the World by Storm – List of Companies That Integrated the Chatbot | Mint.”
(4) Woodie, “Who’s Winning In the $17B AIOps and Observability Market.”