Artificial Intelligence (AI) is taking the world by storm; applications are increasingly leveraging AI to enhance their functionality, enabling smarter decision-making, improving user experience and allow greater efficiency. Applications that leverage AI do so through APIs that interface with large language models (LLMs), bringing us back to API Gateways, which ensure secure API traffic management. Kong Gateway is an established API Gateway that “speaks” AI. Therefore, it’s time to write about the great AI capabilities of the Kong Gateway also in this blog series.
Overview
API Gateways are inherently positioned to serve as the AI traffic management layer. As they evolve into AI Gateways, the distinction between “AI Gateway” and “API Gateway” is expected to increasingly blur.
AI Gateway providers see themselves as a bulwark against the “Wild West” enterprise AI, characterized by fragmented infrastructures, proprietary APIs and inconsistent security practices. Nowadays, organizations often juggle multiple LLM providers, each with unique integration requirements, leading to increased complexity and potential security vulnerabilities. To address these challenges, technology analysts such as Gartner, Forrester, or IDC view AI infrastructure/middleware solutions like AI Gateways as a key approach to centralizing and streamlining AI service integrations with the needed observability into AI operations. This approach not only mitigates the risks associated with the current unregulated AI landscape but also facilitates more efficient and secure AI deployments in enterprise environments.
Especially, companies that have adopted AI early are currently developing their own AI middleware, reflecting a common pattern in tech history of pioneering early adopters. This endeavour requires a significant investment of resources, and the capabilities of these in-house solutions frequently fall short compared to those of established AI Gateways like the Kong Gateway. Consequently, as history has often shown in the tech industry, this approach may not be the most effective or productive.
Gartner effectively outlined the features of AI Gateways in the summary of the report “Innovation Insight: AI Gateways”:
AI gateways manage, secure and protect API connections to AI providers. Software engineering leaders can use AI gateways to apply security; multi-LLM routing; cost visibility and data protection, scanning and risk mitigation controls to their AI usage.
The Gartner report explores the emerging role of AI Gateways in managing and securing interactions between applications and AI services. It emphasizes the necessity for organizations to gain better control and visibility over their AI usage. Forbes also breaks down the core features of AI Gateways.
Multi-LLM Routing: Directing requests to the optimal large language model based on specialization, cost or workload.
Cost Optimization: Tracking and analyzing AI service usage to optimize resource allocation and budget.
Outbound Security: Protecting sensitive API keys and implementing stricter access controls for AI services.
Data Governance Integration: Ensuring sensitive data isn’t transmitted during interactions with external AI providers.
As we will see, all of these features are already available on the Kong AI Gateway.
Ethical AI and AI Risk Management
After weighing my options on whether to introduce the AI Gateway with an overview blog or to dive straight into the individual AI features of the Gateway, I finally decided to go with the introductory blog. In future blogs, I will focus on individual AI features and plugins. This choice reflects my primary motivation to highlight the often-overlooked yet critical topics of Ethical AI and AI Risk Management, as I would like to quote the following definition from Anne Badman:
AI Risk Management is the process of systematically identifying, mitigating and addressing the potential risks associated with AI technologies. It involves a combination of tools, practices and principles. The goal is to minimize AI’s potential negative impacts while maximizing its benefits.
“AI for good” refers to the ethical use of artificial intelligence to address societal challenges. Generative AI, alongside its promising and limitless future, presents numerous ethical issues, including perpetuating biases, manipulation, deception, copyright infringement, and issues of accountability. Safety, robustness, fairness, transparency and environmental impact are other issues to consider.
We need responsible use of generative AI, including monitoring the requested and generated content and intervening when necessary to mitigate potential harm. Content moderation, albeit complex, is a vital aspect of this oversight, as AI engineers work to filter out harmful or biased results and implement safeguards that activate when users search for potentially harmful content, ensuring that the output remains appropriate and safe for specific audiences. IESE Dean, Franz Heukamp, put it succinctly:
There is power, and there is good, and they need to come together.
It’s up to all of us to ensure that unintended impacts don’t occur. It won’t happen by itself, an AI middleware is needed, which promotes the AI power but gives you the means to control. That’s the Kong AI Gateway.
Kong AI Gateway
Below you can find my own Kong AI Gateway overview diagram with a slightly different grouping compared to the official diagrams based on the Enterprise version 3.10. In general, the Kong AI Gateway is nothing else than the Kong API Gateway with AI plugins on top, some are OSS (the green bubbles), others are only available for Enterprise customers (the red bubbles). At the heart and center of the AI Gateway are the prompt engineering controls which directly tackle Ethical AI and AI Risk Management. Access to the LLM APIs is managed through the Unified LLM Access Controls, represented by the faded orange bubbles on the right. The supported vector databases are shown at the bottom, with purple bubbles indicating the default indices used by Kong.
So far, I have omitted two important components from the architecture diagram. However, they impressively demonstrate how Ethical AI and AI Risk Management can be implemented on the Kong Gateway. The “AI PII Anonymizer” services (the blue bubble on the left side) are specialized language-specific services provided by Kong that must run in tandem with the AI sanitizer plugin. Please note that access to the Kong AI PII Anonymizer service requires a private Docker image, which is available upon request from Kong Support. This service detects and sanitizes sensitive data for the “AI Sanitizer” plugin. Finally, the “Azure Content Safety SaaS” service (the faded yellow bubble on the right side) is a special SaaS offering operating on the Azure platform, which safeguards against potential harm by assessing every LLM request in relation to the harm categories “hate and fairness”, “sexual”, “violence” and “self-harm”.
Let’s go through the different control categories and list the available plugins.
The Unified LLM Access Controls are generally used for API LLM provider access. The provided plugins are AI Proxy and AI Proxy Advanced which enable an unified access to the AI APIs of various LLM providers. The AI Proxy Advanced offers additional load balancing and semantic routing capabilities. The AI Request Transformer and AI Response Transformer plugins enhance API requests and responses through a no-code approach, allowing users to easily integrate AI capabilities. These plugins leverage a unified access approach provided by Kong, with the AI Request Transformer introspecting and transforming incoming requests before they reach upstream services and the AI Response Transformer adjusting the upstream responses based on LLM instructions before returning them to clients.
The Cost and Performance Controls are represented by the two plugins AI Rate Limiting Advanced and AI Semantic Cache. AI Rate Limiting introspects LLM responses to calculate token cost and enable rate limits for the LLM backend service. While the AI Semantic Cache plugin allows you to semantically cache responses from LLMs to enhance performance and reduce operational costs.The AI RAG Injector Plugin is a Response Accuracy and Hallucination Mitigation Control. This plugin simplifies the creation of retrieval-augmented generation (RAG) by automatically injecting content from a vector database (Redis or Postgres pgvector) on the existing requests.
The Prompt Engineering Controls form the heart of the AI Gateway. This is where Ethical AI and AI Risk Management (“AI for good”) are implemented. The Azure Content Safety plugin allows administrators to enforce introspection with the Azure Content Safety service for all requests handled by the AI Proxy plugin (see the description above). The powerful AI Sanitizer plugin helps to protect sensitive information in client request bodies before they reach upstream services. The plugin ensures compliance with data privacy regulations. It supports multiple sanitization modes, including replacing sensitive information with fixed placeholders or generating synthetic replacements that retain category-specific characteristics. The AI Prompt Guard plugin allows the configuration of allow/deny lists using regular expressions. Whereas the AI Semantic Prompt Guard plugin allows the configuration of allow/deny lists using semantically similar prompts. The AI Prompt Template plugin enables administrators to provide pre-configured AI prompts to users. The plugin prohibits arbitrary prompts and gives the AI engineers back control and optimization potential. Ethical and risk control additions can be included in the prompt template by the AI engineers (invisible to the AI consumer). Same with the AI Prompt Decorator plugin which injects messages at the start or end of a caller’s chat history.
The final category is AI Observability and Alerting Controls which is crucial for ensuring cost-effective and reliable AI operations. Effective AI observability allows organizations to monitor, troubleshoot, and optimize AI usage in real-time, ensuring their reliability, security, and operational efficiency. These capabilities are supported by well-known observability plugins that have been enhanced with AI-specific features. For more information, visit AI Observability, AI Analytics, and AI Metrics.
Conclusion
I share the view by one of Kong’s competitors that “Evolved AI Gateways from API Gateways” like the Kong Gateway are superior to “Specific Purpose-Built AI Gateways” in the medium and long term. AI Gateways are not a replacement, but an evolution of API Gateways. All existing API Gateway features are also required in the AI context - from hiding AI credentials to in front multi-layer authentication.
Additionally, with Agentic AI, it’s already becoming apparent that AI Gateways will play an important key role in accessing autonomously the API catalog and configured API providers. The key word here is MCP (Model Context Protocol), which will be embedded in the Kong AI Gateway. Not to mention the already existing capabilities to securely safeguard remote MCP servers like traditional APIs through the Gateway.
The recently published Kyndryl survey reveals that approximately four out of ten CEOs believe their current infrastructure is insufficient for AI deployment, or that integrating AI into existing systems presents a significant challenge. With Kong AI Gateway, seamless integration enables organizations to more effectively harness AI capabilities, driving innovation and maintaining a competitive advantage while ensuring risk management and compliance with ethical standards.
And as we have seen once again with all the AI functionality and extensions, with Kong Gateway, you will not run into any limitations.
