In our previous two blog posts of the Kong Gateway series, we focused on token handling in the context of the Zero-Trust Architecture (ZTA) principle ‘never trust, always verify’. For that principle implementation we explored both token validation and therapeutic token cloning to increase security. This time, we will take a closer look at plugin cloning, specifically how it can be used to enhance security through double ACL (Access Control List) checks to enable multi-layer authentication.

Overview

This technical blog is about plugin cloning on the Kong Gateway and how it can increase API security. Kong plugins are powerful components that extend the functionality of the Kong Gateway by allowing administrators and developers to add features to API traffic such as authentication, rate limiting, transformation, and logging. Each plugin operates within the request-response lifecycle, modifying or interacting with requests and responses as they pass through the gateway. Plugins can be declared globally to execute on every request, or scoped to specific Kong entities such as routes, services, consumers, or consumer groups (since Kong version 3.4), all within the Kong processing pipeline. Since Kong is built on Nginx, plugin functionality is executed in a specific Nginx phase, with the execution order defined by the plugin’s priority or ordering instruction (for the structure and layering of the Kong Gateway, see my introductory blog). Kong plugin cloning becomes essential in a few specific scenarios:

• when to run a plugin with a different priority (and the dynamic plugin ordering feature, which is limited to the access phase, cannot help),
• when to execute a plugin twice in one request-response cycle,
• to quickly fix issues in an existing Kong plugin (before the Kong engineering team fixes the issue), or
• to extend the Kong plugin logging or functionality (until the Kong engineering team implements a feature request).

This blog focuses on the first two scenarios, where cloning is essential for running the same plugin twice within a single request-response cycle. However, the last two scenarios are also valuable and have proven useful to me multiple times to provide a quick solution. In this case, one distinguishes between cloning the entire plugin or patching individual Lua modules.

Problem

The situation involves securing services using both mTLS client certificates and OAuth2 access tokens as part of a multi-layered authentication and authorization approach. In this setup, there were two different methods of presenting authentication credentials, each requiring its own access control check using Kong’s ACL (Access Control List) plugin. The Kong ACL plugin restricts access to APIs based on predefined whitelists or blacklists of consumer groups. It ensures that only authorized consumers can access specific services by checking their group membership.

Kong’s architecture prevents the same plugin from being executed multiple times within a single request-response cycle, regardless of its attachment to both a route and a service entity, causing a configuration challenge. I know experienced Kong consultants who sincerely believed that attaching a plugin to both a route and a service would make it execute twice; however, while this sounds logical, it is a common misconception. For instance, if a plugin is attached to both entities, only the route-level plugin will be executed, while the service-level plugin remains inactive. You can also learn more about this topic in the Kong precedence documentation and Kong knowledge article 1076.

Given this behavior, it became clear that attaching the ACL plugin to both the route and the service entity would not trigger multiple executions. Based on the given priority of the ACL plugin (priority 950), it is executed once after the customer identification by the OpenID Connect plugin (priority 1050) but not immediately after the preceding mTLS plugin (priority 1600), which is also able to identify a consumer. This means that only the ACL credentials for service access of the consumer identified by the OpenID Connect plugin are checked.

In most cases, the authentication plugins will likely identify the same consumer, so ACL authorization checking won’t be an issue. However, if the plugins identify different consumers, as in my case, an ACL authorization problem may arise. The following diagram illustrates this complex issue.

Following the request flow in the diagram above, the mTLS plugin is executed first to identify an API consumer by the certificate’s Subject Alternative Name (SAN) or Common Name (CN), with CN used only if SAN is absent. Without such a client certificate in the request, the mTLS plugin would reject the request immediately with a HTTP error 401 (Unauthorized). However, any valid client certificate can be presented here, even one identifying a consumer missing the ACL credentials to access service A, as the ACL plugin for authorization checks is only applied after the OpenID Connect plugin, which might identify another API consumer. A severe security vulnerability in the proposed multi-layer authentication approach.

You might consider using dynamic ordering to place the ACL plugin before the OpenID plugin in the execution sequence. However, this would result in the ACL plugin only checking the ACL credentials of the consumer identified by the mTLS plugin. In our use case, we need to check both consumers for ACL service access permissions. As a result, an alternative approach was needed to facilitate multiple ACL checks within the request processing of the Kong pipeline, including one ACL check for mTLS-authenticated consumers and another check for token authenticated consumers (for example via the AZP claim).

Solution

Basically, there are two ways to clone an existing plugin: manually or automatically. The manual method involves copying the plugin code, updating schema.lua and handler.lua with a new name for coexistence, assigning a new priority if needed, renaming database tables and cache keys to avoid collisions. The second method uses a plugin cloning tool provided by Kong, which wraps an existing plugin and only updates its priority. This approach does not support code changes but allows the same plugin to be run twice with a different priority, which is exactly what we need to run the ACL plugin twice. The tool provided by Kong is called priority-updater and can be obtained from the Kong Git repository.

Using the Kong priority-updater, I created an ACL plugin clone with priority 1595 (command: ./create.sh "acl" 1595). The plugin clone is therefore executed right after the mTLS plugin and thus checks the ACL credentials of the consumer authenticated by the mTLS plugin. The configuration change has created a new situation that meets the requirements of multi-layer authentication.

In this configuration, the service access authorization is checked first by the mTLS-authenticated consumer, and then by the OpenID-Connect-authenticated consumer, which is retained and becomes the final consumer in the further processing pipeline, as no further authentication plugins are used. For example, this consumer will also appear in the logging plugins. The initially identified consumer is overwritten and no longer exists in the Kong processing pipeline.

Conclusion

Authentication credentials arriving at the Kong Gateway must be validated. When mTLS client certificates and OAuth2 access tokens are part of a multi-layer authentication and authorization approach, both the incoming client certificates and the access tokens must be validated for service access authorization in the context of a Zero-Trust Architecture. This blog post outlines a specific implementation scenario where the ACL plugin is cloned and used twice to verify service access authorization, utilizing the Kong priority-updater tool.

And as we have seen once again with the implementation of a plugin cloning approach, with Kong Gateway you will not run into any limitations.

Special thanks also go to Karl Kalckstein and Carl Draper for their detailed information on the behavior of the Kong plugins in the context of ACL checking and the reference to the Kong priority-updater command-line tool.

Credits

Title image by Martin Barraud on iStock