In the last blog post, we discussed the topic of Security Assertion Markup Language (SAML) 2.0 assertions as a form of authentication. This time, we focus on another important security authentication topic that also plays an important role in building a zero-trust architecture (ZTA). Mutual transport layer security (mTLS) with consumer authentication using client certificates at the Kong Gateway is essential in the context of perimeter security. This is still true, even if components that act as TLS terminating reverse proxies (TTRP) are used.

Overview

My previous blog articles ended with the saying that Kong is the API gateway without limits. This technical blog is about mTLS (Mutual Transport Layer Security) in an advanced way or to be more precise, in an extended distributed way. mTLS is an extension of TLS (Transport Layer Security) that is designed to provide authentication and encryption for communication between two parties, such as an API client and an API gateway. Kong refers to API clients and users as “consumers”. In Kong’s terminology, a consumer is an entity that consumes or accesses APIs managed by Kong. This entity embodies a user, an application, or a group of users or applications. So, it’s about the secure communication and authentication of the API consumer which can be implemented on the gateway via mTLS.

In a mutual TLS setup, both the API consumer and the gateway authenticate each other using digital certificates (details on the structure of X.509 certificates can be found in the IETF RFC 5280 specification document). This means that not only does the gateway act as a server presenting a certificate to prove its identity to the consumer (that’s the default part of the TLS protocol), but the consumer also presents a certificate to prove its identity to the gateway (this is the mutual or two-way part of the authentication). This mutual authentication adds an extra layer of security to ensure that both parties are who they claim to be. This extra step of consumer-to-server authentication is commonly used in scenarios where a high level of security is required.

Kong supports the mutual TLS handshake to authenticate the consumer based on the client certificate provided by the Mutual TLS Authentication plugin. The Kong consumer entity has the ability to attach plugins to customize request behavior, similar to service or route entities (details about Kong consumers, plugins, services, and routes can be found in my introductory blog). Basic (meaning username and password), API key, or mTLS credentials can also be assigned to a consumer. These credentials then play a role in authenticating the consumer via HTTP Basic AuthN, API keys AuthN, or mTLS AuthN, as in the considered case.

Problem

But what about the advanced way I mentioned at the beginning? A problem can arise from the protection of the outer boundaries of a network, the perimeter security. IT perimeter security protects an organization’s network, systems and data from unauthorized access and external threats by establishing barriers at the network edge. These barriers can include firewalls, intrusion detection equipment, access controls and other security technologies designed to monitor and control traffic entering the network to be protected.

Perimeter security can terminate SSL/TLS connections to inspect the content of encrypted traffic to detect security threats. This process is called Deep Packet Inspection (DPI), also known as packet sniffing, and is a type of network packet filtering performed by perimeter security components. DPI is only possible by intercepting the SSL/TLS connection and then decrypting and inspecting the payload of the data packet. If there is no security issue, then the data packet is re-encrypted before being forwarded to the intended destination. This process breaks the intended SSL handshake with the gateway because the security device acts as a man-in-the-middle safeguard between the API consumer and the API gateway. This means that the original client certificate does not even arrive at the gateway. As a result, the gateway loses the ability to authenticate the consumer based on the information contained in the client certificate.

Fortunately, advanced SSL/TLS-terminating network components like load balancers (LBs) or web application firewalls (WAFs) can be configured to pass the entire client cert as request header to the originally intended server. Even more advanced is the ability to extract selected fields from the client certificate and inject them as headers into the request being forwarded to the API gateway. For example, one of these advanced products that can pass client certificate fields as request headers is the Google Cloud Platform (GCP) external application load balancer. When mTLS is enabled on this load balancer, custom request headers can be configured to pass information about the mTLS connection to backend services, in our case the API gateway. Therefore, the GCP external application load balancer follows the new standard of IETF RFC 9440 as a TLS terminating reverse proxy (TTRP). This RFC document describes the process of forwarding client certificate information with HTTP extension header fields. The standard only mentions the header fields Client-Cert and Client-Cert-Chain. Of course, the GCP load balancer can extract and forward these mTLS fields, but it can also extract individual certification information. We will take advantage of this feature. The load balancer behavior in the event of missing or invalid certificates can also be configured accordingly. The most important one for authentication is the X-Client-Cert-DNSName-SANs field, which contains a comma-separated base64-encoded list of subject alternative name (SAN) extensions of type DNSName. These SAN extensions are extracted from the client certificate by the load balancer.

The diagram below illustrates a common DMZ setup in the GCP, with perimeter security provided by an external load balancer connected to Cloud Armor, a dedicated web application firewall (WAF), and the Kong API gateway. This setup includes the transport of mTLS client certificate information to the gateway as request headers.

The IETF RFC 6125 specification document defines a standard for validating the identities in secure communication protocols. According to RFC 6125, the SAN is the recommended field to extract, but if the SAN field of type DNSName is missing, the CN (subject common name) has to suffice.

… If a subjectAltName extension of type dNSName is present, that MUST be used as the identity. Otherwise, the (most specific) Common Name field in the Subject field of the certificate MUST be used. Although the use of the Common Name is existing practice, it is deprecated and Certification Authorities are encouraged to use the dNSName instead.

- RFC 6125, Appendix B, HTTP in Representation and Verification of Domain-Based Application Service

Within the SAN extension, each entry can be tagged with a specific type to indicate the type of information it contains. If the SAN has the DNS name tagging (OID 2.5.29.17.2), it must be used for identity identification according to the specification. No tagging means implicit DNS name tagging.

By the way, the mTLS-auth plugin follows these recommendations of the standard specification:

mTLS with consumer authentication via the SAN can play an important role in implementing a zero-trust architecture (ZTA). Therefore, this security option should not be lost by perimeter security. For example, if the access token is missing or lacks sufficient information to perform an authorization check, it’s crucial to ascertain the identity of the caller initiating the request. Alternatively, the security level of the called API is of such a high confidentiality class that both authorization concepts (mTLS and access token) are used.

What does this mean for the API gateway? The perimeter security component placed at the network edge can be trusted and the consumer can be identified based on the transmitted SAN header field. However, within the Kong ecosystem there is currently no official and third-party solution to implement consumer identification based on a SAN request header field. Kong has received related demands from customers already and a feature request has been submitted (Kong feature request GTWY-I_524) but as far as I know, it has not yet been implemented. Let us see how we can overcome this limitation.

Solution

Once again, the solution is to develop your own plugin. I have already covered plugin development in my OBO blog, but in this context I would like to point out a new excellent YouTube video on plugin development by Lokesh Chechani. The Kong video contribution “Kong Plugins Unleashed: A Tech Odyssey into API Mastery” is ideal for getting started with plugin development.

The new plugin should be set up and used on top of the existing Kong mutual TLS authentication configuration. It is intended that the new plugin will work hand-in-hand and preceding the mtls-auth plugin provided by Kong out of the box. First the mtls-header plugin will try a consumer authentication, if that fails and further anonymous processing is allowed, then the mtls-auth plugin can optionally try to authenticate from the SSL/TLS-handshake. This means that mTLS can work for both “outside” and “inside” calls to the same service. Since the mtls-auth plugin is an enterprise plugin and the new plugin builds on the existing structures, an enterprise license is also required for this new mtls-header plugin.

This time, the new plugin does not identify the consumer directly on the client certificate taken from the SSL/TLS-handshake, but on the request SAN header, which has been passed on by perimeter security. The plugin should take the SAN from the request header and look for the appropriate subject name within the configured mtls-auth credentials. The SAN request header name used to look up the SAN value should be configurable. If there is a match, the referenced consumer is set. If no matching subject name is found in the configured mTLS credentials, authentication with an anonymous user can be allowed as a configurable fallback option in the plugin configuration. If the anonymous fallback option is not configured and the plugin cannot find a suitable SAN match, the request fails with the authentication failure code 401 (unauthorized). The way it works with the anonymous configuration should be like the other Kong authentication plugins with anonymous processing.

However, it should come as no surprise that I have already written a Lua mtls-header custom plugin. The implemented plugin is designed to process the certification X-Client-Cert request headers created by the aforementioned Google Load Balancer. However, the plugin should also be applicable to other perimeter security solutions or products. As mentioned above, the request header field names are configurable. Otherwise, the source code can be easily customized.

I am happy to share it with you in case you are also suffering from loss of mTLS functionality due to perimeter security. Please see the NTT DATA GitHub repository for the plugin source code here.

Usage

The execution order is determined by the priority of the plugin. The plugin runs in the NGINX access phase with priority 20000, so its execution in the processing pipeline is ahead of the standard authentication plugins.

The mTLS header plugin is easy to configure. Only the mTLS header name for the SAN with DNS names is mandatory, while the anonymous user configuration and the certificate validity period are optional. The plugin assumes that the SAN information comes from the DNS name tag which is used for consumer authentication. The following DB-less configuration snippet shows a running configuration that implements the mtls-header plugin with the anonymous user option allowed and a certificate expiration checking enabled.

plugins: 
  - name: mlts-header 
    tags: ["mtls-header"] 
    service: <configured service entity> 
    enabled: true 
    config: 
      anonymous: anonymous 
      mtls_san_header_name: X-Client-Cert-DNSName-SANs 
      mtls_valid_period_check_enabled: true 
      mtls_valid_nbf_header_name: X-Client-Cert-Valid-Not-Before 
      mtls_valid_naf_header_name: X-Client-Cert-Valid-Not-After 
      stopwatch: false 

The configuration shown above allows consumer authentication based on the SAN DNS name taken from the X-Client-Cert-DNSName-SANs request header field. If no consumer can be identified, the request will proceed as an anonymous user (and other authentication plugins will optionally get a second chance). Additionally, the plugin will check the validity period of the client certificate. The X-Client-Cert-Valid-Not-Before and X-Client-Cert-Valid-Not-After fields specify the bounds of validity.

The mtls-header plugin has its own stopwatch and shows varying runtimes between 1 and 3 milliseconds in my Google Kubernetes Engine (GKE) test environment. Kong itself caches the mtls-auth credentials and consumer entities, hence the good plugin performance.

Conclusion

mTLS rigorously verifies the identity of the communicating parties in a two-way fashion, reducing the risk of unauthorized access and data breaches. Perimeter security combined with mTLS consumer authentication at the gateway improves the overall security posture of the system, making it more resilient to various threats in today’s dynamic and distributed computing environments. Therefore, easy-to-configure and customizable plugins are important for this task. The discussed mtls-header plugin fills a gap in the Kong plugin portfolio and enables mTLS consumer authentication at the Kong Gateway. The plugin is implemented in Lua for stability and performance.

The presented plugin will solve your mTLS problems that may be triggered by perimeter security. Once again, coming back to my opening statement, with the Kong Gateway you will not run into any limitations. The latest Gartner Magic Quadrant for API Management (here) also highlights this strength of Kong: “Kong takes a developer-focused approach by offering an API management solution with high deployment flexibility, programmability and performance”.

Beyond words: Special thanks to Maximilian Rettig for his yielded expertise in perimeter security and GCP load balancer configuration options.

Credits

Title image by Gorodenkoff on Shutterstock