In the last blog, I briefly introduced Kong as a sophisticated API gateway product. This time, I will discuss a problem related to logging message size limitation and offer a solution. This problem significantly limits root cause analysis and ultimately solution development.

Overview

My last blog ended with the statement that Kong is the API gateway without limits. In this technical blog, I would like to use an example from the important area of logging to illustrate this “without limits” statement. Logs are one of the three pillars of observability, along with monitoring and tracing. In particular, Kong as an intermediary primarily records events and/or message information between different parties, which makes gateway logging so important. Logs can provide fine-grained exchange data that can be used to investigate error situations, problematic activity, or behavior changes. Unfortunately, the gateway architecture imposes a per-entry limit on the number of characters that can be logged due to the underlying Nginx/OpenResty layers. Below we will see how we can overcome this limitation.

Problem

Have you ever wondered why Kong log messages sometimes appear truncated in the logging system? This is not a problem with the log shipper or the central logging system of your choice, it is a problem with the Kong architecture and the underlying components on which the gateway is based. Kong is built on top of Nginx using the OpenResty framework to provide API gateway functionality. However, NGINX has a hardcoded 2048 byte limit on error message length, which affects the Kong Plugin Development Kit (PDK) kong.log(…) and kong.logLEVEL(…) operations. This limit includes all prefix and suffix information added by Kong. Fortunately, the OpenResty intermediary layer patches the Nginx source code to use 4096 bytes as the maximum error size. But that is it, longer messages will be truncated. Especially with debug logging, there are sometimes large blocks of information that need to be parsed. Kong is aware of this problem and has published a knowledge base article about this logging length issue (KB article number: 000001781).

Solution

Kong’s KB article ends with the following suggestion: “If you are logging very large lines, consider altering your calls to logging and splitting long lines into multiple shorter lines”. This is where we start talking about log chunking, which is a method of breaking large message strings into smaller pieces or “chunks” of information. Instead of implementing length checking and - if necessary - message chunking in each serverless function or plugin, a global utility solution should be sought. Therefore, the best solution is a global Lua utility module that can be integrated into both serverless functions and plugins. This module can check if chunking is needed at all. If so, it will split a too-long message into several chunks and output the chunks with the appropriate meta-information, also taking into account the logging level. It should come as no surprise that I have already written a Lua utility logging module as described. I use it for potentially overly long log output for my own serverless functions and custom plugins. I am happy to share it with you in case you also suffer from logging information loss. Feel free to check out the NTT DATA GitHub repository here.

Usage

After the chunk utility module has been copied to the gateway folder (e.g. /usr/local/kong/modules), it must be registered in the package path of the kong.conf configuration file. To do this, you need to extend the LUA_PACKAGE_PATH, which can be easily done using the environment variable mechanism with a kong prefix, e.g.

KONG_LUA_PACKAGE_PATH = /usr/local/kong/modules/?.lua

If the sandbox security is enabled, you still need to allow the module code to run together with the used Penlight string library, which is already part of the Kong distribution, e.g.

KONG_UNSTRUSTED_LUA_SANDBOX_REQUIRES = pl.stringx, kong.modules.chunk_utils

I found the Penlight library to be a great help in speeding up Lua development. After the configuration steps, the chunker utility module can be used in both serverless functions and plugins.

Example of using the chunker module in a serverless function:

1return function()
2  --[[ logging --]]
3 
4  local chunk_utils = require 'kong.modules.chunk_utils'
5  local chunker = chunk_utils.chunker
6 
7  local authorization = kong.request.get_header('Authorization')
8  chunker.logChunks(kong.log.debug, 'authorization header: ', authorization)
9end

Whenever an overly long logging message is chunked, it is output with chunk meta information.

[<chunk id>, <chunk index>, <chunks> <chunk size>]

With this meta information, the associated log chunks can also be automatically searched and reassembled (if necessary).

Conclusion

The last thing you need when analyzing log messages is truncated information that prevents you from finding the root cause and ultimately solving the problem. The introduced module will solve your log length problems once and for all. Coming back to my opening statement, with Kong Gateway you will not run into any limitations.

Credits

Title image by BEST-BACKGROUNDS on Shutterstock