Decrypt TLS traffic to Kafka using Wireshark

Usually, debugging issues related to TLS in a Java application involves setting the debug flag -Djavax.net.debug=ALL. Unfortunately, the logs will then be flooded with debug entries, which makes troubleshooting the TLS communication difficult. Moreover, sometimes the problem can be solved just by decrypting the TLS traffic.

One way to achieve this is to use jSSLKeyLog , which is a Java Agent Library to log TLS session keys to a file that can be loaded into the Wireshark network protocol analyzer. Coupled with the recently improved Kafka Protocol support in Wireshark, this makes decrypting TLS traffic to/from Kafka captured using the Tcpdump packet sniffer easy.

Procedure

Instrument the Kafka Broker and/or Kafka Client with the jSSLKeyLog agent by adding the JVM flag

1-javaagent:/path/to/jSSLKeyLog.jar==/path/to/ssl-key.log

when starting the respective process. The Kafka Broker can be easily instrumented by appending the flag to the KAFKA_OPTS Environment Variable.

Then, use the Tcpdump packet sniffer to capture the traffic to/from the Kafka Broker SSL listener, for example by executing the command

1sudo tcpdump -i eth0 -N -A 'port 9093' -w /path/to/trace.pcap

on the machine running the Kafka Broker (adjust the interface, port or path to the packet capture file accordingly).

After the capture finished, open the packet capture file trace.pcap using Wireshark. Without configuring the session key log, the TLS payload will be displayed as “Application Data”.

Next, configure the location of the session key log generated by jSSLKeyLog by choosing Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename in Wireshark.

This allows us to inspect the decrypted TLS payload. To also take advantage of the Kafka Decoder in Wireshark, select Analyze > Decode As > Kafka as the dissection protocol.

As a result, Wireshark will display the decoded Kafka API Request/Response pairs.

Demo

The accompanying GitHub repository provides a quick way to test the procedure described above using Docker Compose.

Requirements

The demo uses versions of Wireshark and kafkacat not (yet) available in the standard package repositories.

• the attached Makefile will build several Docker containers with the needed versions by downloading the source code and compiling the required binaries.
• the jSSLKeyLog Java Agent will also be built from source and the artifact saved in the tools directory. The directory will then be mounted inside the Docker container running the Kafka Broker and the agent attached via

  1KAFKA_OPTS: "-javaagent:/tmp/tools/jSSLKeyLog.jar==/tmp/ssl-key-log/output.log"

To build all the required components, simply run

1make build-all

inside the project directory.

Warning: building the required Docker containers requires some time as Wireshark in particular is a complex application.

Running the demo

• clone the repository

  1git clone https://github.com/tudrescu/kafka-wireshark-debug-workshop.git
  2cd kafka-wireshark-debug-workshop

• bootstrap the environment

  1make build-all

• create a local CA, Server and Client certificates

  1make certs

• launch the Docker Compose stack

  1docker-compose up -d

• start a Docker container running tcpdump and attach to the container running the Kafka Broker, see also using tcpdump with Docker .

  1docker run \
  2  --tty \
  3  --net=container:kafka-1 \
  4  -v ${PWD}/work/tcpdump-trace:/tmp/tcpdump-trace \
  5  docker-tcpdump:latest \
  6  tcpdump -N -A 'port 9093' -w /tmp/tcpdump-trace/trace.pcap

• enter the kafkacat Docker container and produce some messages

  1docker-compose exec kafkacat bash
  2kafkacat -b kafka-1:9093 -L -F /tmp/certs/kafkacat.conf
  3kafkacat -b kafka-1:9093 -t new_topic -P -F /tmp/certs/kafkacat.conf

• stop the packet capture using tcpdump
• open the packet capture using Wireshark. For convenience, the Docker stack provides a containerized Wireshark which is made available in the browser using Xpra . Access the Wireshark via the browser at

  1https://localhost:14500/?username=wireshark&password=wireshark

• configure Wireshark to decode the Kafka TLS traffic as detailed above . The session key log and the packet capture are saved in the ${PWD}\work\ssl-key-log and ${PWD}\work\tcpdump-trace directories, respectively. The directories are mounted in read-only mode inside the Wireshark container under the /home/wireshark/work path.
• stop and destroy the stack when finished

  1docker-compose down -v

Summary

With this approach, we can easily capture and decode TLS traffic to troubleshoot Kafka related issues by inspecting the Kafka protocol requests and responses. The improved support for the dissection of the Kafka protocol available in the latest Wireshark releases might also prove valuable when trying to understand how Kafka works internally.

References

• Kafka protocol guide
• Kafka 2.3+ dissesction support in Wireshark
• Bugzilla ticket with sample captures for Kafka 2.3+
• Practical Packet Analysis, 3rd Edition
• Using tcpdump with Docker
• Wireshark Web Container Image