Apache Kafka ZooKeeper authentication

About

This post will describe how to enable authentication on Apache ZooKeeper and configure Apache Kafka to authenticate to Zookeeper.

Why authenticate with ZooKeeper

Kafka uses ZooKeeper to store meta information, this includes any configured ACLs.

By default Zookeeper is accessible by anyone who can access to the network. This means anyone can :

  • Escalate privileges by manipulating any configured Kafka ACLs
  • Poison the Kafka cluster by maliciously updating the meta information stored in Zookeeper

Enabling authentication on Zookeeper helps prevent the above. It should be noted while we will prevent malicious users from modifying znodes in Zookeeper they are still world readable if you have access to the ZookKeeper service.

Enabling auth on ZooKeeper

For the below example we will use digest authentication, essentially this is a static username and password. We use digest auth as it is the simplest to demonstrate, ideally the Kerberos option would provide a better solution and is a case of adjusting the jaas.conf files appropriately.

The steps should be performed on each Zookeeper instance.

Modify zoo.cfg

Modify the the zoo.cfg file adding the below lines, we will assume this file is at /etc/zookeeper/zoo.cfg:

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

Note authProvider.1 is used as multiple authentication providers can be enabled, this would look like authProvider.2 =....

Create a jaas.conf file

Create the file /etc/zookeeper/jaas.conf with the below replacing the password values with your passwords:

Server {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  user_admin="password"
  user_kafka="password";
};

Note the semicolons, these are important, the last item in the server block must have a colon, the other lines must not.

Set jaas.conf permissions

The jaas.conf file contains passwords so we restrict who can access the file. The below assumes ZooKeeper has it’s own system user and group called zookeeper.

sudo chown root:zookeeper /etc/zookeeper/jaas.conf
sudo chmod 640 /etc/zookeeper/jaas.conf

Update the ZooKeeper JVM flags

This will vary depending on how you run Zookeeper. For this example we will assume you have the file /etc/sysconfig/zookeeoer:

JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/jaas.conf -Dzookeeper.allowSaslFailedClients=false"

The file may be different depending on how you install Zookeeper, JVMFLAGS may also be different such as JVM_OPTS instead. Please check your ZooKeeper startup script.

Restart ZooKeeper

Restart the ZooKeeper instance

Enabling Kafka ZooKeeper auth

The below steps are performed on the Kafka brokers. This is a multi staged process, we need to:

  • Enable ZooKeeper authentication on all Kafka brokers and restart
  • Run the zookeeper-security-migration script
  • Force Kafka to set ACLs for ZooKeeper and restart all brokers

Create Kafka to Zookeper Authentication jaas.conf

Create a file at /etc/kafka/jaas.conf with the below, update the password value with the one you set in the ZooKeeper config:

ZkClient {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  username="kafka"
  password=password";
};

Set the jaas.conf permissions

Like for ZooKeeper as the jaas.conf here contains a password we restrict who can access the file. The below is based on the assumption Kafka has it’s own system user and group:

sudo chown root:kafka /etc/kafka/jaas.conf
sudo chown 640 /etc/kafka/jaas.conf

Update Kafka JVM options

This will vary depending on how you have installed Kafka. The below assumes you have the file /etc/sysconfig/kafka file:

KAFKA_JVM_PERFORMANCE_OPTS=... -Dzookeeper.sasl.client=true -Dzookeeper.sasl.clientconfig=ZkClient -Dzookeeper.sasl.client.username=kafka -Djava.security.auth.login.config=/etc/kafka/jaas.conf

Note ... is any existing set options

Restart the brokers

After performing the above restart the broker.

Perform ACL migration

This step assumes you have applied the above config and restarted all brokers. Kafka comes with the tool zookeeper-security-migration.sh which updates the ACL’s in ZooKeeper for any existing Kafka data.

The below assumes your Kafka bin directort is at /opt/kafka/bin, update the zookeeper.connect to match the hostname of your ZooKeeper instance:

sudo KAFKA_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.clientconfig=ZkClient -Dzookeeper.sasl.client.username=kafka -Djava.security.auth.login.config=/etc/kafka/jaas.conf" \
  /opt/kafka/bin/zookeeper-security-migration.sh --zookeeper.acl secure --zookeeper.connect zookeeper00-internal.prod

Note: we use sudo here so we can read the /etc/kafka/jaas.conf file.

Configure Kafka to use ZooKeeper ACL’s

Modify the Kafka configuration adding the below option. The example assumes you have a file /etc/kafka/server.properties:

zookeeper.set.acl=true

Restart the broker

Restart the broker for the configuration change to take affect. After completion apply the change to the rest of the brokers

Verify ZooKeeper data

Using the ZooKeeper cli you can verify the data now has ACL’s set:

[zk: localhost:2181(CONNECTED) 5] getAcl /config
'world,'anyone
: r
'digest,'kafka
: cdrwa

Summary

Kafka should now authenticate when connecting to ZooKeeper.

Any data Kafka saves to ZooKeeper will be only modifiable by the kafka user in ZooKeeper.

The zookeeper-security-migration script does not remove the world readable for Kafka data. Likewise when enabling authentication on ZooKeeper anonymous users can still connect and view any data not protected by ACL’s. With this and the recommended ZooKeeper of 3.4.x not supporting SSL the Kafka/ZooKeeper security story isn’t great but we can protect around data poisoning. Run your ZooKeeper cluster in a private trusted network.

Notes

The paths and configuration used here are based on the Kafka and ZooKeeper RPM’s built from the scripts at: