Apache Kafka ZooKeeper authentication
About
This post will describe how to enable authentication on Apache ZooKeeper and configure Apache Kafka to authenticate to Zookeeper.
Why authenticate with ZooKeeper
Kafka uses ZooKeeper to store meta information, this includes any configured ACLs.
By default Zookeeper is accessible by anyone who can access to the network. This means anyone can :
- Escalate privileges by manipulating any configured Kafka ACLs
- Poison the Kafka cluster by maliciously updating the meta information stored in Zookeeper
Enabling authentication on Zookeeper helps prevent the above. It should be noted while we will prevent malicious users from modifying znodes in Zookeeper they are still world readable if you have access to the ZookKeeper service.
Enabling auth on ZooKeeper
For the below example we will use digest authentication, essentially this is a static username and password. We use digest auth as it is the simplest to demonstrate, ideally the Kerberos option would provide a better solution and is a case of adjusting the jaas.conf
files appropriately.
The steps should be performed on each Zookeeper instance.
Modify zoo.cfg
Modify the the zoo.cfg
file adding the below lines, we will assume this file is at /etc/zookeeper/zoo.cfg
:
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000
Note authProvider.1
is used as multiple authentication providers can be enabled, this would look like authProvider.2 =...
.
Create a jaas.conf file
Create the file /etc/zookeeper/jaas.conf
with the below replacing the password values with your passwords:
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_admin="password"
user_kafka="password";
};
Note the semicolons, these are important, the last item in the server block must have a colon, the other lines must not.
Set jaas.conf permissions
The jaas.conf
file contains passwords so we restrict who can access the file. The below assumes ZooKeeper has it’s own system user and group called zookeeper
.
sudo chown root:zookeeper /etc/zookeeper/jaas.conf
sudo chmod 640 /etc/zookeeper/jaas.conf
Update the ZooKeeper JVM flags
This will vary depending on how you run Zookeeper. For this example we will assume you have the file /etc/sysconfig/zookeeoer
:
JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/jaas.conf -Dzookeeper.allowSaslFailedClients=false"
The file may be different depending on how you install Zookeeper, JVMFLAGS
may also be different such as JVM_OPTS
instead. Please check your ZooKeeper startup script.
Restart ZooKeeper
Restart the ZooKeeper instance
Enabling Kafka ZooKeeper auth
The below steps are performed on the Kafka brokers. This is a multi staged process, we need to:
- Enable ZooKeeper authentication on all Kafka brokers and restart
- Run the
zookeeper-security-migration
script - Force Kafka to set ACLs for ZooKeeper and restart all brokers
Create Kafka to Zookeper Authentication jaas.conf
Create a file at /etc/kafka/jaas.conf
with the below, update the password
value with the one you set in the ZooKeeper config:
ZkClient {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password=“password";
};
Set the jaas.conf permissions
Like for ZooKeeper as the jaas.conf here contains a password we restrict who can access the file. The below is based on the assumption Kafka has it’s own system user and group:
sudo chown root:kafka /etc/kafka/jaas.conf
sudo chown 640 /etc/kafka/jaas.conf
Update Kafka JVM options
This will vary depending on how you have installed Kafka. The below assumes you have the file /etc/sysconfig/kafka
file:
KAFKA_JVM_PERFORMANCE_OPTS=... -Dzookeeper.sasl.client=true -Dzookeeper.sasl.clientconfig=ZkClient -Dzookeeper.sasl.client.username=kafka -Djava.security.auth.login.config=/etc/kafka/jaas.conf
Note ...
is any existing set options
Restart the brokers
After performing the above restart the broker.
Perform ACL migration
This step assumes you have applied the above config and restarted all brokers. Kafka comes with the tool zookeeper-security-migration.sh
which updates the ACL’s in ZooKeeper for any existing Kafka data.
The below assumes your Kafka bin directort is at /opt/kafka/bin
, update the zookeeper.connect
to match the hostname of your ZooKeeper instance:
sudo KAFKA_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.clientconfig=ZkClient -Dzookeeper.sasl.client.username=kafka -Djava.security.auth.login.config=/etc/kafka/jaas.conf" \
/opt/kafka/bin/zookeeper-security-migration.sh --zookeeper.acl secure --zookeeper.connect zookeeper00-internal.prod
Note: we use sudo here so we can read the /etc/kafka/jaas.conf
file.
Configure Kafka to use ZooKeeper ACL’s
Modify the Kafka configuration adding the below option. The example assumes you have a file /etc/kafka/server.properties
:
zookeeper.set.acl=true
Restart the broker
Restart the broker for the configuration change to take affect. After completion apply the change to the rest of the brokers
Verify ZooKeeper data
Using the ZooKeeper cli you can verify the data now has ACL’s set:
[zk: localhost:2181(CONNECTED) 5] getAcl /config
'world,'anyone
: r
'digest,'kafka
: cdrwa
Summary
Kafka should now authenticate when connecting to ZooKeeper.
Any data Kafka saves to ZooKeeper will be only modifiable by the kafka
user in ZooKeeper.
The zookeeper-security-migration
script does not remove the world readable for Kafka data. Likewise when enabling authentication on ZooKeeper anonymous users can still connect and view any data not protected by ACL’s. With this and the recommended ZooKeeper of 3.4.x
not supporting SSL the Kafka/ZooKeeper security story isn’t great but we can protect around data poisoning. Run your ZooKeeper cluster in a private trusted network.
Notes
The paths and configuration used here are based on the Kafka and ZooKeeper RPM’s built from the scripts at: