Home / server / Questions / 1088082
Accepted
Cabbage Parachute
Asked: 2021-12-24 06:04:54 +0800 CST

Kafka stores log4j logs in directory literally called ${kafka.logs.dir}

  • 772

I installed a preview version of Kafka 3.0 with log4j2 support (http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/) on a RHEL 8 server. Kafka and Zookeeper are running successfully as systemd user services. I set the environment variable KAFKA_LOG4J_OPTS="-Dlog4j.configurationFile=file:/home/username/kafka/bin/../config/log4j2.properties" in the systemd unit file so that log4j 2.17 ist used.

However, there is one strange thing: all log4j logs are stored in a directory literally called ${kafka.logs.dir} in the home directory. The garbage collector successfully stores logs in the correct log location ~/kafka/logs. So the directory ~/kafka/logs contains files like kafkaServer-gc.log.0.current, while the hilariously strange directory ~/${kafka.logs.dir} contains server.log, controller.log, etc.

When looking at the processes for kafka and zookeeper, I can see that they both have the argument -Dkafka.logs.dir=/home/username/kafka/bin/../logs. Shouldn't this define the environment variable kafka.logs.dir, which is used in the file config/log4j2.properties? How come that Zookeeper and Kafka apparently don't have access to this $kafka.logs.dir environment variable?

java systemd log4j kafka rhel8

2 Answers

  1. Best Answer

    Just add sys: prefix to the kafka.logs.dir system variable in the log4j2.properties file. That is:

    • AS-IS: ${kafka.logs.dir}
    • TO-BE: ${sys:kafka.logs.dir}

    DISCLAIMER: As you can see in my name, I wrote this code (log4j2 migration of Apache Kafka) and am maintaining the preview build. The fix will be released soon. Sorry for being late.

    • 2

  2. I have followed the steps and am able to upgrade the log4j to logj2, however, I'm having issues with controller.log which is not getting appended. Could you please help?

    • 0