Keywords: Apache Kafka | ZooKeeper | Topic Management
Abstract: This technical paper addresses the challenge of obtaining Kafka topic lists in version 0.10 environments where direct ZooKeeper access is unavailable. Through architectural dependency analysis, it presents a comprehensive solution using embedded ZooKeeper instances, covering service startup, configuration validation, and command execution. The paper also compares topic management approaches across Kafka versions, providing practical guidance for legacy system maintenance and version migration.
Problem Context and Technical Challenges
In typical Apache Kafka 0.10 deployment scenarios, users frequently encounter situations where topic list retrieval is required but direct ZooKeeper service access is unavailable. The conventional kafka-topics.sh --list --zookeeper localhost:2181 command relies on ZooKeeper connectivity, which fails to function properly in certain restricted environments.
Kafka Architecture Dependency Analysis
Apache Kafka, as a distributed streaming platform, heavily depended on ZooKeeper for metadata management and coordination services in its early versions (including 0.10). ZooKeeper maintains critical metadata such as topic information, partition states, and consumer offsets. When users cannot establish ZooKeeper connections, all ZooKeeper-based Kafka administration commands become ineffective.
Solution: Embedded ZooKeeper Instance Deployment
For scenarios lacking independent ZooKeeper access, Kafka provides a convenient built-in solution. By starting the single-node ZooKeeper instance bundled with Kafka, users can quickly establish a complete Kafka runtime environment.
ZooKeeper Service Startup
First, initiate the single-node ZooKeeper instance using the convenience script provided in the Kafka installation package:
bin/zookeeper-server-start.sh config/zookeeper.propertiesThis command reads the default ZooKeeper configuration file and starts the ZooKeeper service listening on port 2181. The startup process outputs log information including data directory initialization, port binding status, and other critical details.
Kafka Server Startup
With the ZooKeeper service running normally, proceed to start the Kafka server process:
bin/kafka-server-start.sh config/server.propertiesDuring startup, the Kafka server automatically connects to the local ZooKeeper instance, completing initialization operations such as broker registration and topic metadata synchronization. Ensure the zookeeper.connect parameter in the server.properties configuration file is set to localhost:2181.
Topic List Retrieval
Once the Kafka cluster is operational, use the standard command to obtain the topic list:
bin/kafka-topics.sh --list --zookeeper localhost:2181This command queries the topic metadata stored in ZooKeeper, returning a list of all available topic names in the current Kafka cluster. The output format presents one topic name per line, facilitating subsequent processing and analysis.
Version Evolution and Alternative Approaches
It is noteworthy that starting from Kafka version 2.2, new parameters based on bootstrap-server were introduced, allowing direct topic information retrieval through Kafka brokers, thereby reducing ZooKeeper dependency:
bin/kafka-topics.sh --list --bootstrap-server <BROKER-LIST>This new approach utilizes Kafka's native protocol for metadata queries, offering improved performance and maintainability. However, for early versions like Kafka 0.10, the aforementioned ZooKeeper-based solution remains necessary.
Implementation Considerations
During actual deployment, several aspects require attention: ensure correct startup sequence for ZooKeeper and Kafka services; verify network connectivity and port availability; monitor service logs to troubleshoot potential issues. For production environments, using independent ZooKeeper clusters rather than single-node instances is recommended to enhance system reliability and scalability.