Month: August 2016

How To: Kafka 0.9 on Mac

Kafka is a distributed messaging system used for streaming data.  It works as a distributed commit log and if you want to really understand why you should use Kafka then it’s worth the time to read this article by Jay Kreps.  Now if you just want to get hands on with Kafka on your laptop, follow these steps from the quick start guide (which should work on linux also for a sandbox environment).  I didn’t hit errors along the way so this is pretty similar to what is in the documentation but I thought its worth sharing as a reference to the actual commands I used and a place for me to reference when I post more articles about working with Kafka.

  1. Go to and download the version you want.  I chose kafka_2.11-
  2. Follow instructions here for initial setup:
    1. unzip: tar -xzf kafka_2.11-
    2. go to folder: cd kafka_2.
    3. start zookeeper: bin/ config/
    4. open new terminal window and go to folder
    5. start kafka: bin/ config/
    6. test creating topic: bin/ --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
    7. test listing a topic: bin/ --list --zookeeper localhost:2181
  3. Follow additional steps to get to multiple brokers since you would never use a single broker setup for a real environment (though for a production cluster there would be some different steps and a server per broker, of course)
    1. copy config: cp config/ config/
    2. edit config/
    3. copy config again: cp config/ config/
    4. edit config/
    5. keep zookeeper running but stop kafka (CMD+C on the terminal it is running under)
    6. run all 3 brokers as background processes:
      bin/ config/ &
      bin/ config/ &
      bin/ config/ &
    7. test creating topic with replication factor of 3: bin/ --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
    8. might as well publish a message to the topic: bin/ --broker-list localhost:9092 --topic test
      {"value": "Test message 1"}
    9. then test out the consumer: bin/ --zookeeper localhost:2181 --topic test --from-beginning

The quick start guide and additional documentation have a lot more info that is worth exploring, but if things went well you now have a local instance to test with.  Congrats!