How To: Kafka 0.9 on Mac

Kafka is a distributed messaging system used for streaming data.  It works as a distributed commit log and if you want to really understand why you should use Kafka then it’s worth the time to read this article by Jay Kreps.  Now if you just want to get hands on with Kafka on your laptop, follow these steps from the quick start guide (which should work on linux also for a sandbox environment).  I didn’t hit errors along the way so this is pretty similar to what is in the documentation but I thought its worth sharing as a reference to the actual commands I used and a place for me to reference when I post more articles about working with Kafka.

  1. Go to http://kafka.apache.org/downloads.html and download the version you want.  I chose kafka_2.11-0.9.0.0.tgz.
  2. Follow instructions here for initial setup: http://kafka.apache.org/documentation.html#quickstart
    1. unzip: tar -xzf kafka_2.11-0.9.0.0.tgz
    2. go to folder: cd kafka_2.11.0.9.0.0
    3. start zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties
    4. open new terminal window and go to folder
    5. start kafka: bin/kafka-server-start.sh config/server.properties
    6. test creating topic: bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
    7. test listing a topic: bin/kafka-topics.sh --list --zookeeper localhost:2181
  3. Follow additional steps to get to multiple brokers since you would never use a single broker setup for a real environment (though for a production cluster there would be some different steps and a server per broker, of course)
    1. copy config: cp config/server.properties config/server-1.properties
    2. edit config/server-1.properties:
      broker.id=1
      listeners=PLAINTEXT://:9093
      log.dir=/tmp/kafka-logs-1
    3. copy config again: cp config/server.properties config/server-2.properties
    4. edit config/server-2.properties:
      broker.id=2
      listeners=PLAINTEXT://:9094
      log.dir=/tmp/kafka-logs-2
    5. keep zookeeper running but stop kafka (CMD+C on the terminal it is running under)
    6. run all 3 brokers as background processes:
      bin/kafka-server-start.sh config/server.properties &
      bin/kafka-server-start.sh config/server-1.properties &
      bin/kafka-server-start.sh config/server-2.properties &
    7. test creating topic with replication factor of 3: bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
    8. might as well publish a message to the topic: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
      {"value": "Test message 1"}
    9. then test out the consumer: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

The quick start guide and additional documentation have a lot more info that is worth exploring, but if things went well you now have a local instance to test with.  Congrats!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s