I'm having a difficult time troubleshooting, what should be a simple kafka-issue.
I'm trying to publish a few messages from
kafak-console-producer. When I type 'hi' I get:
/opt/kafka/bin/kafka-console-producer.sh --topic test --broker-list 172.17.0.21:9092 hi [2016-01-25 12:56:19,839] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: Batch Expired (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
I don't seem to be able to find any kafka-log4j-logs...
I'm running the kafka server the way it is described in the quick-start
I can create the topic & describe it.
One note which maybe important is that the kafka & zookeeper are running in a docker-container (
172.17.0.21) I'm creating the topic and sending messages from the host.
Please note the both the (kafka-server) 9092 & (zk) 2181 are open and accessible from the host, I've confirmed this using netstat & telnet.dockerapache-kafka
Turned out to be more of a Docker issue...
OK, when running the client
kafka-console-consumer.sh I saw that it was trying to connect to the broker using the container's name (which is the first few characters of a GUID).
My host cannot resolve connecting to docker-container using host-name.
Since I had already mapped the ports to my host, I changed the kafak-server config's
advertised.host.name to point to my host's IP address.
That way whenever zk is queried for the broker my host's IP address is returned, then through port forwarding I'm connected to the broker in docker.
BTW, you know that you can set that variable from the
docker run command, right?
E.g. this is how I start Kafka & ZK in a single container, for my local development environment:
docker run --name st-kafka -p 2181:2181 -p 9092:9092 -e ADVERTISED_HOST=`docker-machine ip machine_name` -e ADVERTISED_PORT=9092 -d spotify/kafka
Then, to create a topic:
docker exec -ti st-kafka bash -c "/opt/kafka_*/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 4 --topic topic_name"
That's on OS X, which is why I use
docker-machine ip MACHINE_NAME.
As you can see, that's Spotify's image. Although there was no
v0.9 last time I checked, it's trivial to fork the
Dockerfile and adjust for
I am trying with kafka 0.10.0.0 in docker and found below
advertised.host.name DEPRECATED: only used when
listenersare not set. Use
advertised.listenersinstead. Hostname to publish to ZooKeeper for clients to use. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, it will use the value for
host.nameif configured. Otherwise it will use the value returned from java.net.InetAddress.getCanonicalHostName().
so set advertised.listeners=PLAINTEXT://yourhost:9092 will work