Is there other web pages that have tutorial on how to operate Filebeat on Windows with Kafka output?
The official document is very confusing and useless
The Getting Started Guide has steps for Windows installation. I suggest getting Filebeat up and running with the Elasticsearch output first.
Then once you have it working. Remove the output.elasticsearch and setup output.kafka as per the kafka output docs.
Runs on ES.
I kept getting the following entries in the logs
2017-01-04T15:20:43+08:00 WARN producer/broker/2 starting up
2017-01-04T15:20:43+08:00 WARN producer/broker/2 state change to [open] on test-beat/0
2017-01-04T15:20:43+08:00 WARN producer/leader/test-beat/0 selected broker 2
2017-01-04T15:20:43+08:00 WARN producer/broker/2 maximum request accumulated, waiting for space
2017-01-04T15:20:45+08:00 WARN Failed to connect to broker host-s2:9092: dial tcp: lookup host-s2: getaddrinfow: No such host is known.
2017-01-04T15:20:45+08:00 WARN producer/broker/2 state change to [closing] because dial tcp: lookup host-s2: getaddrinfow: No such host is known.
2017-01-04T15:20:45+08:00 WARN producer/broker/2 state change to [closing] because dial tcp: lookup host-s2: getaddrinfow: No such host is known.
2017-01-04T15:20:45+08:00 WARN producer/leader/test-beat/0 state change to [flushing-23]
2017-01-04T15:20:45+08:00 WARN producer/leader/test-beat/0 state change to [normal]
2017-01-04T15:20:45+08:00 WARN producer/leader/test-beat/0 state change to [retrying-24]
2017-01-04T15:20:45+08:00 WARN producer/leader/test-beat/0 abandoning broker 2
2017-01-04T15:20:45+08:00 WARN producer/broker/2 shut down
It looks like a broker is unavailable. The output will block if one broker is unavailable. See reachable_only in the kafka docs. It allows you to change this behavior but it's not recommended. It would be better to fix the unreachable broker.
I have specified of the ip address in the yml file of the filebeat. This warning is telling me the name of the host's ip address is host-s2. This makes me puzzled. How can no such host is known when it has it hostname from ip address
I have tried with 2 Kafka brokers and 1 Kafka broker. I keep getting the same error
Did a google on
reachable only kafka
Nothing useful came out
Tried different versions. 0.10, 0.9. just cannot get the filebeat to work kafka. It's alway the same error.
We concluded that filebeat 5.1.1 cannot work with Kafka on our system. We will only revisit this software when 3rd party documentation is available.
@gringo to understand the error, you have to know some details about connection bootstrapping in kafka. As kafka does not use a simple server-client architecture, but provides a full cluster, the client somehow needs to get to know about the actual cluster in order to use/connect to all Leaders (kinda Master-nodes). This process is called bootstrapping. The ips you configure in the kafka output are used for bootstrapping.
In the bootstrapping process the client fetches a cluster-state (meta-data) with all brokers known to the cluster. These meta-data do contain all kafka broker addresses (host/ip, port) + topics and partitions each broker is assigned to. The host/ip and port number returned by the broker is the advertised host (configurable in kafka server itself). Once a broker joins a cluster, it sends (advertises) a host-port-pair to be used to reach the broker. It's important for the host-port-pair advertised to be reachable (this is a common error ).
Your broker advertises host-s2. For kafka to operate correctly, one must ensure the advertised host being correct and reachable from every producer/consumer/broker in the cluster.
This topic was automatically closed after 21 days. New replies are no longer allowed.