We are trying to harvest csv data using filebeat to Elasticsearch. Data pipeline is set up to stream to Kafka -> Logstash -> Elasticsearch.
We have a hybrid cloud setup of corporate cloud data center and aws. filebeat is installed in corporate data center hosts where as Kafka, logstash and elasticsearch servers are in AWS VPC.
Kafka ip address (aws) is 10.x.x.x which is directly reachable from corporate data center hosts. 10.x.x.x IP address is configured in filebeat.yml. Here is the config file.
filebeat.registry_file: /var/lib/filebeat/registry filebeat.shutdown_timeout: 30s filebeat.prospectors: - input_type: log paths: - /home/svcload3/s4/logs/perf-UI*.csv scan_frequency: 5s fields: log_type: perflogs service: load3 product: s4 exclude_lines: ['SourcePage'] output.kafka: hosts: ["10.x.x.x:9092", "10.x.x.y:9092", "10.x.x.z:9092"] topic: perflogs processors: - add_cloud_metadata:
In the filebeat logs, we observed that it is trying to connect to the internal dns address of kafka server which is not reachable from corporate data center hosts. How do we resolve this issue?
2018-01-13T05:31:21-08:00 INFO Failed to connect to broker [[ip-10-x-x-x.us-west-1.compute.internal:9092 dial tcp: lookup ip-10-169-48-77.us-west-1.compute.internal on 10.x.z.z:53: no such host]]: %!s(MISSING)