Filebeat Always Failing to Connect to Elasticsearch

Elastic Stack Beats
1

I am trying to run a simple elastic stack configuration (Filebeat + Elasticsearch + Kibana) on my local machine. Filebeat should take all the docker logs and output them to elasticsearch. I am running everything in docker containers and I start them with a docker-compose.yml file. Every time I run "docker-compose up" filebeat always fails to connect to the elasticsearch host ("Failed to connect to backoff(elasticsearch(http://localhost:9200))"). I have not been able to find any other topics that could help me resolve this. Thanks in advance.

docker-compose.yml

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2
    container_name: elasticsearch
    environment:
      - cluster.name=elastic-cluster
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
      memlock:
        soft: -1
        hard: -1
    volumes:
      - elastic-data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - elastic-net
  kibana:
    depends_on:      
      - "elasticsearch"
    image: docker.elastic.co/kibana/kibana:6.6.2
    container_name: kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
    networks:
      - elastic-net
  filebeat:
    depends_on:
      - "elasticsearch"
    image: test/filebeat:v1
    container_name: filebeat
    volumes:
      - "./filebeat.yml:/usr/share/filebeat/filebeat.yml"
      - "/var/lib/docker/containers:/usr/share/filebeat/dockerlogs:ro"
      - "/var/run/docker.sock:/var/run/docker.sock"
    networks:
      - elastic-net

volumes:
  elastic-data:

networks:
  elastic-net:

filebeat.yml

filebeat.prospectors:
- type: log
  json.keys_under_root: true
# Json key name, which value contains a sub JSON document produced by our application Console Appender  
  json.message_key: log
  enabled: true
  encoding: utf-8
  document_type: docker
  paths:
# Location of all our Docker log files (mapped volume in docker-compose.yml)  
    - '/usr/share/filebeat/dockerlogs/*/*.log'

#setup.template.name: "filebeat-"
#setup.template.pattern: "filebeat-*"
#setup.dashboards.enabled: true
  
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  #index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

setup.kibana:
  host: "http://localhost:5601"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

Dockerfile

`FROM docker.elastic.co/beats/filebeat:6.6.2
# Copy our custom configuration file
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
# Create a directory to map volume with all docker log files
RUN mkdir /usr/share/filebeat/dockerlogs
RUN chown -R root /usr/share/filebeat/
RUN chmod -R go-w /usr/share/filebeat/`
2

Didnt have enough characters to post everything so here are the logs.

filebeat logs:

2019-03-26T14:26:22.170Z	INFO	instance/beat.go:616	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-03-26T14:26:22.171Z	INFO	instance/beat.go:623	Beat UUID: c484865d-e9b2-475c-a5a6-06ada1cfa0aa
2019-03-26T14:26:22.172Z	INFO	[seccomp]	seccomp/seccomp.go:116	Syscall filter successfully installed
2019-03-26T14:26:22.172Z	INFO	[beat]	instance/beat.go:936	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "c484865d-e9b2-475c-a5a6-06ada1cfa0aa"}}}
2019-03-26T14:26:22.172Z	INFO	[beat]	instance/beat.go:945	Build info	{"system_info": {"build": {"commit": "1eea934ce81be553337f2828bd12131896fea8e4", "libbeat": "6.6.2", "time": "2019-03-06T14:17:59.000Z", "version": "6.6.2"}}}
2019-03-26T14:26:22.172Z	INFO	[beat]	instance/beat.go:948	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.10.8"}}}
2019-03-26T14:26:22.173Z	INFO	[beat]	instance/beat.go:952	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-03-26T09:35:34Z","containerized":true,"name":"7d85bed9039b","ip":["127.0.0.1/8","192.168.16.3/20"],"kernel_version":"4.15.0-46-generic","mac":["02:42:c0:a8:10:03"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-03-26T14:26:22.174Z	INFO	[beat]	instance/beat.go:981	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-03-26T14:26:20.759Z"}}}
2019-03-26T14:26:22.174Z	INFO	instance/beat.go:281	Setup Beat: filebeat; Version: 6.6.2
2019-03-26T14:26:22.174Z	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
2019-03-26T14:26:22.175Z	INFO	[publisher]	pipeline/module.go:110	Beat name: 7d85bed9039b
2019-03-26T14:26:22.175Z	WARN	[cfgwarn]	beater/filebeat.go:81	DEPRECATED: prospectors are deprecated, Use `inputs` instead. Will be removed in version: 7.0.0
2019-03-26T14:26:22.175Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2019-03-26T14:26:22.175Z	INFO	instance/beat.go:403	filebeat start running.
2019-03-26T14:26:22.176Z	INFO	registrar/registrar.go:97	No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
2019-03-26T14:26:22.200Z	INFO	registrar/registrar.go:134	Loading registrar data from /usr/share/filebeat/data/registry
2019-03-26T14:26:22.200Z	INFO	registrar/registrar.go:141	States Loaded from registrar: 0
2019-03-26T14:26:22.200Z	INFO	crawler/crawler.go:72	Loading Inputs: 1
2019-03-26T14:26:22.201Z	INFO	log/input.go:138	Configured paths: [/usr/share/filebeat/dockerlogs/*/*.log]
2019-03-26T14:26:22.201Z	INFO	input/input.go:114	Starting input of type: log; ID: 3859208192045851309 
2019-03-26T14:26:22.201Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2019-03-26T14:26:22.202Z	INFO	log/harvester.go:255	Harvester started for file: /usr/share/filebeat/dockerlogs/a06a2b9c3d89e4cce05e9efb4bdf77d0d6bae5ea1c43cf481722cbe8c2b0c294/a06a2b9c3d89e4cce05e9efb4bdf77d0d6bae5ea1c43cf481722cbe8c2b0c294-json.log
2019-03-26T14:26:22.203Z	INFO	log/harvester.go:255	Harvester started for file: /usr/share/filebeat/dockerlogs/cec7229681c9a33b56a1616fcc84da2450887408cc324cc2ad56a8af09d340fe/cec7229681c9a33b56a1616fcc84da2450887408cc324cc2ad56a8af09d340fe-json.log
2019-03-26T14:26:22.206Z	INFO	log/harvester.go:255	Harvester started for file: /usr/share/filebeat/dockerlogs/d962a1d8f39006ae04c1eb15c67a63ad10b2956f1450a099cc5b1010ea45559a/d962a1d8f39006ae04c1eb15c67a63ad10b2956f1450a099cc5b1010ea45559a-json.log
2019-03-26T14:26:22.207Z	INFO	log/harvester.go:255	Harvester started for file: /usr/share/filebeat/dockerlogs/448930d0e8cea972bc4a405c2ff33e9ebce8af999b0f4fd497f13caf7244b321/448930d0e8cea972bc4a405c2ff33e9ebce8af999b0f4fd497f13caf7244b321-json.log
2019-03-26T14:26:22.208Z	INFO	log/harvester.go:255	Harvester started for file: /usr/share/filebeat/dockerlogs/5580609a0627bfdaf2ef3f7d0f9202c63feb345a9b0e6c2a116c2299678f6997/5580609a0627bfdaf2ef3f7d0f9202c63feb345a9b0e6c2a116c2299678f6997-json.log
2019-03-26T14:26:22.208Z	INFO	log/harvester.go:255	Harvester started for file: /usr/share/filebeat/dockerlogs/791540ea12c42907e1487a0d6b9def8560467f0e1968313cd5d97717c6096683/791540ea12c42907e1487a0d6b9def8560467f0e1968313cd5d97717c6096683-json.log
2019-03-26T14:26:22.208Z	INFO	log/harvester.go:255	Harvester started for file: /usr/share/filebeat/dockerlogs/7d85bed9039b09a1a4994925e63724a9f015f104ebc7ae513410b439078db1fb/7d85bed9039b09a1a4994925e63724a9f015f104ebc7ae513410b439078db1fb-json.log
2019-03-26T14:26:22.209Z	INFO	log/harvester.go:255	Harvester started for file: /usr/share/filebeat/dockerlogs/89d08c634ad1bb3bee96f108147dbe8f13f2757708f51bf1cf13aa66d2c196f8/89d08c634ad1bb3bee96f108147dbe8f13f2757708f51bf1cf13aa66d2c196f8-json.log
2019-03-26T14:26:23.207Z	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://localhost:9200))
2019-03-26T14:26:24.872Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2019-03-26T14:26:24.872Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 1 reconnect attempt(s)
2019-03-26T14:26:28.827Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp [::1]:9200: connect: cannot assign requested address
3

This is a Docker compose configuration error. Based on their documentation (https://docs.docker.com/compose/networking/), you can reach elasticsearch if you configure elasticsearch:9200 instead of localhost:9200, if you configure the default network.

3 Likes
4

Thank you very much, it works perfectly now.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.