Hi Filebeat Community,
I'm stuck in a problem in which I couldn't make further progress because of the lack of error or warning logs. Let me express my situation clearly.
****************** START of Working Case **********************
In my Ubuntu 18.04.2 LTS system I have an autodiscover config for filebeat which writes docker logs of selected containers to elasticsearch container.
filebeat.yml
filebeat.autodiscover:
providers:
- type: docker
templates:
- condition:
contains.docker.container.name: 'tomcat'
config:
- type: log
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
fields:
environment: Development
fields_under_root: true
json.message_key: log
json.keys_under_root: false
multiline.pattern: '^[[:space:]]'
multiline.pattern: '^[[:space:]]+|^Caused by:'
multiline.negate: false
multiline.match: after
processors:
- add_docker_metadata: ~
logging.json: false
output.elasticsearch:
hosts: ["my-elasticsearch:9200"]
docker-compose.yml:
version: '3'
services:
my-elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.4.2
container_name: my-elasticsearch
environment:
- node.name=es-node
- cluster.name=docker-cluster
#For Development mode, set transport.host and http.host here
- transport.host=127.0.0.1
- http.host=0.0.0.0
- network.host=0.0.0.0
- network.bind_host=0.0.0.0
- http.port=9200-9300
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.monitoring.enabled=false
- xpack.security.enabled=false
networks:
- mynet
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200:9200
- 9300:9300
my-filebeat:
image: docker.elastic.co/beats/filebeat:6.4.2
container_name: my-filebeat
user: root
networks:
- mynet
command: filebeat -e -strict.perms=false
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/containers/:/var/lib/docker/containers/:ro
- ./test/beats/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
depends_on:
- my-elasticsearch
links:
- my-elasticsearch
It works fine in this system and delivers at the end of the logs:
2019-03-07T09:02:30.818Z INFO input/input.go:114 Starting input of type: log; ID: 7224347413328794170
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Connection attempt Logs START !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-03-07T09:02:30.818Z INFO log/harvester.go:251 Harvester started for file: /var/lib/docker/containers/30812207a78bfdc7cc2f4fbb4a78365b183c115cd25e4fd09588d4c3dc3f3844/30812207a78bfdc7cc2f4fbb4a78365b183c115cd25e4fd09588d4c3dc3f3844-json.log
2019-03-07T09:02:30.818Z INFO log/harvester.go:251 Harvester started for file: /var/lib/docker/containers/86f7c3fb22f390ef7fd9795171a0bc15f48ff2366926573e5d6cc0cd4c2ef53e/86f7c3fb22f390ef7fd9795171a0bc15f48ff2366926573e5d6cc0cd4c2ef53e-json.log
|2019-03-07T09:02:31.818Z|INFO|pipeline/output.go:95|Connecting to backoff(elasticsearch(http://my-elasticsearch:9200))|
|---|---|---|---|
|2019-03-07T09:02:32.819Z|ERROR|pipeline/output.go:100|Failed to connect to backoff(elasticsearch(http://my-elasticsearch:9200)): Get http://my-elasticsearch:9200: dial tcp 172.18.0.7:9200: connect: connection refused|
|2019-03-07T09:02:32.819Z|INFO|pipeline/output.go:93|Attempting to reconnect to backoff(elasticsearch(http://my-elasticsearch:9200)) with 1 reconnect attempt(s)|
|2019-03-07T09:02:34.820Z|ERROR|pipeline/output.go:100|Failed to connect to backoff(elasticsearch(http://my-elasticsearch:9200)): Get http://my-elasticsearch:9200: dial tcp 172.18.0.7:9200: connect: connection refused|
|2019-03-07T09:02:34.820Z|INFO|pipeline/output.go:93|Attempting to reconnect to backoff(elasticsearch(http://my-elasticsearch:9200)) with 2 reconnect attempt(s)|
|2019-03-07T09:02:38.821Z|ERROR|pipeline/output.go:100|Failed to connect to backoff(elasticsearch(http://my-elasticsearch:9200)): Get http://my-elasticsearch:9200: dial tcp 172.18.0.7:9200: connect: connection refused|
|2019-03-07T09:02:38.821Z|INFO|pipeline/output.go:93|Attempting to reconnect to backoff(elasticsearch(http://my-elasticsearch:9200)) with 3 reconnect attempt(s)|
|2019-03-07T09:02:38.877Z|INFO|elasticsearch/client.go:712|Connected to Elasticsearch version 6.4.2|
|2019-03-07T09:02:38.886Z|INFO|template/load.go:82|Loading template for Elasticsearch version: 6.4.2|
|2019-03-07T09:02:39.074Z|INFO|template/load.go:145|Elasticsearch template with name 'filebeat-6.4.2' loaded|
|2019-03-07T09:02:39.074Z|INFO|pipeline/output.go:105|Connection to backoff(elasticsearch(http://my-elasticsearch:9200)) established|
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Connection attempt Logs END !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-03-07T09:03:00.806Z INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s
And curl 'localhost:9200/_cat/indices?v' shows me a newly created index with filebeat- prefix.
****************** END of Working Case **********************
****************** START of Failed Case **********************
In my Ubuntu 18.04.1 LTS system, I did the same settings, but elasticsearch is not showing any index and also filebeat logs does not show any connection attempts at all:
2019-03-07T08:12:10.983Z INFO input/input.go:114 Starting input of type: log; ID: 15949815225808049576
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Connection attempt Logs START !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Connection attempt Logs END !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-03-07T08:12:40.959Z INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s
****************** END of Failed Case **********************
Can someone tell me why I dont get Harvesting or Connection attempt logs and also cannot able to harvest or connect with same setting on different system?
Thanks,
Sudeep