Shiping logs from kubernetes to elasticsearch using beats - Failed to connect: Get http://xxxxxx.us-central1.gcp.cloud.es.io:9243: read tcp 13.17.4.10:40766->31.177.77.2:9243: read: connection reset by peer

dinaWork · December 18, 2017, 1:23pm

following

beats fails to ship logs
get the following log:

ERR Failed to connect: Get http://xxxxxx.us-central1.gcp.cloud.es.io:9243: read tcp 13.17.4.10:40766->31.177.77.2:9243: read: connection reset by peer
11:38:19.230  reload.go:222: INFO Dynamic config reloader stopped
11:38:19.230  crawler.go:135: INFO Crawler stopped
11:38:19.230  registrar.go:210: INFO Stopping Registrar
11:38:19.230  registrar.go:165: INFO Ending Registrar
11:38:19.231  forwarder.go:35: INFO Prospector outlet closed
11:38:19.234  metrics.go:51: INFO Total non-zero values: beat.info.uptime.ms=40449 beat.memstats.gc_next=25194960 beat.memstats.memory_alloc=13894472 beat.memstats.memory_total=43826552 filebeat.events.active=4130 filebeat.events.added=4140 filebeat.events.done=10 filebeat.harvester.closed=4 filebeat.harvester.open_files=16 filebeat.harvester.running=16 filebeat.harvester.started=20 libbeat.config.module.running=1 libbeat.config.module.starts=1 libbeat.config.reloads=2 libbeat.output.read.errors=6 libbeat.output.type=elasticsearch libbeat.output.write.bytes=1434 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=4116 libbeat.pipeline.events.failed=4 libbeat.pipeline.events.filtered=20 libbeat.pipeline.events.published=4116 libbeat.pipeline.events.retry=150 libbeat.pipeline.events.total=4140 registrar.states.current=6 registrar.states.update=6 registrar.writes=8
11:38:19.234  metrics.go:52: INFO Uptime: 40.449909073s
11:38:19.234  beat.go:284: INFO filebeat stopped.

here is the full log:

gist.github.com

https://gist.github.com/DinaWork/96c5b6b7319e9b1d570ac63cc6b5b954

beats-log.txt

11:37:38.621  beat.go:436: INFO Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
11:37:38.623  beat.go:443: INFO Beat UUID: 12871799-2c64-417c-850f-4575f082bdb6
11:37:38.623  beat.go:203: INFO Setup Beat: filebeat; Version: 6.1.0
11:37:38.628  metrics.go:23: INFO Metrics logging every 30s
11:37:38.636  add_cloud_metadata.go:301: INFO add_cloud_metadata: hosting provider type detected as gce, metadata={"availability_zone":"projects/11111111/zones/us-central1-f","instance_id":"333333333","instance_name":"gke-cluster-1-default-pool-d2e02dd6-h975","machine_type":"projects/11111111/machineTypes/n1-standard-1","project_id":"example","provider":"gce"}
11:37:38.637  client.go:123: INFO Elasticsearch url: http://xxxxxx.us-central1.gcp.cloud.es.io:9243
11:37:38.637  module.go:76: INFO Beat name: filebeat-h36p9
11:37:38.638  beat.go:276: INFO filebeat start running.
11:37:38.638  registrar.go:71: INFO No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
11:37:38.643  registrar.go:108: INFO Loading registrar data from /usr/share/filebeat/data/registry

This file has been truncated. show original

Christian_Dahlqvist · December 18, 2017, 1:39pm

Should that not be https (or maybe you should leave the protocol out)?

dinaWork · December 20, 2017, 9:38am

@Christian_Dahlqvist thanks!
I changed

        - name: ELASTICSEARCH_HOST
          value: xxxxxxxxxxx.us-central1.gcp.cloud.es.io

to

        - name: ELASTICSEARCH_HOST
          value: https://xxxxxxxxxxx.us-central1.gcp.cloud.es.io

now I dont get failed to connect , but still filebeat fails with the log:

08:35:47.409927 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30002 beat.memstats.gc_next=30422496 beat.memstats.memory_alloc=15212880 beat.memstats.memory_total=141696232 filebeat.events.active=4119 filebeat.events.added=13031 filebeat.events.done=8912 filebeat.harvester.open_files=20 filebeat.harvester.running=20 filebeat.harvester.started=20 libbeat.config.module.running=1 libbeat.config.module.starts=1 libbeat.config.reloads=2 libbeat.output.read.bytes=157671 libbeat.output.type=elasticsearch libbeat.output.write.bytes=8987379 libbeat.pipeline.clients=2 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.filtered=20 libbeat.pipeline.events.published=13008 libbeat.pipeline.events.retry=50 libbeat.pipeline.events.total=13029 libbeat.pipeline.queue.acked=8892 registrar.states.current=20 registrar.states.update=8912 registrar.writes=184
08:35:48.006481 filebeat.go:323: INFO Stopping filebeat
08:35:48.006524 crawler.go:109: INFO Stopping Crawler
08:35:48.006536 crawler.go:119: INFO Stopping 0 prospectors
08:35:48.006559 reload.go:222: INFO Dynamic config reloader stopped
08:35:48.006580 reload.go:222: INFO Dynamic config reloader stopped
08:35:48.006589 crawler.go:135: INFO Crawler stopped
08:35:48.006613 registrar.go:210: INFO Stopping Registrar
08:35:48.006640 registrar.go:165: INFO Ending Registrar
08:35:48.007093 forwarder.go:35: INFO Prospector outlet closed
08:35:48.014069 metrics.go:51: INFO Total non-zero values: beat.info.uptime.ms=30331 beat.memstats.gc_next=33630224 beat.memstats.memory_alloc=30452528 beat.memstats.memory_total=143441184 filebeat.events.active=4116 filebeat.events.added=13026 filebeat.events.done=8910 filebeat.harvester.closed=1 filebeat.harvester.open_files=16 filebeat.harvester.running=16 filebeat.harvester.started=17 libbeat.config.module.running=1 libbeat.config.module.starts=1 libbeat.config.reloads=2 libbeat.output.read.bytes=157857 libbeat.output.type=elasticsearch libbeat.output.write.bytes=9566414 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=4116 libbeat.pipeline.events.failed=1 libbeat.pipeline.events.filtered=17 libbeat.pipeline.events.published=13008 libbeat.pipeline.events.retry=50 libbeat.pipeline.events.total=13026 libbeat.pipeline.queue.acked=8892 registrar.states.current=17 registrar.states.update=8909 registrar.writes=185
08:35:48.014112 metrics.go:52: INFO Uptime: 30.331515461s
08:35:48.014121 beat.go:284: INFO filebeat stopped.
08:35:48.002759 filebeat.go:323: INFO Stopping filebeat
08:35:48.002784 crawler.go:109: INFO Stopping Crawler
08:35:48.002806 crawler.go:119: INFO Stopping 0 prospectors
08:35:48.002830 reload.go:222: INFO Dynamic config reloader stopped
08:35:48.002847 reload.go:222: INFO Dynamic config reloader stopped
08:35:48.002854 crawler.go:135: INFO Crawler stopped
08:35:48.002861 registrar.go:210: INFO Stopping Registrar
08:35:48.002888 registrar.go:165: INFO Ending Registrar
08:35:48.003162 forwarder.go:35: INFO Prospector outlet closed
08:35:48.003223 forwarder.go:35: INFO Prospector outlet closed
08:35:48.003251 forwarder.go:35: INFO Prospector outlet closed
08:35:48.009565 metrics.go:51: INFO Total non-zero values: beat.info.uptime.ms=30602 beat.memstats.gc_next=30422496 beat.memstats.memory_alloc=17696448 beat.memstats.memory_total=144179800 filebeat.events.active=4116 filebeat.events.added=13231 filebeat.events.done=9115 filebeat.harvester.closed=3 filebeat.harvester.open_files=17 filebeat.harvester.running=17 filebeat.harvester.started=20 libbeat.config.module.running=1 libbeat.config.module.starts=1 libbeat.config.reloads=2 libbeat.output.read.bytes=161229 libbeat.output.type=elasticsearch libbeat.output.write.bytes=9187054 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=4116 libbeat.pipeline.events.failed=3 libbeat.pipeline.events.filtered=20 libbeat.pipeline.events.published=13208 libbeat.pipeline.events.retry=50 libbeat.pipeline.events.total=13231 libbeat.pipeline.queue.acked=9092 registrar.states.current=20 registrar.states.update=9112 registrar.writes=189
08:35:48.009588 metrics.go:52: INFO Uptime: 30.602265961s
08:35:48.009595 beat.go:284: INFO filebeat stopped.

here is the full log:

gist.github.com

https://gist.github.com/DinaWork/f6aeb30029544bac64f6e2b9f181c1a2

filebeat-log.txt


08:35:17.456429 beat.go:436: INFO Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
08:35:17.456638 beat.go:443: INFO Beat UUID: d5d00e36-4ba2--0aba-b447-3f96e4c0418d
08:35:17.456655 beat.go:203: INFO Setup Beat: filebeat; Version: 6.1.0
08:35:17.458913 metrics.go:23: INFO Metrics logging every 30s
08:35:17.469026 add_cloud_metadata.go:301: INFO add_cloud_metadata: hosting provider type detected as gce, metadata={"availability_zone":"projects/123456/zones/us-central1-f","instance_id":"7372638706743012272","instance_name":"gke-cluster-1-default-pool-d2e02dd6-w809","machine_type":"projects/123456/machineTypes/n1-standard-1","project_id":"example","provider":"gce"}
08:35:17.469287 client.go:123: INFO Elasticsearch url: https://xxxxxx.us-central1.gcp.cloud.es.io:9243
08:35:17.470715 module.go:76: INFO Beat name: filebeat-tc2vj
08:35:17.471075 beat.go:276: INFO filebeat start running.
08:35:17.471114 registrar.go:71: INFO No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.

This file has been truncated. show original

exekias · December 20, 2017, 10:45am

Uhm,

From the log it looks like everything worked, and then a normal stop was triggered, are you sure you didn't delete the pod? You can get the history with kubectl describe

system · January 8, 2018, 1:30pm

This topic was automatically closed after 21 days. New replies are no longer allowed.