FileBeat to logstash via IPv6 - cannot assign requested address

Elastic Stack Beats
1

Hi Folks

An ELK/Beats noob here

I'm trying to send audit log entries from the app server to an ELK server using filebeats (docker)

These are the errors i'm seeing

2019-10-31T12:16:57.109Z INFO log/harvester.go:251 Harvester started for file: /var/www/vhosts/domain.tld/shared/storage/logs/audit-log.log
2019-10-31T12:16:58.109Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044))
2019-10-31T12:17:00.039Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044)): dial tcp [MY-ELK-IPv6-ADDRESS]:5044: connect: cannot assign requested address

I've no control over the ELK server but logstash is listening and i can get a connection via telnet

telnet [MY-ELK-IPv6-ADDRESS] 5044
Trying [MY-ELK-IPv6-ADDRESS]...
Connected to [MY-ELK-IPv6-ADDRESS].
Escape character is '^]'.

output from ELK server to confirm it's listening

lsof -i -P -n

java 996 logstash 96u IPv6 21775 0t0 TCP 127.0.0.1:39994->127.0.0.1:9200 (ESTABLISHED)
java 996 logstash 109u IPv6 21244 0t0 TCP *:5044 (LISTEN)
java 996 logstash 112u IPv6 21789 0t0 TCP 127.0.0.1:9600 (LISTEN)
java 996 logstash 121u IPv6 21251 0t0 TCP 127.0.0.1:5044->127.0.0.1:53650 (ESTABLISHED)

I'm running beats via docker

docker run --name=filebeat20 --user=root --volume="/home/deployer/filebeat-qa.yml:/usr/share/filebeat/filebeat.yml:ro" --volume="/var/www/vhosts/domain.tld/shared/storage/logs/:/var/www/vhosts/domain.tld/shared/storage/logs/:ro" docker.elastic.co/beats/filebeat:7.4.1 filebeat -e -strict.perms=false

this is my config file

app.home: /var/www/vhosts/domain.tld/shared/storage/logs/

filebeat.inputs:

  • type: log
    paths:
    • ${app.home}/audit-log.log
      encoding: utf-8
      json.keys_under_root: true
      json.add_error_key: true
      json.message_key: message
      processors:
    • add_tags:
      tags: ["blah blah","qa"]
      target: "environment"

#----------------------------- Logstash output -------------------------------
output.logstash:
hosts: ["[MY-ELK-IPv6-ADDRESS]:5044"]

totally stumped so any help is appreciated

thanks

Dean

2

full log

2019-10-31T13:03:13.173Z INFO instance/beat.go:607 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-10-31T13:03:13.196Z INFO instance/beat.go:615 Beat ID: 90602ca9-90f9-4372-a0fc-4205ad22a900
2019-10-31T13:03:13.197Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2019-10-31T13:03:13.197Z INFO [beat] instance/beat.go:903 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "90602ca9-90f9-4372-a0fc-4205ad22a900"}}}
2019-10-31T13:03:13.198Z INFO [beat] instance/beat.go:912 Build info {"system_info": {"build": {"commit": "12ee6cd05c1bfdc69721ddab1f473417b1514576", "libbeat": "7.4.1", "time": "2019-10-22T16:22:37.000Z", "version": "7.4.1"}}}
2019-10-31T13:03:13.198Z INFO [beat] instance/beat.go:915 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.9"}}}
2019-10-31T13:03:13.205Z INFO [beat] instance/beat.go:919 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-04-03T12:20:50Z","containerized":true,"name":"749cfd1a167e","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"4.15.0-46-generic","mac":["02:42:ac:11:00:02"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-10-31T13:03:13.205Z INFO [beat] instance/beat.go:948 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-10-31T13:03:07.350Z"}}}
2019-10-31T13:03:13.205Z INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.4.1
2019-10-31T13:03:13.209Z INFO [publisher] pipeline/module.go:97 Beat name: 749cfd1a167e
2019-10-31T13:03:13.241Z WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-10-31T13:03:13.242Z INFO instance/beat.go:422 filebeat start running.
2019-10-31T13:03:13.242Z INFO registrar/migrate.go:104 No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2019-10-31T13:03:13.242Z INFO registrar/migrate.go:112 Initialize registry meta file
2019-10-31T13:03:13.243Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2019-10-31T13:03:13.300Z INFO registrar/registrar.go:108 No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2019-10-31T13:03:13.304Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2019-10-31T13:03:13.304Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2019-10-31T13:03:13.304Z WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-10-31T13:03:13.304Z INFO crawler/crawler.go:72 Loading Inputs: 1
2019-10-31T13:03:13.306Z INFO log/input.go:152 Configured paths: [/var/www/vhosts/domain.tld/shared/storage/logs/audit-log.log]
2019-10-31T13:03:13.307Z INFO input/input.go:114 Starting input of type: log; ID: 16756111730527578735
2019-10-31T13:03:13.307Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-10-31T13:03:13.317Z INFO log/harvester.go:251 Harvester started for file: /var/www/vhosts/domain.tld/shared/storage/logs/audit-log.log
2019-10-31T13:03:14.318Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044))
2019-10-31T13:03:15.477Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044)): dial tcp [MY-ELK-IPv6-ADDRESS]:5044: connect: cannot assign requested address
2019-10-31T13:03:15.478Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044)) with 1 reconnect attempt(s)
2019-10-31T13:03:18.915Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044)): dial tcp [MY-ELK-IPv6-ADDRESS]:5044: connect: cannot assign requested address
2019-10-31T13:03:18.915Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044)) with 2 reconnect attempt(s)
2019-10-31T13:03:24.731Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044)): dial tcp [MY-ELK-IPv6-ADDRESS]:5044: connect: cannot assign requested address
2019-10-31T13:03:24.731Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044)) with 3 reconnect attempt(s)
2019-10-31T13:03:33.457Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044)): dial tcp [MY-ELK-IPv6-ADDRESS]:5044: connect: cannot assign requested address
2019-10-31T13:03:33.457Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://[MY-ELK-IPv6-ADDRESS]:5044)) with 4 reconnect attempt(s)

3

continued

2019-10-31T13:03:37.885Z INFO beater/filebeat.go:443 Stopping filebeat
2019-10-31T13:03:37.885Z INFO crawler/crawler.go:139 Stopping Crawler
2019-10-31T13:03:37.885Z INFO crawler/crawler.go:149 Stopping 1 inputs
2019-10-31T13:03:37.885Z INFO input/input.go:149 input ticker stopped
2019-10-31T13:03:37.885Z INFO input/input.go:167 Stopping Input: 16756111730527578735
2019-10-31T13:03:37.885Z INFO log/harvester.go:272 Reader was closed: /var/www/vhosts/domain.tld/shared/storage/logs/audit-log.log. Closing.
2019-10-31T13:03:37.885Z INFO crawler/crawler.go:165 Crawler stopped
2019-10-31T13:03:37.885Z INFO registrar/registrar.go:367 Stopping Registrar
2019-10-31T13:03:37.885Z INFO registrar/registrar.go:293 Ending Registrar
2019-10-31T13:03:37.902Z INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":48}},"total":{"ticks":290,"time":{"ms":299},"value":290},"user":{"ticks":250,"time":{"ms":251}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"61768916-49e0-404d-8335-285394edada4","uptime":{"ms":24811}},"memstats":{"gc_next":8566352,"memory_alloc":5186024,"memory_total":10297352,"rss":44851200},"runtime":{"goroutines":13}},"filebeat":{"events":{"active":3,"added":4,"done":1},"harvester":{"closed":1,"open_files":0,"running":0,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":3,"filtered":1,"published":3,"retry":12,"total":4}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":3,"total":3}},"system":{"cpu":{"cores":2},"load":{"1":0.05,"15":0.15,"5":0.01,"norm":{"1":0.025,"15":0.075,"5":0.005}}}}}}
2019-10-31T13:03:37.902Z INFO [monitoring] log/log.go:154 Uptime: 24.811508444s
2019-10-31T13:03:37.902Z INFO [monitoring] log/log.go:131 Stopping metrics logging.
2019-10-31T13:03:37.902Z INFO instance/beat.go:432 filebeat stopped.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.