We are having issue sending logs from filebeat to logstash. We do not see the network connections being made, nor errors. There is no firewall between the two components. Filebeat registry is showing files being ingested, but not sent. The logs don't show any errors. Logstash is set to receive a beats input at 5044. Filebeat is configured to connect at 5044. Any suggestions?
Could you share logs, configs and versions of Logstash and Filebeat?
I updated to 6.1 yesterday. I had the same issue with 6.0
The filename that should be used is DataDomainFilesysStatus_JSON_201712191011
The date is in the filename (2017_12_19, 10:11am)
Filebeat: https://pastebin.com/4PSzfeRg
Logstash: https://pastebin.com/VqXndgcx
filebeat.yml: https://pastebin.com/Me1ysed1
filter.conf: https://pastebin.com/L8QzTHQX
Thank You
In the filebeat logs are the following errors: ERR Failed to connect: dial tcp 127.0.0.1:5044: getsockopt: connection refused. It seems Logstash is not reachable. Is it on localhost?
Yes. All components are on the same host, at this time.
I see the connection problem that you mentioned in filebeat. I believe all services weren't running at that time. I restarted filebeat, and have uploaded the lastest to:
https://pastebin.com/XdzDx4iM
Any ideas? I've restarted services, and rebooted. We don't see anything in the log. Has anyone else seen this issue?
One note for the logstash log: Quite a few errors / deprecated notice show up because of the type usage. I wonder if you have some issue here.
From filebeat perspective the events are definitively confirmed to have been sent. On the LS side log unfortunately stops a soon as LS is ready to receive events. Can you remove the ES output and just us the stdout output to see if any events are flowing in. If you start LS in debug mode, you should also see more info.
Ruflin,
Where do we remove the type usage?
I have removed the ES part of the configuration file, and still see the error.
I am running logstash in Debug mode:
/usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.config /etc/logstash/conf.d/
At 11:30am, I restarted the services:
filebeat: https://pastebin.com/R23BbBVb
logstash: https://pastebin.com/R74C0VRT
logstash/conf.d/filter.conf: https://pastebin.com/En4Vnrmh
Any other ideas? Thank you.
That is pretty strange as I would not expect this message when you remove the elasticsearch output.
Ignoring the error for now, do you see the events coming into LS?
I still don't see events. Filebeat sees the file change, but logstash doesn't see it. Any ideas?
Could you run filebeat with the debug log enabled and share it?
It looks like we got a lot more information from the debug.
Logstash - https://pastebin.com/NVuuUy7b
filebeat.degub - https://pastebin.com/gfsy1Zyb
filebeat - https://zerobin.net/?bdb30e7c26eecd50#rJqNQDDlwsgRrB8oPyY1zSSblh/f2p26m3vzRxwZ12g=
Can you share the content of for example /mnt/syslogs/Storage/Reports/DataDomain/FilesysStatus/DataDomainFilesysStatus_JSON_201712270530.txt Do your files have a new line at the end?
I have attached my JSON files. One file is from 12/27. The other is more recent 1/12.
DataDomainFilesysStatus_JSON_201801120530.txt
https://pastebin.com/ktkqDZtu
DataDomainFilesysStatus_JSON_201712270530.txt
https://pastebin.com/97ZKnNgb
Can you verify logstash logs to see if there are errors reaching out to elasticsearch? You may have to specify complete host url to elasticsearch instead of localhost:9200 in your logstash pipeline configuration
I have noticed that my regular file seems to work, but the JSON didn't. I have added a newline character to the JSON file, and it is being passed to logstash.
With the new line character, this issue is resolved. WE can close this.
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.