I have configured beats to ouput logs to logstash via 2 servers. The problem is that for some reason logs are not being shipped to Logstash, I even ran a tcpdump on logstash servers and can't find anything relevant. What can I check from beats side ? Network is open.
Hey @charlot_Attard,
What beats are you using? Could you share the configuration? Do you see anything in Beats logs?
HI @jsoriano,
Thanks for your reply.
Beats Version is 7.16.6
Basically we are shipping logs from a filebeat windows server to a Logstash Linux Server. On the filebeat server we are not seeing any exceptions but on the Logstash server we are seeing an exception similar to the below
" local:0.0.0.0:5044 remote: IPOFBEATSSERVER Handling exception
Received fatal alert: bad_certificate
An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last"
Beats is configured with the below certs
ssl_enable: true
ssl.certificate_authorities:
ssl.certificate:
ssl.key:
Thanks,
Can you share the whole Beats configuration? Where is this ssl configuration enclosed?
I guess they have actual values in the configuration?
The error clearly seems to point to some issue with some certificate.
Hello @jsoriano
Thanks for your help and sorry for not replying before.
What I noticed is that when configuring filebeat.yml to connect to logstash servers with hostnames, there is no network activity on the logstash servers. I confirmed this with a tcpdump. When I configured filebeat to connect to logstash servers with IP's there is network connectivity.
Do you have an idea of what might be the problem ?
Thanks a lot,
Charlot
Ok, this, and the bad certificate error may indicate that some certificate is only valid when using the IP, but not when using the hostname. This could be because the certificate has the IP as alternate name but not the hostname.
How are you generating these certificates? Can you check what alternate names and IPs they have?
1 Like
The bad_certificate i am getting is because I tried to connect from filebeat to logstash via IP and the certificates were generated with the hostname of the servers.
So at first I had hostnames configured and after not seeing any network activity on the logstash servers I decided to switch for IP's. When I did I start seeing network activity via tcpdump and on logstash logs bad_certificate ( Reason being because the certs were generated with hostname as CN. (I confirmed this now again with openssl). So the bottom line that I came up with is that the filebeat agent establishes a session with IP's but not with hostnames. Please note that I used host files to bypass any DNS issues on filebeat servers and also tried nslookup which worked on filebeat agents.
To discard any issue with the logstash output or logstash communication, did you try to output to console or to file?
You could you try to comment out the logstash output in the configuration, and add a file output? Then check if filebeat is writing events to these files.
If it is not, then there may be no problem with the logstash output, but the problem is in filebeat collecting logs.
1 Like
Thanks for the good suggestion @jsoriano
I did it and it worked, so beats is able to catch and write logs. So the problem lies with sending events to logstash I believe
1 Like
Ok, so the issue seems to be in the logstash output, yes. Try to enable debug logging for it, for that, run filebeat with -d logstash.
If Filebeat fails to connect or to send events to Logstash there should be something in the logs.
Btw, you can also consider updating Filebeat, at least to last 7.17 if you don't want to jump to 8.x yet.
I m getting this
Unable to write data to the transport connection An established connection was aborted by the software in your host machine
Quite intresting googling a bit about it.
Hello @jsoriano ,
What is happening now Im getting
DNS lookup failure no such host.
NSLookup on the server works and I also did a host file.
Umm, it could be that beats is not using the host file 
You mentioned that Filebeat is running on Windows, right? How are you configuring the host file there?
How are you installing and running Filebeat?
I solved this by connecting filebeat to only one logstash node.
Now Im not seeing any more errors on filebeat. Is there a way to know on logstash that logs are being received by filebeat ?
Please Note : When running tcpdump on logstash now i am seeing network activity from filebeat. 
What outputs have you configured in logstash? You could check there if there is any document.
I have configured
output.logstash - I'm querying data in kibana and can't find any documents.
My idea is if it is possible to search in logstash and maybe I check where is the data being shipped from filebeat. But I don't know if logstash writes data on the logstash server of filebeat.
I was asking about the configuration in logstash. Do you have any output configured there?
Sorry I misunderstood you @jsoriano
Output is rabbitmq