Filebeat in windows is unable to send logs to logstash in Ubuntu server


(Sasanka Mourya) #1

My filebeat in windows runs fine. but i am unable to send logs from filebeat in windows to logstash in ubuntu server.
Im running both windows and ubuntu in VM's.
Here is my config file and the error.

filebeat.prospectors:

  • input_type: log
    paths:
    • C:\computenext\tracelogs*.txt
      document-type: syslog
      output.logstash:
      hosts: ["172.16.1.84:5443"]
      bulk_max_size: 2048
      template.name: "filebeat"
      template.path: "filebeat.template.json"
      template.overwrite: false

The error is:
2017-05-27T23:25:33+05:30 ERR Failed to publish events caused by: read tcp 172.16.1.240:55392->172.16.1.84:5443: wsarecv: An existing connection was forcibly closed by the remote host.
2017-05-27T23:25:33+05:30 INFO Error publishing events (retrying): read tcp 172.16.1.240:55392->172.16.1.84:5443: wsarecv: An existing connection was forcibly closed by the remote host.


Filebeat has to be restarted to send logs to elasticsearch
(Thiago Souza) #2

Hello,

This error means that the remote host is closing the connection. You need to check logstash logs to understand what's going on.

Cheers


(Sasanka Mourya) #3

Hello,
My logstash logs do not show any errors relating to the windows server. It just shows old logs. I will upload the old logs if you need to see, but they dont really relate to the errors in windows server.

Thanks.


(Sasanka Mourya) #4

UPDATE: I changed the output of filebeat to elasticsearch instead of logstash. So now the logs go directly into elasticsearch. It works. But the problem is I cannot apply filters in elasticsearch. I need logstash to do the filtering. But through logstash connection is being refused. Need assistance.

Thanks


(Tudor Golubenco) #5

Can you show us your LS config as well?

I've also noticed that you set the template settings on the LS output. That doesn't work, you need to load the templates manually if you use LS.

What sort of filters do you need? Perhaps the ES Ingest Node is enough?


(Sasanka Mourya) #6

LS- beats input config:
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}

LS- output ES config:
output {
elasticsearch { hosts => ["172.16.1.84:9200"]
hosts => "172.16.1.84:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
These are the config files in logstash.

Filters which process the data according to the need of our company.
I think the problem is with ssl certificate. I havent included ssl in filebeat output config in windows. If i include ssl in the configuration the filebeat wont start.

Thanks


(Thiago Souza) #7

If you define ssl in beats logstash input, then filebeat logstash output will also require ssl configuration. When you say that filebeat won't start, what error do you get? Can you show us the log?


(Sasanka Mourya) #8

It is the ssl certificate. I just removed ssl certificate everywhere and its working. The filebeat in windows server send logs to logstash in ubuntu. But, I know it is not advisable to use it without ssl certificate.
Here is the error that i get when i include ssl:

Restart-Service : Service 'filebeat (filebeat)' cannot be started due to the following error: Cannot start service
filebeat on computer '.'.
At line:1 char:1

  • Restart-Service filebeat
  •   + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Restart-Service]
     , ServiceCommandException
      + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.RestartServiceCommand

(Thiago Souza) #9

Well, that really depend on your local policy. Considering a private network that you trust, it may not be needed at all.

That error is not telling much. Can you please attach your filebeat configuration again (with ssl set)?


(Sasanka Mourya) #10

Here is my filebeat config:

filebeat.prospectors:

  • input_type: log
    paths:
    • C:\computenext\tracelogs*.txt
      document-type: syslog
      output.logstash:
      hosts: ["172.16.1.84:5443"]
      bulk_max_size: 2048
      ssl.certificate_authorities: ["C:\Program Files\filebeat\logstash"]
      template.name: "filebeat"
      template.path: "filebeat.template.json"
      template.overwrite: false

(Thiago Souza) #11

Your configuration has a set of template.* configuration that the logstash output does not supports, but that should not be blocking it from starting up.

But yet, it is not clear to me why it is not starting. Are you sure that the file C:\Program Files\filebeat\logstash exists and it is a valid certificate authority file? Can you get the full filebeat log file?


(Sasanka Mourya) #12

I dont understand what u mean by valid certificate authority file. I've downloaded the same certificate that ive generated while installing ELK-Stack.
In the image you can see that it has logstash in it.


(Thiago Souza) #13

I think you are missing a proper file extension in that logstash CA file. Can you go into command prompt, execute dir inside the C:\Program Files\filebeat directory and post the result here?


(Sasanka Mourya) #14
>  Volume Serial Number is BC75-F572

>  Directory of C:\Program Files\filebeat

> 05/29/2017  11:45 AM    <DIR>          .
> 05/29/2017  11:45 AM    <DIR>          ..
> 04/28/2017  05:35 PM                41 .build_hash.txt
> 04/28/2017  05:36 PM        16,108,032 filebeat.exe
> 04/28/2017  05:36 PM            37,819 filebeat.full.yml
> 04/28/2017  05:36 PM            24,203 filebeat.template-es2x.json
> 04/28/2017  05:36 PM            19,419 filebeat.template-es6x.json
> 04/28/2017  05:36 PM            19,467 filebeat.template.json
> 05/29/2017  04:30 PM             4,843 filebeat.yml
> 04/28/2017  05:36 PM             4,310 filebeat.yml-BKP
> 04/28/2017  05:36 PM               507 install-service-filebeat.ps1
> 05/29/2017  04:30 PM    <DIR>          logs
> 05/19/2017  06:34 PM             1,830 logstash.crt
> 05/19/2017  06:34 PM             3,272 logstash.key
> 04/28/2017  05:24 PM    <DIR>          module
> 04/28/2017  05:34 PM            97,026 NOTICE
> 04/28/2017  05:35 PM               811 README.md
> 04/28/2017  05:36 PM    <DIR>          scripts
> 04/28/2017  05:36 PM               184 uninstall-service-filebeat.ps1
>               14 File(s)     16,321,764 bytes
>                5 Dir(s)  85,266,006,016 bytes free

>

Here is the output of what you've asked.


(Thiago Souza) #15

Ok, so the ssl.certificate_authorities setting should be with C:\Program Files\filebeat\logstash.crt instead.


(Sasanka Mourya) #16

I have changed it. But the error is the same when i try to restart it.

PS C:\Program Files\Filebeat> Restart-Service filebeat
Restart-Service : Service 'filebeat (filebeat)' cannot be started due to the following error: Cannot start service
filebeat on computer '.'.
At line:1 char:1

  • Restart-Service filebeat
  •   + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Restart-Service]
     , ServiceCommandException
      + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.RestartServiceCommand

(Thiago Souza) #17

Ok. Then we really need to see the latest logs from filebeat. It is inside the logs directory, should be a file named filebeat only.


(Sasanka Mourya) #18
2017-05-29T16:50:44+05:30 INFO Stopping filebeat
2017-05-29T16:50:44+05:30 INFO Stopping Crawler
2017-05-29T16:50:44+05:30 INFO Stopping 1 prospectors
2017-05-29T16:50:44+05:30 INFO Prospector channel stopped because beat is stopping.
2017-05-29T16:50:44+05:30 INFO Scan aborted because prospector stopped.
2017-05-29T16:50:45+05:30 INFO Prospector ticker stopped
2017-05-29T16:50:45+05:30 INFO Stopping Prospector: 7528349933066091638
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125046-notification[5680].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125028-authorization[1716].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-165016-insightv2[4140].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164524-insightv2[7652].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164724-insightv2[9092].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164424-insightv2[6580].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164624-insightv2[7608].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164324-insightv2[5828].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164224-insightv2[8152].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125043-gateway[5152].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-165024-insightv2[8304].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125029-providerv2[4272].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125040-billingv2[272].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125041-background[4728].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125039-apiv2[4944].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125031-workloadsv2[4472].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125044-resourcesv2[5348].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125025-authentication[3160].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125021-CnRestApiConsole-INTERNAL[3440].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125024-archive[3744].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125019-monitorservice[4004].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164824-insightv2[8212].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164924-insightv2[6512].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164516-insightv2[7416].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Crawler stopped
2017-05-29T16:50:45+05:30 INFO Stopping spooler
2017-05-29T16:50:45+05:30 INFO Stopping Registrar
2017-05-29T16:50:45+05:30 INFO Ending Registrar
2017-05-29T16:50:45+05:30 INFO Total non-zero values:  filebeat.harvester.closed=24 filebeat.harvester.started=24 libbeat.logstash.call_count.PublishEvents=30 libbeat.logstash.publish.read_bytes=750 libbeat.logstash.publish.write_bytes=114833 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=2639 libbeat.logstash.published_but_not_acked_events=8 libbeat.publisher.published_events=2639 publish.events=11880 registrar.states.current=8777 registrar.states.update=11880 registrar.writes=34
2017-05-29T16:50:45+05:30 INFO Uptime: 4m2.7424832s
2017-05-29T16:50:45+05:30 INFO filebeat stopped.
    indent preformatted text by 4 spaces

This is the error when i uncomment "ssl.certificate_authorities" line and try to restart filebeat.


(Thiago Souza) #19

This is not an error, it is simply telling g that filebeat is stopping. Is it all that is inside log file? Can you paste the complete log file in pastebin.com and attach link here?


(Sasanka Mourya) #20

https://pastebin.com/qRezUxmb

Here is the link.
The logs that send data to logstash are when i run filebeat without ssl_certificate.