Filebeat in windows is unable to send logs to logstash in Ubuntu server

sasanka_mourya · May 27, 2017, 6:12pm

My filebeat in windows runs fine. but i am unable to send logs from filebeat in windows to logstash in ubuntu server.
Im running both windows and ubuntu in VM's.
Here is my config file and the error.

filebeat.prospectors:

input_type: log
paths:
- C:\computenext\tracelogs*.txt
  document-type: syslog
  output.logstash:
  hosts: ["172.16.1.84:5443"]
  bulk_max_size: 2048
  template.name: "filebeat"
  template.path: "filebeat.template.json"
  template.overwrite: false

The error is:
2017-05-27T23:25:33+05:30 ERR Failed to publish events caused by: read tcp 172.16.1.240:55392->172.16.1.84:5443: wsarecv: An existing connection was forcibly closed by the remote host.
2017-05-27T23:25:33+05:30 INFO Error publishing events (retrying): read tcp 172.16.1.240:55392->172.16.1.84:5443: wsarecv: An existing connection was forcibly closed by the remote host.

thiago · May 29, 2017, 4:50am

Hello,

This error means that the remote host is closing the connection. You need to check logstash logs to understand what's going on.

Cheers

sasanka_mourya · May 29, 2017, 5:59am

Hello,
My logstash logs do not show any errors relating to the windows server. It just shows old logs. I will upload the old logs if you need to see, but they dont really relate to the errors in windows server.

Thanks.

sasanka_mourya · May 29, 2017, 7:41am

UPDATE: I changed the output of filebeat to elasticsearch instead of logstash. So now the logs go directly into elasticsearch. It works. But the problem is I cannot apply filters in elasticsearch. I need logstash to do the filtering. But through logstash connection is being refused. Need assistance.

Thanks

tudor · May 29, 2017, 8:08am

Can you show us your LS config as well?

I've also noticed that you set the template settings on the LS output. That doesn't work, you need to load the templates manually if you use LS.

What sort of filters do you need? Perhaps the ES Ingest Node is enough?

sasanka_mourya · May 29, 2017, 8:17am

LS- beats input config:
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}

LS- output ES config:
output {
elasticsearch { hosts => ["172.16.1.84:9200"]
hosts => "172.16.1.84:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
These are the config files in logstash.

Filters which process the data according to the need of our company.
I think the problem is with ssl certificate. I havent included ssl in filebeat output config in windows. If i include ssl in the configuration the filebeat wont start.

Thanks

thiago · May 29, 2017, 10:21am

If you define ssl in beats logstash input, then filebeat logstash output will also require ssl configuration. When you say that filebeat won't start, what error do you get? Can you show us the log?

sasanka_mourya · May 29, 2017, 10:28am

It is the ssl certificate. I just removed ssl certificate everywhere and its working. The filebeat in windows server send logs to logstash in ubuntu. But, I know it is not advisable to use it without ssl certificate.
Here is the error that i get when i include ssl:

Restart-Service : Service 'filebeat (filebeat)' cannot be started due to the following error: Cannot start service
filebeat on computer '.'.
At line:1 char:1

Restart-Service filebeat

  + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Restart-Service]
 , ServiceCommandException
  + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.RestartServiceCommand

thiago · May 29, 2017, 10:32am

Well, that really depend on your local policy. Considering a private network that you trust, it may not be needed at all.

That error is not telling much. Can you please attach your filebeat configuration again (with ssl set)?

sasanka_mourya · May 29, 2017, 10:35am

Here is my filebeat config:

filebeat.prospectors:

input_type: log
paths:
- C:\computenext\tracelogs*.txt
  document-type: syslog
  output.logstash:
  hosts: ["172.16.1.84:5443"]
  bulk_max_size: 2048
  ssl.certificate_authorities: ["C:\Program Files\filebeat\logstash"]
  template.name: "filebeat"
  template.path: "filebeat.template.json"
  template.overwrite: false

thiago · May 29, 2017, 10:49am

Your configuration has a set of template.* configuration that the logstash output does not supports, but that should not be blocking it from starting up.

But yet, it is not clear to me why it is not starting. Are you sure that the file C:\Program Files\filebeat\logstash exists and it is a valid certificate authority file? Can you get the full filebeat log file?

sasanka_mourya · May 29, 2017, 10:58am

I dont understand what u mean by valid certificate authority file. I've downloaded the same certificate that ive generated while installing ELK-Stack.
In the image you can see that it has logstash in it.

thiago · May 29, 2017, 11:04am

I think you are missing a proper file extension in that logstash CA file. Can you go into command prompt, execute dir inside the C:\Program Files\filebeat directory and post the result here?

sasanka_mourya · May 29, 2017, 11:08am

>  Volume Serial Number is BC75-F572

>  Directory of C:\Program Files\filebeat

> 05/29/2017  11:45 AM    <DIR>          .
> 05/29/2017  11:45 AM    <DIR>          ..
> 04/28/2017  05:35 PM                41 .build_hash.txt
> 04/28/2017  05:36 PM        16,108,032 filebeat.exe
> 04/28/2017  05:36 PM            37,819 filebeat.full.yml
> 04/28/2017  05:36 PM            24,203 filebeat.template-es2x.json
> 04/28/2017  05:36 PM            19,419 filebeat.template-es6x.json
> 04/28/2017  05:36 PM            19,467 filebeat.template.json
> 05/29/2017  04:30 PM             4,843 filebeat.yml
> 04/28/2017  05:36 PM             4,310 filebeat.yml-BKP
> 04/28/2017  05:36 PM               507 install-service-filebeat.ps1
> 05/29/2017  04:30 PM    <DIR>          logs
> 05/19/2017  06:34 PM             1,830 logstash.crt
> 05/19/2017  06:34 PM             3,272 logstash.key
> 04/28/2017  05:24 PM    <DIR>          module
> 04/28/2017  05:34 PM            97,026 NOTICE
> 04/28/2017  05:35 PM               811 README.md
> 04/28/2017  05:36 PM    <DIR>          scripts
> 04/28/2017  05:36 PM               184 uninstall-service-filebeat.ps1
>               14 File(s)     16,321,764 bytes
>                5 Dir(s)  85,266,006,016 bytes free

>

Here is the output of what you've asked.

thiago · May 29, 2017, 11:11am

Ok, so the ssl.certificate_authorities setting should be with C:\Program Files\filebeat\logstash.crt instead.

sasanka_mourya · May 29, 2017, 11:16am

I have changed it. But the error is the same when i try to restart it.

PS C:\Program Files\Filebeat> Restart-Service filebeat
Restart-Service : Service 'filebeat (filebeat)' cannot be started due to the following error: Cannot start service
filebeat on computer '.'.
At line:1 char:1

Restart-Service filebeat

  + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Restart-Service]
 , ServiceCommandException
  + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.RestartServiceCommand

thiago · May 29, 2017, 11:19am

Ok. Then we really need to see the latest logs from filebeat. It is inside the logs directory, should be a file named filebeat only.

sasanka_mourya · May 29, 2017, 11:24am

2017-05-29T16:50:44+05:30 INFO Stopping filebeat
2017-05-29T16:50:44+05:30 INFO Stopping Crawler
2017-05-29T16:50:44+05:30 INFO Stopping 1 prospectors
2017-05-29T16:50:44+05:30 INFO Prospector channel stopped because beat is stopping.
2017-05-29T16:50:44+05:30 INFO Scan aborted because prospector stopped.
2017-05-29T16:50:45+05:30 INFO Prospector ticker stopped
2017-05-29T16:50:45+05:30 INFO Stopping Prospector: 7528349933066091638
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125046-notification[5680].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125028-authorization[1716].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-165016-insightv2[4140].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164524-insightv2[7652].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164724-insightv2[9092].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164424-insightv2[6580].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164624-insightv2[7608].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164324-insightv2[5828].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164224-insightv2[8152].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125043-gateway[5152].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-165024-insightv2[8304].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125029-providerv2[4272].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125040-billingv2[272].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125041-background[4728].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125039-apiv2[4944].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125031-workloadsv2[4472].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125044-resourcesv2[5348].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125025-authentication[3160].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125021-CnRestApiConsole-INTERNAL[3440].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125024-archive[3744].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-125019-monitorservice[4004].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164824-insightv2[8212].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164924-insightv2[6512].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Reader was closed: C:\computenext\tracelogs\ABTIS-QA-APP-20170529-164516-insightv2[7416].txt. Closing.
2017-05-29T16:50:45+05:30 INFO Crawler stopped
2017-05-29T16:50:45+05:30 INFO Stopping spooler
2017-05-29T16:50:45+05:30 INFO Stopping Registrar
2017-05-29T16:50:45+05:30 INFO Ending Registrar
2017-05-29T16:50:45+05:30 INFO Total non-zero values:  filebeat.harvester.closed=24 filebeat.harvester.started=24 libbeat.logstash.call_count.PublishEvents=30 libbeat.logstash.publish.read_bytes=750 libbeat.logstash.publish.write_bytes=114833 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=2639 libbeat.logstash.published_but_not_acked_events=8 libbeat.publisher.published_events=2639 publish.events=11880 registrar.states.current=8777 registrar.states.update=11880 registrar.writes=34
2017-05-29T16:50:45+05:30 INFO Uptime: 4m2.7424832s
2017-05-29T16:50:45+05:30 INFO filebeat stopped.
    indent preformatted text by 4 spaces

This is the error when i uncomment "ssl.certificate_authorities" line and try to restart filebeat.

thiago · May 29, 2017, 11:28am

This is not an error, it is simply telling g that filebeat is stopping. Is it all that is inside log file? Can you paste the complete log file in pastebin.com and attach link here?

sasanka_mourya · May 29, 2017, 11:47am

https://pastebin.com/qRezUxmb

Here is the link.
The logs that send data to logstash are when i run filebeat without ssl_certificate.

Topic		Replies	Views
Filebeat on Windows ==> Logstash Ubuntu Beats filebeat	2	449	October 24, 2018
Filebeat to logstash: Failed to publish events caused by: read tcp 192.168.155.177:55376->47.102.46.68:5045:wsarecv:An existing connection was forcibly closed by the remote host Beats filebeat	7	2367	February 5, 2020
Filebeat to Logstash ERROR, wsasend, wsarecv Logstash	1	2127	February 28, 2017
Filebeat on windows to ES instance on Ubuntu Logstash	5	415	April 18, 2018
ERR Failed to publish events caused by: write tcp - wsasend: An existing connection was forcibly closed by the remote host Beats filebeat	1	681	April 22, 2021

Filebeat in windows is unable to send logs to logstash in Ubuntu server

Related topics