Trying to setup winlogbeat to ship windows events to working ELK stack


(Iqbal Nazir) #1

Hello,
I have a working ELK server on Ubuntu 16.04. I have another ubuntu 14.04 which is sending logs to my ELK successfully. Now, I want to add a windows client to my ELK which will send its events to ELK. To do so, I'm trying to install winlogbeat on my windows 10. I'm following the official documentation. From there, when I try to run Invoke-WebRequest -Method Put -InFile winlogbeat.template.json -Uri http://My_ELK_server_IP:9200/_template/winlogbeat?pretty command in the powershell it shows me the following error:

`Invoke-WebRequest : Unable to connect to the remote server
At line:1 char:1

  • Invoke-WebRequest -Method Put -InFile winlogbeat.template.json -Uri 1 ...
  •   + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], WebException
      + FullyQualifiedErrorId : System.Net.WebException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand`
    

I would really appreciate if someone comes forward to help me.

--best regards,
Iqbal


(Andrew Kroh) #2

Based on the error, I would say you have a connectivity problem between the Windows host and Elasticsearch. Can you ping the ES host from Windows? Is there a firewall somewhere in between?


(Iqbal Nazir) #3

Hi,

Yes, I can ping my ES host from windows. And they are in the same network, so no firewall in between.


(Andrew Kroh) #4

Are you able to access http://My_ELK_server_IP:9200/ in the browser from that Windows host?


(Iqbal Nazir) #5

No, but I can browse http://My_ELK_server_IP/ which redirects to Kibana. I can't telnet My_ELK_server_IP with 9200 from Windows host.


(Andrew Kroh) #6

Can you run netstat -an | grep 9200 on the host running Elasticsearch and provide the output. Maybe ES is not listening on the interface which you are trying to connect to.


(Iqbal Nazir) #7
root@ELK:~# netstat -an | grep 9200
tcp        0      0 127.0.0.1:51494         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51458         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51560         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51432         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51586         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51446         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51558         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51564         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51452         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51436         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51514         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51512         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51418         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51546         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51392         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51450         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51466         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51496         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51400         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51524         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51580         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51490         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51456         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51442         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51448         127.0.0.1:9200          TIME_WAIT

(Iqbal Nazir) #8
    tcp        0      0 127.0.0.1:51556         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51588         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51388         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51542         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51462         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51414         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51474         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51390         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51422         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51424         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51464         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51508         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51434         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51590         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51502         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51396         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51540         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51426         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51444         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51568         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51398         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51520         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51402         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51522         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51538         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51576         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51554         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51498         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51412         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51440         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51416         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51548         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51510         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51488         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51584         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51550         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51582         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51504         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51470         127.0.0.1:9200          TIME_WAIT
    tcp        0      0 127.0.0.1:51536         127.0.0.1:9200          TIME_WAIT

(Iqbal Nazir) #9
tcp        0      0 127.0.0.1:51578         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51478         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51410         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51516         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51528         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51534         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51482         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51484         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51394         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51486         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51526         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51544         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51500         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51518         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51460         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51428         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51530         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51562         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51408         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51430         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51454         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51552         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51570         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51572         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51492         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51468         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51480         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51406         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51574         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51476         127.0.0.1:9200          TIME_WAIT
tcp        0      0 127.0.0.1:51420         127.0.0.1:9200          TIME_WAIT
tcp6       0      0 ::1:9200                :::*                    LISTEN
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN
tcp6       0      0 127.0.0.1:9200          127.0.0.1:51566         ESTABLISHED
tcp6       0      0 127.0.0.1:38368         127.0.0.1:9200          ESTABLISHED
tcp6       0      0 127.0.0.1:51472         127.0.0.1:9200          TIME_WAIT
tcp6       0      0 127.0.0.1:37920         127.0.0.1:9200          ESTABLISHED
tcp6       0      0 127.0.0.1:51438         127.0.0.1:9200          TIME_WAIT
tcp6       0      0 127.0.0.1:51506         127.0.0.1:9200          TIME_WAIT
tcp6       0      0 127.0.0.1:51566         127.0.0.1:9200          ESTABLISHED
tcp6       0      0 127.0.0.1:51404         127.0.0.1:9200          TIME_WAIT
tcp6       0      0 127.0.0.1:9200          127.0.0.1:38368         ESTABLISHED
tcp6       0      0 127.0.0.1:9200          127.0.0.1:37920         ESTABLISHED
tcp6       0      0 127.0.0.1:51370         127.0.0.1:9200          TIME_WAIT
tcp6       0      0 127.0.0.1:51532         127.0.0.1:9200          TIME_WAIT

(Andrew Kroh) #10

It looks like Elasticsearch is only listening on the loopback interface (which is the default for security purposes). You need to configure it to accept outside connections. See https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#common-network-settings


(Iqbal Nazir) #11

Hi,

Thanks a lot. I have tried changing network.host value from localhost to my_ELK_server_IP with http.port 9200. It makes my ELK "RED". I am a newbie. Could you please tell what I should change exactly?


(Andrew Kroh) #12

The simplest (and most insecure) change would be to set network.host: 0.0.0.0.


(Iqbal Nazir) #13

ok.. thanks again.. now I can see a log file is created in (C:)\ProgramData\winlogbeat\Logs\winlogbeat and it shows error as the following:
2016-06-14T16:57:45+02:00 INFO Error publishing events (retrying): EOF 2016-06-14T16:57:45+02:00 INFO send fail 2016-06-14T16:57:45+02:00 INFO backoff retry: 1m0s 2016-06-14T16:58:45+02:00 INFO Error publishing events (retrying): EOF 2016-06-14T16:58:45+02:00 INFO send fail 2016-06-14T16:58:45+02:00 INFO backoff retry: 1m0s 2016-06-14T16:59:45+02:00 INFO Error publishing events (retrying): EOF 2016-06-14T16:59:45+02:00 INFO send fail 2016-06-14T16:59:45+02:00 INFO backoff retry: 1m0s


(Andrew Kroh) #14

Can you set the logging level to debug in your configuration file. This may provide bit more detail around what the error is.

Are you now able to access http://My_ELK_server_IP:9200/ from the browser on the Windows host?


(Iqbal Nazir) #15

Yes, now I can access my_elk_server_ip:9200 which shows:
{ "name" : "ELK_1st_node", "cluster_name" : "ELK", "version" : { "number" : "2.3.3", "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde", "build_timestamp" : "2016-05-17T15:40:04Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" }

For your 1st question: sorry I don't know how to set the logging level to 'debug' in config file.


Kibana and plugin web sites not available at the same time
(Andrew Kroh) #16

You must modify the configuration file and restart the service. See docs for level: https://www.elastic.co/guide/en/beats/winlogbeat/current/configuration-logging.html#_level


(Iqbal Nazir) #17

Hi again,

I have set the logging level to debug as you said. Still getting the same error:

2016-06-15T13:26:34+02:00 INFO Error publishing events (retrying): EOF
2016-06-15T13:26:34+02:00 INFO send fail
2016-06-15T13:26:34+02:00 INFO backoff retry: 1m0s

Do you want to review my config file?
regards.


(Andrew Kroh) #18

Yeah, can you please share you config.

Also please check you Elasticsearch logs to see if there is anything interesting going on there.


(Iqbal Nazir) #19

Hi,

I have been able to ship Windows logs to my ELK stack. What I did was that I kept elasticsearch as the output and commented out logstash. Till yesterday, I was trying to keep logstash as output commenting out the elasticsearch part , since I have logstash installed on my ELK server and the document was telling me to do so.

Just to know, is Winlogbeat the only beat which ships Windows events to ELK stack or I can use the others (filebeat, topbeat etc.) too?


(Kevin Csuka) #20

Yeah you can use them too.
See below: https://www.elastic.co/support/matrix