Winlogbeat not outputting to Logstash

[Diggy]

All,

I've read several posts here regarding getting Winlogbeat to send logs to an ELK server. However, these haven't helped me get it working. As I usually do, I probably am missing something simple.

I already have Filebeat, Topbeat, and Packetbeat working, so I'm fairly certain that my ELK configs are correct; I can post these if it would help with a solution. My winlogbeat.yml file looks like this:

winlogbeat:
  registry_file: C:/ProgramData/winlogbeat/.winlogbeat.yml
  event_logs:
    - name: Application
    - name: Security
    - name: System
output:
  logstash:
    hosts: ["10.0.101.101:5044"]
    template:
      # Template name. By default the template name is winlogbeat.
      #name: "winlogbeat"
      # Path to template file
      #path: "winlogbeat.template.json"
      # Overwrite existing template
      #overwrite: false
    tls:
      certificate_authorities: ["C:/Program Files/Winlogbeat/logstash-forwarder.crt"]
logging:
  to_files: true
  files:
    path: C:/ProgramData/winlogbeat/Logs
  level: info

I read in googled post that someone had copied winlogbeat.template.json to the ELK server, and initialized it with "curl -XPOST 'http://10.0.101.101:9200/_template/winlogbeat?pretty' -d @/home/me/winlogbeat.template.json", and that that made his setup work. I tried that, but no joy.

Help would be appreciated.

[andrewkroh]

What version of Winlogbeat are you using?

Have you following the Getting Started guide for that version? 1.2 Getting Started or [5.0 Getting Started] (https://www.elastic.co/guide/en/beats/winlogbeat/5.0/winlogbeat-getting-started.html) depending on version.

It looks like you are using Logstash and those template options you set under the logstash output and not supported. Those are for the elasticsearch output only. You must manually install the index template to your Elasticsearch server (instructions are included in the Getting Started guide).

[Diggy]

Andrew,

I'm using Winlogbeat-1.2.3.

I did follow the directions in the Getting Started Guide, with the exception of loading the template (I think). Forgive me, but I'm not seeing how to manually install the index template to my Elasticsearch server in the Guide.

Diggy

[Diggy]

Well, I guess I've done something partly right, as I'm now receiving logs. However, the information is rather worthless, as follows:

{
"_index": "logstash-2016.08.08",
"_type": "eventlog",
"_id": "AVZrOnCmusKqwnAhAmft",
"score": null,
"source": {
"message": "ڟ.f\xCDU\x88ë\u0019\vF@\xAE\xC7M\xEB\xE61\xC6\xD6\u0019M\u000E\u0003\xCB3v\xC9\xC0\x95\e\u000F\xC62\x83>v\u0004\xE9*\xFC\xEFy\xFE\xDC\xDAYY\xAD\x94Z\xB1\e\u0003-'{\u000E;L\xFEbu\xFC\t\u0000\u0000\xFF\xFF3\t\x88\xC92W\u0000\u0000\u0000\u00012C\u0000\u0000\u0001\u0011x^l\x90Ok\xF30\f\xC6\xFB~\u0013\xA3\xD3;Ȇ\x93\x96\xD2\xE6\xD4\xC1.\e\xECV\u0018\xECR\\[m\xCC\xE2?\xD8\xCA\xD2\u0011\xFAI\xF6e\xA7v\xCB\xD8!F\u0017=\xD2\xCFң\xEAi6\x9B\xFD\xE3\xF8\u001C`㐔Q\xA4\xA0\u001E\x80>\"B\r\xBD\xF5\xF8\x8E\x9E\xDAp\x84\u0002\xF6\xA8\xE8[\xE4\xFC\x9A\x9C\vС\xF3\xAC\x96\u00058\xCCY\u001D/ضA\xF1\x80\xF9\x8DB\u0014/֛Ћg幖D\xA3\xB2\xC0\x93%4\xA2\xB7\xD4\b\u001D\f\x8A\xFF\xF2d$\xBF\xEA\x807\xBFs\u0006hB&\xAF\xDC\xE5Ǭx*a&9\xE7\x86\t\x917ِ\xE5\u0015H\xB9\xC8@%\xCB\xE5\xAD\\ql\xCBe=\xD5Uy\xC7\u0013^\x99N\xA8C2;߹=&n],\xE4\xA2b}\xD2\xF3\xD5\xFE\xCE\u001A\xA8\xD7R\xAE\vhYh\u0019z􇐜\"\e<\xA3|\x8F\xDD\xCFR\xF71\xB6V\x8F\x85\u001C\xBA\xA4q\xACM߄y\u001D\\\xEC\b\xD3\xD8\xF8\xD7\xEE\xF9+\u0000\u0000\xFF\xFF\xA5{\x88\xC92W\u0000\u0000\u0000\u00012C\u0000\u0000\u0001\u000Fx^lP\xC1J\u00031\u0010\xAD\u007F\u0012rR\xA8\x92ݖ\xD2\u0017\u0005o\u0005\xC1˒&\xD3np\x93\tɬ[Y\xFA%\xFE\xAC\xD3\xD6",
"tags": [
"_jsonparsefailure",
"windows",
"eventlog"
],
"@version": "1",
"@timestamp": "2016-08-08T17:36:46.788Z",
"host": "10.0.101.144",
"port": 55281,
"type": "eventlog"
},
"fields": {
"@timestamp": [
1470677806788
]
},
"sort": [
1470677806788
]
}

I note the "_jsonparsefailure" tag in the output. Is that due to not installing the index template?

[andrewkroh]

@Diggy

I'm not seeing how to manually install the index template to my Elasticsearch server in the Guide.

No problem, here's the location. Step 4: Loading the Index Template in Elasticsearch | Winlogbeat Reference [1.2] | Elastic

@Diggy

I note the "_jsonparsefailure" tag in the output. Is that due to not installing the index template?

No, that sounds more like a Logstash configuration issue. What's your Logstash config look like? Here's our recommended minimal configuration for use with Beats. Installing Logstash (Optional) | Beats Platform Reference [1.2] | Elastic

[Diggy]

Andrew,

I did the manual install of the index. On the sending side, of course.

Here's the relevant part of my configs:

input {
  beats {
    port => 5044
    type => "log"
    tags => ['log']
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

output {
  if [@metadata][beat] {
    elasticsearch {
      hosts => ["http://10.0.101.101:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  } else {
    elasticsearch { hosts => ["10.0.101.101:9200"] }
    stdout { codec => rubydebug }
  }
}

Diggy

[andrewkroh]

@Diggy

Here's the relevant part of my configs

How about the parts of the config that are not relevant? I suspect there might be some invalid conditionals in your filters. The type => "log" does nothing in this context because type is already set by Winlogbeat. The type on those events will be either eventlogging or wineventlog depending on the Windows operating system.

[Diggy]

Not sure where I might need to make the change. I've posted my configs here: http://pastebin.com/Q2pn0B67. If it's not too big a pita for you to take a look, that would be greatly appreciated. I'm sure your eyes will roll when you see them .

[andrewkroh]

@Diggy

"tags": [ "_jsonparsefailure", "windows", "eventlog" ],

@pastebin

input {
  tcp {
    type => "eventlog"
    port => "5140"
    tags => ['windows', 'eventlog']
    codec => "json"
  }
}

Based on the tags in the event and the configuration you posted to pastebin, I'd say you are sending the Winlogbeat data to the Logstash tcp input and not the beats input. Please double check that the port number you are using in your Winlogbeat config is the port number assigned to the Logstash beats input.

[Diggy]

Yes, you're right, a mis-config. I have specified port 5044 in winlogbeat.yml, which is the beats port specified in the Logstash beats input. Now, I'm not receiving any logs again. Huh?

[Diggy]

This morning, I installed Topbeat on a Win2k8 server, with the same results - I get no output in ELK (using the Logstash output). As the Windows platform makes up 40% of our infrastructure, I'd really like to get this working, so your help would be greatly appreciated.

[Diggy]

Well, solved, sort of (at least where Topbeat is concerned). On my Linux servers, I have TLS turned on in Topbeat.yml, as follows:

tls:
   certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

If I set up something similar in Windows, as in:

tls:
  certificate_authorities: ["C:\Program Files\Topbeat\logstash-forwarder.crt"]

the Topbeat service errors, and won't start. However, if I comment out "certificate_authorities", and instead set "insecure: true", then the service starts, and I get output in ELK. Does TLS have to be set up differently to work with Windows, or is it not supported?

[andrewkroh]

Try using single quotes around your Windows paths. Otherwise you would need to escape the \ characters to be correct.

Another option for Windows is to just use forward slashes in your paths. They are automatically converted to backslashes.

[Diggy]

Thanks, Andrew. Single quotes did the trick.