Issue shipping Winlogbeat 5.x to Elasticsearch 2.x

[Ethan]

This is a really basic question, but I have done some googling with no positive results.

I just spun up an install of winlogbeat using the instructions in the configuration documentation. It was all fine and dandy, and is now running .

My issue is that my logs are not being shipped to my elasticsearch instance. I have no TLS enabled. Here is the error I am seeing in the logs:

2016-04-13T18:23:35Z DBG ES Ping(url=http://10.8.112.215:9200/tmp/winlogbeat, timeout=1m30s)
2016-04-13T18:23:35Z DBG Ping request failed with: 404 Not Found

When I try to access this URL, I get the following:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No feature for name [winlogbeat]"}],"type":"illegal_argument_exception","reason":"No feature for name [winlogbeat]"},"status":400}

Here is my winlogbeat.yml (relevant portion)

output:

  ### Elasticsearch as output
  elasticsearch:
    hosts: ["10.8.112.215"]

So my elasticsearch is available, but the feature winlogbeat is not available. Did I miss some simple config somewhere in the setup?

For the record, I already have logstash running on the same box as Elasticsearch with no issues.

Thanks in advance!

[andrewkroh]

Hi @Ethan, please post your complete configuration file to http://pastebin.com and share it here.

@Ethan

ES Ping(url=http://10.8.112.215:9200/tmp/winlogbeat, timeout=1m30s)

Something is causing tmp/winlogbeat to be appended to URL, and since this isn't a valid endpoint on the Elasticsearch server, it's returning 404. Did you set the path: tmp/winlogbeat in your config file?

[Ethan]

Andrew,

Here is the config:

winlogbeat.yml

I did some investigating and I accidentally configured parameters in the Path section for of the winlogbeat.yml that were intended Logstash; I believe this caused the original break. I commented that section out and am now seeing my winlogbeat.log files being created, and executing -configtest shows a 200 response code for the ES Ping.

Now my final step is getting the files to actually ship; I see many log entries filling up my \Log\winlogbeat.X directory, but no entries are visible in Kibana, despite the -configtest giving me an all good.

Here is the head of each log file, if that helps. I will continue to troubleshoot and see if I can solve this.

2016-04-13T21:34:59Z DBG  Disable stderr logging
2016-04-13T21:34:59Z DBG  configuration []
2016-04-13T21:34:59Z DBG  filters: 
2016-04-13T21:34:59Z DBG  Filters: 
2016-04-13T21:34:59Z DBG  Configuration validated. config=&{{[map[name:Application ignore_older:72h] map[name:Security] map[name:System]] {} C:/ProgramData/winlogbeat/.winlogbeat.yml} map[shipper:map[tags:[SAP ECC1]] logging:map[level:debug to_files:true files:map[rotateeverybytes:10485760 path:F:/Winlogbeat/Logs]] winlogbeat:map[registry_file:C:/ProgramData/winlogbeat/.winlogbeat.yml event_logs:[map[name:Application ignore_older:72h] map[name:Security] map[name:System]]] output:map[elasticsearch:map[template:map[name:winlogbeat path:F:/Winlogbeat/Winlogbeat/winlogbeat.template.json overwrite:true] hosts:[10.8.112.215]]]]}
2016-04-13T21:34:59Z INFO State will be read from and persisted to C:\ProgramData\winlogbeat\.winlogbeat.yml
2016-04-13T21:34:59Z INFO Setup Beat: winlogbeat; Version: 5.0.0-alpha1
2016-04-13T21:34:59Z DBG  Initializing output plugins
2016-04-13T21:34:59Z INFO GeoIP disabled: No paths were set under shipper.geoip.paths
2016-04-13T21:34:59Z DBG  ES Ping(url=http://10.8.112.215:9200, timeout=1m30s)
2016-04-13T21:34:59Z DBG  Ping status code: 200
2016-04-13T21:34:59Z INFO Loading template enabled. Trying to load template: F:/Winlogbeat/Winlogbeat/winlogbeat.template.json
2016-04-13T21:34:59Z DBG  HEAD http://10.8.112.215:9200/_template/winlogbeat <nil>
2016-04-13T21:34:59Z INFO Existing template will be overwritten, as overwrite is enabled.
2016-04-13T21:34:59Z INFO Elasticsearch template with name 'winlogbeat' loaded
2016-04-13T21:34:59Z INFO Activated elasticsearch as output plugin.
2016-04-13T21:34:59Z DBG  Create output worker
2016-04-13T21:34:59Z DBG  No output is defined to store the topology. The server fields might not be filled.
2016-04-13T21:34:59Z INFO Publisher name: MYSERVER
2016-04-13T21:34:59Z INFO Flush Interval set to: 1s
2016-04-13T21:34:59Z INFO Max Bulk Size set to: 50
2016-04-13T21:34:59Z DBG  create bulk processing worker (interval=1s, bulk size=50)
2016-04-13T21:34:59Z DBG  Using highest priority API, wineventlog, for event log Application
2016-04-13T21:34:59Z DBG  Initialized EventLog[Application]
2016-04-13T21:34:59Z DBG  Using highest priority API, wineventlog, for event log Security
2016-04-13T21:34:59Z DBG  Initialized EventLog[Security]
2016-04-13T21:34:59Z DBG  Using highest priority API, wineventlog, for event log System
2016-04-13T21:34:59Z DBG  Initialized EventLog[System]
2016-04-13T21:34:59Z INFO winlogbeat start running.
2016-04-13T21:34:59Z DBG  WinEventLog[System] using subscription query=<QueryList>

[andrewkroh]

Everything you have provided indicates that it is working. But the logs cut off before it starts providing any indication of whether or not data is being sent to elasticsearch. Maybe post some more log data...

When you stop the process it will dump out a bunch of metrics to the end of the log file. These metrics will indicate how many events were sent to Elasticsearch. It's a JSON map and the publishedEvents.total value tells you how many events were read and published to ES.

You can also use the Elasticsearch API to get the total in the index. In your case the URL would be http://10.8.112.215:9200/winlogbeat-*/_count?pretty.

Did you install the Elasticsearch index template before you ran Winlogbeat? https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html There is a known issue where the template is not automatically installed by the Beat when running on Windows. This should be fixed in 5.0.0-alpha2.

[Ethan]

Andrew,

Here is a snippet of a section of the log file, indicating Events are being read:

 "computer_name": "HOSTNAME",
  "event_data": {
    "Binary": "8E450000140000000D00000043004100500041004D004C00410042004500430043003100000000000000",
    "param1": "8009030c",
    "param2": "14",
    "param3": "AcceptSecurityContext failed. The Windows error code indicates the cause of failure.",
    "param4": "The logon attempt failed ",
    "param5": " [CLIENT: \u003clocal machine\u003e]"
  },
  "event_id": 17806,
  "keywords": [
    "Classic"
  ],
  "level": "Error",
  "log_name": "Application",
  "message": "SSPI handshake failed with error code 0x8009030c, state 14 while establishing a connection with integrated security; the connection has been closed. Reason: AcceptSecurityContext failed. The Windows error code indicates the cause of failure. The logon attempt failed   [CLIENT: \u003clocal machine\u003e]",
  "record_number": "307373",
  "source_name": "MSSQLSERVER",
  "tags": [
    "SAP",
    "ECC1"
  ],
  "task": "Logon",
  "type": "wineventlog"
}
2016-04-14T14:16:33Z DBG  Publish: {
  "@timestamp": "2016-04-14T14:16:32.000Z",
  "beat": {
    "hostname": "HOSTNAME",
    "name": "HOSTNAME"
  },
  "computer_name": "HOSTNAME",
  "event_data": {
    "Binary": "144800000E0000000D00000043004100500041004D004C004100420045004300430031000000070000006D00610073007400650072000000",
    "param1": " [CLIENT: \u003clocal machine\u003e]"
  },

And here is the published events entry at the end of the file:

2016-04-14T14:24:09Z INFO publishedEvents={"Application": 41671, "Security": 31152, "System": 41, "failures": 0, "total": 72864}

Here is the return for http://10.8.112.215:9200/winlogbeat-*/_count?pretty:

{
"count" : 0,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
}
}

I also see the following error repeatedly throughout the log:

2016-04-14T14:16:33Z WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"Failed to parse mapping [default]: No handler for type [text] declared on field [message]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [text] declared on field [message]"}

It would appear some mapping is incorrect, and the inbound messages are being bounced. The only thing that confuses me is why Elasticsearch is reporting 0 input, 0 failed, 0 succeeded when the logs on the source system clearly show publish events.

[andrewkroh]

It looks like you are using the Winlogbeat 5.0.0-alpha1 with Elasticsearch 1.x or 2.x? There is an incompatibility with the winlogbeat.template.json index template provided in Winlogbeat v5 and earlier versions of Elasticsearch because the template uses the text keyword that was introduced in ES v5.

To continue to use Winlogbeat 5.x with ES 1.x or 2.x you'll need to grab the index template provided in Winlogbeat 1.x and install it to ES.

Cleanup:

1. Stop Winlogbeat.
2. Delete the .winlogbeat.yml registry file so that it reindexes your data when it restarts.
3. Delete any data in your index. curl -XDELETE http://es:9200/winlogbeat-*
4. Delete the current mapping. curl -XDELETE http://es:9200/_template/winlogbeat

Install 1.x index template:

1. Replace the file you have at F:/Winlogbeat/Winlogbeat/winlogbeat.template.json with the file from Winlogbeat 1.X.
2. Start Winlogbeat and it should install the template to ES for you.

In your config I would set overwrite: false. I think it's better for you to be in control of changes to the index template.

[Ethan]

Andrew,

Everything is working now. Thank you for your excellent assistance!

Take care!