Winlogbeat and Logstash Input Codec

Elastic Stack Beats
1

We are evaluating ELK as a distributed monitoring system. I have installed ELK and have some syslogs shipping from various systems.

I have just tried installing winlogbeat 1.1.1 on a Windows 8 VM in Parallels.

Logstash 2.2

I am seeing traffic all the way into the ELK server (via tcpdump) but not getting consistent records - i am seeing this error stanza regularly - seems partial log events are getting passed along. Anything I should set on the client to only send complete records? I don't see anything obvious in the configuration elements.

at [Source: [B@77ae0b1; line: 1, column: 5]>, :data=>"The Windows Filtering Platform has permitted a connection.\n\nApplication Information:\n\tProcess ID:\t\t1368\n\tApplication Name:\t\device\harddiskvolume2\windows\system32\svchost.exe\n\nNetwork Information:\n\tDirection:\t\tInbound\n\tSource Address:\t\t239.255.255.250\n\tSource Port:\t\t1900\n\tDestination Address:\t127.0.0.1\n\tDestination Port:\t\t63569\n\tProtocol:\t\t17\n\nFilter Information:\n\tFilter Run-Time ID:\t69068\n\tLayer Name:\t\tReceive/Accept\n\tLayer Run-Time ID:\t44", :level=>:error}
{:timestamp=>"2016-02-23T11:40:25.248000-0800", :message=>"JSON parse failure. Falling back to plain-text", :error=>#<LogStash::Json::ParserError: Unrecognized token 'The': was expecting ('true', 'false' or 'null')

client config:

winlogbeat:
  event_logs:
    - name: Application
      ignore_older: 72h 
    - name: Security
    - name: System

output:
  logstash:
    hosts: ["smshepherd01.rand.org:5044"]
    worker: 1
    index: winlogbeat-cthomas

  file:
    path: "C:/Data/winlogbeat"

Server logstash config related to beats

input {
  beats {
    port => "5044"
    type => "wincli-log"
    codec => "json"
  }
}

output stanza portion:

  } else if [type] == "wincli-log" {
      elasticsearch {
         hosts => ["logstashserver:9201"]
         index => "journal-%{+YYYY.MM.dd}"
     }
     stdout { codec => rubydebug { metadata => true } }
2

Remove the json codec from your Logstash configuration. The message field is not JSON, it's the plain text message from the Windows event log record.

1 Like
4

Thank you. I will try that.

I am somewhat surprised as in looking at a copy of the log information written locally using the client file stanza it certainly looks and smells like json. linklint says it's json. I am obviously not understanding what is being forwarded by the beat. I am posting another question about the 'type' value i am seeing...

here is an entry redacted a little to protect the ignorant (or innocent):

{"@metadata":{"beat":"winlogbeat-cthomas","type":"wineventlog"},"@timestamp":"2016-02-23T19:31:24.317Z","beat":{"hostname":"thomas-c-pvm","name":"thomas-c-pvm"},"category":"Filtering Platform Connection","computer_name":"thomas-c-pvm.company.com","count":1,"event_id":5156,"level":"Information","log_name":"Security","message":"The Windows Filtering Platform has permitted a connection.\n\nApplication Information:\n\tProcess ID:\t\t4488\n\tApplication Name:\t\device\harddiskvolume2\program files\winlogbeat\winlogbeat.exe\n\nNetwork Information:\n\tDirection:\t\tOutbound\n\tSource Address:\t\t10.211.55.3\n\tSource Port:\t\t50908\n\tDestination Address:\t99.99.99.99\n\tDestination Port:\t\t9999\n\tProtocol:\t\t6\n\nFilter Information:\n\tFilter Run-Time ID:\t69060\n\tLayer Name:\t\tConnect\n\tLayer Run-Time ID:\t48","record_number":"11936981","source_name":"Microsoft-Windows-Security-Auditing","type":"wineventlog"}

5

Yes, what you posted is JSON. The beat sends its events a JSON. The input codec you used applies only to the message field of that JSON event. This tells Logstash to take the contents of the message field and unmarshal it as JSON.

Take a look at the logstash-input-beats documentation. In particular see the target_field_for_codec and codec docs.

An example use case for the JSON codec would be if you were reading log lines with Filebeat and each of those lines was a JSON object.

6

Ahh, that clears up that piece. Thanks!