# Windows nxlog filtering

**URL:** https://discuss.elastic.co/t/windows-nxlog-filtering/40061
**Category:** Logstash
**Created:** [January 25, 2016, 8:16pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061 "2016-01-25T20:16:20Z")
**Posts on this page:** 14
**Page:** 1

<div class="post-metadata">

### Author: ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)
#### Post date: [January 25, 2016, 8:16pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/1 "2016-01-25T20:16:20Z")

</div>

So now that I am receiving logs from my Windows server with NXLOG I an having trouble filtering it and making it useful. Most of the configurations below is from this example. [https://gist.github.com/stuart-warren/6726081](https://gist.github.com/stuart-warren/6726081) I have tried multiple other ways and it still ever ends up correct.

Here is my NXLOG config:

```
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
Module xm_json
</Extension>

<Input in>
    Module im_msvistalog
# For windows 2003 and earlier use the following:
# Module im_mseventlog
</Input>

<Output out>
    Module om_tcp
    Host 192.168.200.90
    Port 5001
</Output>

<Route 1>
    Path in => out
</Route>

```

Here is my input file:

```
input {
  tcp {
    port => 5001
    codec => "json"
    type => "nxlog-json"
    tags => ["windows","nxlog"]
    }
}
```

---

<div class="post-metadata">

### Author: ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)
#### Post date: [January 25, 2016, 8:16pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/2 "2016-01-25T20:16:30Z")

</div>

And here is my filter:

```
filter {

  if [type] == "nxlog-json" {
    date {
      match => ["[EventTime]", "YYYY-MM-dd HH:mm:ss"]
      timezone => "Europe/London"
    }
    mutate {
        rename => ["AccountName", "user"]
        rename => ["AccountType", "[eventlog][account_type]" ]
        rename => ["ActivityId", "[eventlog][activity_id]" ]
        rename => ["Address", "ip6"]
        rename => ["ApplicationPath", "[eventlog][application_path]" ]
        rename => ["AuthenticationPackageName", "[eventlog][authentication_package_name]" ]
        rename => ["Category", "[eventlog][category]" ]
        rename => ["Channel", "[eventlog][channel]" ]
        rename => ["Domain", "domain"]
        rename => ["EventID", "[eventlog][event_id]" ]
        rename => ["EventType", "[eventlog][event_type]" ]
        rename => ["File", "[eventlog][file_path]" ]
        rename => ["Guid", "[eventlog][guid]" ]
        rename => ["Hostname", "hostname"]
        rename => ["Interface", "[eventlog][interface]" ]
        rename => ["InterfaceGuid", "[eventlog][interface_guid]" ]
        rename => ["InterfaceName", "[eventlog][interface_name]" ]
        rename => ["IpAddress", "ip"]
        rename => ["IpPort", "port"]
        rename => ["Key", "[eventlog][key]" ]
        rename => ["LogonGuid", "[eventlog][logon_guid]" ]
        rename => ["Message", "message"]
        rename => ["ModifyingUser", "[eventlog][modifying_user]" ]
        rename => ["NewProfile", "[eventlog][new_profile]" ]
        rename => ["OldProfile", "[eventlog][old_profile]" ]
        rename => ["Port", "port"]
        rename => ["PrivilegeList", "[eventlog][privilege_list]" ]
        rename => ["ProcessID", "pid"]
        rename => ["ProcessName", "[eventlog][process_name]" ]
        rename => ["ProviderGuid", "[eventlog][provider_guid]" ]
        rename => ["ReasonCode", "[eventlog][reason_code]" ]
        rename => ["RecordNumber", "[eventlog][record_number]" ]
        rename => ["ScenarioId", "[eventlog][scenario_id]" ]
        rename => ["Severity", "level"]
        rename => ["SeverityValue", "[eventlog][severity_code]" ]
        rename => ["SourceModuleName", "nxlog_input"]
        rename => ["SourceName", "[eventlog][program]" ]
        rename => ["SubjectDomainName", "[eventlog][subject_domain_name]" ]
        rename => ["SubjectLogonId", "[eventlog][subject_logonid]" ]
        rename => ["SubjectUserName", "[eventlog][subject_user_name]" ]
        rename => ["SubjectUserSid", "[eventlog][subject_user_sid]" ]
        rename => ["System", "[eventlog][system]" ]
        rename => ["TargetDomainName", "[eventlog][target_domain_name]" ]
        rename => ["TargetLogonId", "[eventlog][target_logonid]" ]
        rename => ["TargetUserName", "[eventlog][target_user_name]" ]
        rename => ["TargetUserSid", "[eventlog][target_user_sid]" ]
        rename => ["ThreadID", "thread"]

    }
    mutate {
        remove_field => [
                    "CurrentOrNextState",
                    "Description",
                    "EventReceivedTime",
                    "EventTime",
                    "EventTimeWritten",
                    "IPVersion",
                    "KeyLength",
                    "Keywords",
                    "LmPackageName",
                    "LogonProcessName",
                    "LogonType",
                    "Name",
                    "Opcode",
                    "OpcodeValue",
                    "PolicyProcessingMode",
                    "Protocol",
                    "ProtocolType",
                    "SourceModuleType",
                    "State",
                    "Task",
                    "TransmittedServices",
                    "Type",
                    "UserID",
                    "Version"
                    ]
    }
  }

}

```

And lastly here is my ouput:

```
output {
  elasticsearch { hosts => ["<ES1 server IP>", "<ES2 server IP>", "<ES3 server IP>"] }
# stdout { codec => rubydebug }
  }
```

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 25, 2016, 9:12pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/3 "2016-01-25T21:12:22Z")

</div>

What's the raw input from NXLog (`stdout { codec => rubydebug }` _without_ filters would be great)? What do you get now (`stdout { codec => rubydebug }` _with_ filters)?

> ```
> match => ["[EventTime]", "YYYY-MM-dd HH:mm:ss"]
> 
> ```

Remove the square brackets around the field name.

---

<div class="post-metadata">

### Author: ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)
#### Post date: [January 25, 2016, 9:24pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/4 "2016-01-25T21:24:20Z")

</div>

When you say ray input thats what you can see in ES? on JSON tab?

With filtering off:

```
{
  "_index": "logstash-2016.01.25",
  "_type": "nxlog-json",
  "_id": "AVJ6qJvUQ7EodV25mwQL",
  "_score": null,
  "_source": {
    "tags": [
      "_jsonparsefailure",
      "windows",
      "nxlog",
      "packetfilter"
    ],
    "@version": "1",
    "@timestamp": "2016-01-25T21:20:16.116Z",
    "host": "192.168.56.102",
    "port": 49508,
    "type": "nxlog-json",
    "program": "Id",
    "protocol": null
  },
  "fields": {
    "@timestamp": [
      1453756816116
    ]
  },
  "highlight": {
    "host.raw": [
      "@kibana-highlighted-field@192.168.56.102@/kibana-highlighted-field@"
    ],
    "host": [
      "@kibana-highlighted-field@192.168.56.102@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1453756816116
  ]
}

```

With filtering on:

```
{
  "_index": "logstash-2016.01.25",
  "_type": "nxlog-json",
  "_id": "AVJ6q2J_2MYPny3yVFuU",
  "_score": null,
  "_source": {
    "tags": [
      "_jsonparsefailure",
      "windows",
      "nxlog",
      "packetfilter"
    ],
    "@version": "1",
    "@timestamp": "2016-01-25T21:23:18.134Z",
    "host": "192.168.56.102",
    "port": 49559,
    "type": "nxlog-json",
    "program": "Id",
    "protocol": null
  },
  "fields": {
    "@timestamp": [
      1453756998134
    ]
  },
  "highlight": {
    "host.raw": [
      "@kibana-highlighted-field@192.168.56.102@/kibana-highlighted-field@"
    ],
    "host": [
      "@kibana-highlighted-field@192.168.56.102@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1453756998134
  ]
}
```

---

<div class="post-metadata">

### Author: ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)
#### Post date: [January 26, 2016, 2:08am UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/5 "2016-01-26T02:08:40Z")

</div>

Anyone have any other thoughts? Im still not getting any useful data out of my Windows servers.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 26, 2016, 6:55am UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/6 "2016-01-26T06:55:47Z")

</div>

> When you say ray input thats what you can see in ES? on JSON tab?

I suggested that you post the output of `stdout { codec => rubydebug }`. Anyway, the presence of the `_jsonparsefailure` tag suggests that Logstash isn't able to parse the payload as JSON. I thought it would fall back to the plain codec but apparently not. Disable `codec => json` for the tcp input so that we can have a look at the actual message.

---

<div class="post-metadata">

### Author: ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)
#### Post date: [January 26, 2016, 4:40pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/7 "2016-01-26T16:40:16Z")

</div>

Ok i disabled the codec in the input file as well as in the nxlog configuration. Here is what im seeing on the JSON tab now.

```
{
  "_index": "logstash-2016.01.26",
  "_type": "nxlog-json",
  "_id": "AVJ-zCriwZTWU8UbfsWS",
  "_score": null,
  "_source": {
    "@version": "1",
    "@timestamp": "2016-01-26T16:37:35.449Z",
    "host": "192.168.56.102",
    "port": 50266,
    "type": "nxlog-json",
    "tags": [
      "windows",
      "nxlog",
      "packetfilter"
    ],
    "program": "08",
    "protocol": null
  },
  "fields": {
    "@timestamp": [
      1453826255449
    ]
  },
  "highlight": {
    "host.raw": [
      "@kibana-highlighted-field@192.168.56.102@/kibana-highlighted-field@"
    ],
    "host": [
      "@kibana-highlighted-field@192.168.56.102@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1453826255449
  ]
}
```

---

<div class="post-metadata">

### Author: ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)
#### Post date: [January 26, 2016, 7:06pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/8 "2016-01-26T19:06:56Z")

</div>

I have also disabled all filters. So I technically only have an input file (Only concerned with the top TCP input):

```
input {
  #Windows Servers
  tcp {
    port => 5002
    type => "nxlog-json"
    tags => ["windows","nxlog"]
  }

  #tcp syslog stream via 5140
  #Local firewall forwards TCP/UDP port 514 to 5140
  tcp {
    port => 5140
        type => "syslog"
  }
}

input {
  #udp syslogs tream via 5140
  #Local firewall forwards TCP/UDP port 514 to 5140
  udp {
    port => 5140
    type => "syslog"
  }
}

```

And an output file:

```
output {
  elasticsearch { hosts => ["192.168.56.86:9200", "192.168.56.87:9200", "192.168.56.89:9200"] }
  stdout { codec => rubydebug }
  }
```

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 26, 2016, 7:47pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/9 "2016-01-26T19:47:36Z")

</div>

Two avenues to explore:

- Use e.g. Wireshark to look into exactly what NXLog is sending so that we can establish whether the problem is on the NXLog or the Logstash side.
- Double and triple check what configuration you're actually using. I don't understand what the `program` and `protocol` fields come from. And the packetfilter tag? There's nothing like that in the configuration you've shown. Starting Logstash with `--verbose` or even `--debug` could give more clues in this department.

---

<div class="post-metadata">

### Author: ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)
#### Post date: [January 26, 2016, 8:12pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/10 "2016-01-26T20:12:28Z")

</div>

> [@magnusbaeck](#):
>
> And the packetfilter tag?

I was thinking the same thing. I have no idea where that tag is coming from. Im going to trash everything thats within my conf.f directory and start from scratch.

---

<div class="post-metadata">

### Author: ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)
#### Post date: [January 26, 2016, 8:38pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/11 "2016-01-26T20:38:51Z")

</div>

I think I got something by blowing away all of my configs. I think there was a gok file in a subfolder that I didnt create that was doing something weird. Here is the raw messages that im getting now:

{"EventTime":"2016-01-26 12:34:12","Hostname":"","Keywords":-9223372036854775808,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":2,"SourceName":"Microsoft-Windows-ServerManager-ManagementProvider","ProviderGuid":"{C2E6D0D9-5DF8-4C77-A82B-C96C84579543}","Version":0,"Task":1,"OpcodeValue":2,"RecordNumber":447190,"ProcessID":158280,"ThreadID":154872,"Channel":"Microsoft-Windows-ServerManager-MgmtProvider/Operational","Domain":"NT AUTHORITY","AccountName":"NETWORK SERVICE","UserID":"NETWORK SERVICE","AccountType":"Well Known Group","Message":"Unloading the management provider","Category":"Provider initialization","Opcode":"Stop","EventReceivedTime":1453840453,"SourceModuleName":"eventlog","SourceModuleType":"im\_msvistalog"}

---

<div class="post-metadata">

### Author: ![Diggy](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@Diggy](https://discuss.elastic.co/u/Diggy)
#### Post date: [January 26, 2016, 9:17pm UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/12 "2016-01-26T21:17:27Z")

</div>

I'm a real newb to logstash, and so can't be of much help to folks (yet). However, I can contribute the following cautionary tale: I cobbled together my logstash configs mainly based on 'net posts. Slowly, but surely (and with help from Magnus, and lots of experimentation), I've come to realize that the configs have to be treated kindly, and that more config files aren't necessarily better. I, too, had a filter like yours in place, and couldn't understand why I wasn't getting the Windows Eventlog fields that I expected. Once I blew away that specific Eventlog-related config, I started to see pretty much what you're seeing:

January 26th 2016, 15:46:44.436 Hostname:sawintest01.mydomain.com EventTime:2016-01-26 15:47:00 Keywords:-9,223,372,036,854,775,808 EventType:INFO SeverityValue:2 Severity:INFO EventID:318 SourceName:Microsoft-Windows-TaskScheduler ProviderGuid:{DE7B24EA-73C8-4A09-985D-5BDADCFA9017} Version:0 Task:318 OpcodeValue:2 RecordNumber:62,887 ActivityID:{018B1C58-F800-0003-E19C-FAF7304ED101} ProcessID:832 ThreadID:12,728 Channel:Microsoft-Windows-TaskScheduler/Operational Domain:NT AUTHORITY AccountName:SYSTEM UserID:SYSTEM AccountType:User Message:Task Scheduler shutdown Task Engine "S-1-5-18:NT AUTHORITY\System:Service:" process. Category:Task engine properly shut down Opcode:Stop TaskEngineName:S-1-5-18:NT AUTHORITY\System:Service: EventReceivedTime:1,453,841,221SourceModuleName:eventlog SourceModuleType:im\_msvistalog @version:1 @timestamp:January 26th 2016, 15:46:44.436 host:192.168.192.208 port:64,638 type:eventlog tags:windows, eventlog \_id:AVJ\_sEZ5O7Q5gAnAwcKB \_type:eventlog \_index:logstash-2016.01.26 \_score:

Beautiful, just what a want to see!

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 27, 2016, 7:13am UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/13 "2016-01-27T07:13:18Z")

</div>

> I think I got something by blowing away all of my configs. I think there was a gok file in a subfolder that I didnt create that was doing something weird. Here is the raw messages that im getting now:

Okay, now we're talking. You should be able to enable the json codec and get reasonable results.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:14am UTC](https://discuss.elastic.co/t/windows-nxlog-filtering/40061/14 "2017-07-06T05:14:12Z")

</div>