# Windows Event Logs - missing fields

**URL:** <https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706>\
**Category:** Logstash\
**Created:** [September 21, 2015, 2:15pm UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706 "2015-09-21T14:15:38Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![selspiero](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@selspiero](https://discuss.elastic.co/u/selspiero)\
**Post date:** [September 21, 2015, 2:15pm UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/1 "2015-09-21T14:15:38Z")

</div>

Hi there,

I am currently in the process of setting up ELK for windows event logs for a production environment. I am just testing every thing, and i think i have the a nice simple configuration to start with:

nxlog is shipping the logs with this config:

```
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Input in>
    Module im_msvistalog
	Query	<QueryList>\
                    	<Query Id="0">\
                       		<Select Path="Microsoft-Windows-TaskScheduler/Operational">*</Select>\
                	</Query>\
						<Query Id="1">\
                       		<Select Path="Application">*</Select>\
                	</Query>\
						<Query Id="2">\
                       		<Select Path="System">*</Select>\
                	</Query>\
		</QueryList>

</Input>

<Output out>
    Module om_tcp
    Host 192.168.0.1
    Port 5140
</Output>

<Route 1>
    Path in => out
</Route>

```

Then the logstash.conf is as follows:

```
input {  
    tcp {
        type => 'Win32-Eventlog'
        port => "5140"
        codec => multiline {
        pattern => "^%{TIMESTAMP_ISO8601} "
        negate => true
        what => previous
    }
        tags => ["eventlog"]
    }
}

output { 
    elasticsearch { 

        host => "localhost"
        cluster => "eventlogs"
        node_name => "servername"
    } 
} # end output

```

So as i said pretty simple, and i can see the logs are in ES and Kibana. The only problem i have, is that i don't have all of the fields i would expect, so i only really have 'message' which contains the log information, and 'host' which is the server it comes from

From what i have read, i would expect to see severity,event id,etc,etc - but not getting any of these.

Any ideas? will i have to use the mutate option in the filter to get these? and if so, could you point me in the direction of an example?

Thanks!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 21, 2015, 2:40pm UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/2 "2015-09-21T14:40:14Z")

</div>

This is probably more of an NXLog question. I suspect you need to configure it to encode outgoing messages as JSON. Consequently you also need to configure Logstash to interpret inbound messages as JSON, either by having the tcp input use the json code or by using a json filter.

---

<div class="post-metadata">

**Author:** ![selspiero](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@selspiero](https://discuss.elastic.co/u/selspiero)\
**Post date:** [September 21, 2015, 2:58pm UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/3 "2015-09-21T14:58:55Z")

</div>

ah nice, thanks for the reply.

So as for the logstash config - type should just be 'json' ? i'll look at the nxlog config now

thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 21, 2015, 3:00pm UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/4 "2015-09-21T15:00:29Z")

</div>

The _type_ doesn't matter. That's just a string you choose. But the _codec_ should be json, or you can use a json filter.

---

<div class="post-metadata">

**Author:** ![selspiero](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@selspiero](https://discuss.elastic.co/u/selspiero)\
**Post date:** [September 22, 2015, 5:39am UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/5 "2015-09-22T05:39:29Z")

</div>

Can i use multiple codecs?

using multiline as it fixes an issue i have with some eventlogs being split over multiple entires when i used the json codec, or the json\_lines one. Or do you think this would have happened as i wasn't specifying json in nxlog?

thanks again for the replies

Ian

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 22, 2015, 5:55am UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/6 "2015-09-22T05:55:51Z")

</div>

> Can i use multiple codecs?

Not with the current Logstash. It might be planned for future releases.

> Or do you think this would have happened as i wasn't specifying json in nxlog?

Not sure exactly what you mean, but IIRC from the little work I did with NXLog about six months ago it works about the same as Logstash in that it internally keeps a messages as a set of key/value pairs. Those pairs are serialized upon output, but if the output format/codec doesn't support key/value-style structures it'll (more or less) just emit the "message" part and scrap the rest.

So, if NXLog collects structured data you'll want to use an output codec that supports structured data. On the Logstash side you'll want to use a codec and/or filter that matches what you get over the wire.

If you want more concrete help connecting NXLog and Logstash, please be more explicit about what Logstash is receiving. Starting with raw input without filters applied probably helps. People here know Logstash but usually not NXLog.

---

<div class="post-metadata">

**Author:** ![selspiero](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@selspiero](https://discuss.elastic.co/u/selspiero)\
**Post date:** [September 22, 2015, 6:00am UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/7 "2015-09-22T06:00:42Z")

</div>

Makes sense, and sounds like what is happening.

I'll get nxlog sending JSON properly and try using the json\_lines codec again.

When you say raw input - how would i gather that? sorry for the basic questions!

Cheers  
E

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 22, 2015, 7:49am UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/8 "2015-09-22T07:49:14Z")

</div>

> When you say raw input - how would i gather that?

Skip all input codecs and filters and use a `stdout { codec => rubydebug }` output.

---

<div class="post-metadata">

**Author:** ![selspiero](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@selspiero](https://discuss.elastic.co/u/selspiero)\
**Post date:** [September 22, 2015, 9:40am UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/9 "2015-09-22T09:40:42Z")

</div>

Have managed to resolve, so thank you very much for your help.

Just to confirm, my nxlog config wasn't quite right. I have changed it to:

```
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module xm_json
</Extension>

<Input in>
    Module im_msvistalog

    
Exec $raw_event = to_json();
    
    Query <QueryList>\
                            <Query Id="0">\
                          <Select Path="Microsoft-Windows-TaskScheduler/Operational">*</Select>\
                  </Query>\
            <Query Id="1">\
                          <Select Path="Application">*</Select>\
                  </Query>\
            <Query Id="2">\
                          <Select Path="System">*</Select>\
                  </Query>\
    </QueryList>

</Input>

<Output out>
    Module om_tcp
    Host server
    Port 5140
</Output>

<Route 1>
    Path in => out
</Route>

```

Then changed the codec in my logstash.conf from multiline to json\_lines

All looks good - the only remaining thing to do is change the EventID field, its currently showing as a number so event id shows as 1,450 rather than 1450.

Thanks again!

E

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:28am UTC](https://discuss.elastic.co/t/windows-event-logs-missing-fields/29706/10 "2017-07-06T05:28:25Z")

</div>