Windows Event Logs - missing fields

Elastic Stack Logstash
1

Hi there,

I am currently in the process of setting up ELK for windows event logs for a production environment. I am just testing every thing, and i think i have the a nice simple configuration to start with:

nxlog is shipping the logs with this config:

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Input in>
    Module      im_msvistalog
	Query	<QueryList>\
                    	<Query Id="0">\
                       		<Select Path="Microsoft-Windows-TaskScheduler/Operational">*</Select>\
                	</Query>\
						<Query Id="1">\
                       		<Select Path="Application">*</Select>\
                	</Query>\
						<Query Id="2">\
                       		<Select Path="System">*</Select>\
                	</Query>\
		</QueryList>

</Input>

<Output out>
    Module      om_tcp
    Host        192.168.0.1
    Port        5140
</Output>

<Route 1>
    Path        in => out
</Route>

Then the logstash.conf is as follows:

input {  
    tcp {
        type   => 'Win32-Eventlog'
        port   => "5140"
        codec => multiline {
        pattern => "^%{TIMESTAMP_ISO8601} "
        negate => true
        what => previous
    }
        tags => ["eventlog"]
    }
}



output { 
    elasticsearch { 

        host => "localhost"
        cluster => "eventlogs"
        node_name => "servername"
    } 
} # end output

So as i said pretty simple, and i can see the logs are in ES and Kibana. The only problem i have, is that i don't have all of the fields i would expect, so i only really have 'message' which contains the log information, and 'host' which is the server it comes from

From what i have read, i would expect to see severity,event id,etc,etc - but not getting any of these.

Any ideas? will i have to use the mutate option in the filter to get these? and if so, could you point me in the direction of an example?

Thanks!

2

This is probably more of an NXLog question. I suspect you need to configure it to encode outgoing messages as JSON. Consequently you also need to configure Logstash to interpret inbound messages as JSON, either by having the tcp input use the json code or by using a json filter.

3

ah nice, thanks for the reply.

So as for the logstash config - type should just be 'json' ? i'll look at the nxlog config now

thanks

4

The type doesn't matter. That's just a string you choose. But the codec should be json, or you can use a json filter.

5

Can i use multiple codecs?

using multiline as it fixes an issue i have with some eventlogs being split over multiple entires when i used the json codec, or the json_lines one. Or do you think this would have happened as i wasn't specifying json in nxlog?

thanks again for the replies

Ian

6

Can i use multiple codecs?

Not with the current Logstash. It might be planned for future releases.

Or do you think this would have happened as i wasn't specifying json in nxlog?

Not sure exactly what you mean, but IIRC from the little work I did with NXLog about six months ago it works about the same as Logstash in that it internally keeps a messages as a set of key/value pairs. Those pairs are serialized upon output, but if the output format/codec doesn't support key/value-style structures it'll (more or less) just emit the "message" part and scrap the rest.

So, if NXLog collects structured data you'll want to use an output codec that supports structured data. On the Logstash side you'll want to use a codec and/or filter that matches what you get over the wire.

If you want more concrete help connecting NXLog and Logstash, please be more explicit about what Logstash is receiving. Starting with raw input without filters applied probably helps. People here know Logstash but usually not NXLog.

7

Makes sense, and sounds like what is happening.

I'll get nxlog sending JSON properly and try using the json_lines codec again.

When you say raw input - how would i gather that? sorry for the basic questions!

Cheers
E

8

When you say raw input - how would i gather that?

Skip all input codecs and filters and use a stdout { codec => rubydebug } output.

1 Like
9

Have managed to resolve, so thank you very much for your help.

Just to confirm, my nxlog config wasn't quite right. I have changed it to:

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module      xm_json
</Extension>

<Input in>
    Module      im_msvistalog

    
Exec $raw_event = to_json();
    
    Query <QueryList>\
                            <Query Id="0">\
                          <Select Path="Microsoft-Windows-TaskScheduler/Operational">*</Select>\
                  </Query>\
            <Query Id="1">\
                          <Select Path="Application">*</Select>\
                  </Query>\
            <Query Id="2">\
                          <Select Path="System">*</Select>\
                  </Query>\
    </QueryList>

</Input>

<Output out>
    Module      om_tcp
    Host        server
    Port        5140
</Output>

<Route 1>
    Path        in => out
</Route>

Then changed the codec in my logstash.conf from multiline to json_lines

All looks good - the only remaining thing to do is change the EventID field, its currently showing as a number so event id shows as 1,450 rather than 1450.

Thanks again!

E