NXLog Best Practices and Multiple Log Inputs

Elastic Stack Logstash
1

Has anyone seen a best practices for how to configure NXLog? Im looking for recommendations on the best way to configure CPU and memory buffers, Inputs, Outputs, Routes, ect.

Also we will have multiple Input log files besides the Windows Eventlogs that we need to pull from. Currently all of the logs are sent to port 5003 of the Logstash server and tagged as Type "WindowsEventLog". Whats the best way to break out the application logs from the Windows Event Logs? Should I just send the application logs to a different port? Is there a way of setting the Type in the NXLog config and not in the Logstash input? Is there a better way to do this?

2

Also, I have a test file that I am using for configuration testing. But for some reason I dont see any information showing up within Elasticsearch. Im not sure whats going on. Here is the configuration that I am using:

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module      xm_json
</Extension>

<Extension _syslog>
    Module      xm_syslog
</Extension>

########## INPUTS ###########

<Input eventlog>
    Module      im_msvistalog
    Exec $Message = to_json();
</Input>


<Input test_file_watch_1>
 Module im_file
 File "C:\Logtest.log"
 SavePos TRUE
</Input>

############ OUTPUTS ##############

<Processor mem_buffer>
    Module  pm_buffer
    MaxSize 1024
    Type Mem
    WarnLimit 512
</Processor>

############ OUTPUTS ##############

<Output out>
    Module      om_tcp
    Host        192.168.100.90
    Port        5003
</Output>

############ ROUTES TO CHOOSE #####

<Route 1>
    Path        eventlog, test_file_watch_1 => mem_buffer => out
</Route>
3

Why would you need mem_buffer processor? Personally, I think nxlog already works great by default.

On nxlog, you should start by outputting to a local file first to see if events are parsed correctly. After that, output Logstash to console to see if LS receives events. If both are good, then you can check Elasticsearch.

Yes, you can add more fields to each message BEFORE converting it to JSON

$type = "wineventlog"
4

We are still new to ELK and I saw it in a example, so I thought it might help some.

Ok so I did as you suggested and outputted to a local file.

Here is the contents of the logfile that im writing to:

Did
this
work?
How
about
now?

And the output file from NXLog contained:

Did਍this਍work?਍How਍about਍now?਍`

So it would seem like I have a formatting issue. Correct?

5

Try this config to read from a file

<Input test_file_watch_1>
    Module	im_file
    File "C:\Logtest.log"
    InputType LineBased
    <Exec>
        $message = $raw_event;
        to_json();
    </Exec>
</Input>

<Output out_file>
    Module	om_file
    File	"C:\output.txt"
</Output>

<Route to_file>
    Path	test_file_watch_1 => out_file
</Route>

Or you can use winlogbeat and filebeat.

6

Hi Anh,

So I made the suggest changes and here is the output:

{"EventReceivedTime":"2016-10-03 09:44:36","SourceModuleName":"test_file_watch_1","SourceModuleType":"im_file","message":"ÿþD\u0000i\u0000d\u0000"}
{"EventReceivedTime":"2016-10-03 09:44:36","SourceModuleName":"test_file_watch_1","SourceModuleType":"im_file","message":"t\u0000h\u0000i\u0000s\u0000"}
{"EventReceivedTime":"2016-10-03 09:44:36","SourceModuleName":"test_file_watch_1","SourceModuleType":"im_file","message":"w\u0000o\u0000r\u0000k\u0000?\u0000"}
{"EventReceivedTime":"2016-10-03 09:44:43","SourceModuleName":"test_file_watch_1","SourceModuleType":"im_file","message":"D\u0000i\u0000d\u0000"}
{"EventReceivedTime":"2016-10-03 09:44:43","SourceModuleName":"test_file_watch_1","SourceModuleType":"im_file","message":"t\u0000h\u0000i\u0000s\u0000"}
{"EventReceivedTime":"2016-10-03 09:44:43","SourceModuleName":"test_file_watch_1","SourceModuleType":"im_file","message":"w\u0000o\u0000r\u0000k\u0000?\u0000"}

Looks like its still not reading the input file correctly. Here is my input file and you can see that the Input and Output dont match

Did
this
work?
Did
this
work?

How would going with winlogbeat and filebeat help with this situation?

7

Ok I tested out beats and it still resulted in the same type of message:

"message":"\ufffd\ufffdD\u0000o\u0000e\u0000s\u0000 \u0000t\u0000h\u0000i\u0000s\u0000 \u0000w\u0000o\u0000r\u0000k\u0000?\u0000\r\u0000"

What am I missing?

8

I think it's just the log content that you are trying to parse. Usually you have to use a module to parse a log lines before converting them to json documents. Do you have a sample log content, a csv file for instance?

9

I was able to get it working by changing the encoding to utf-16. Thanks for the help!