Windows NXLOG Connecting but not showing up in ES

Elastic Stack Logstash
1

I am trying to setup my first Windows server to send logs, via NXLOG, to Logstash and having some difficulty. According to the NXLOG log file on my Windows host it is successfully connecting to Logstash. So I think the issue is with how Logstash is handling the logs. It is not showing up withing Elasticsearch. Here is my input file:

#logstash-forwarder (Linux Servers)
input {
  lumberjack {
    port => 5000
    type => "linux-logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

#nxlog-forwarder (Windows Servers)
input {
  lumberjack {
    port => 5001
    codec => "line"
    type => "WindowsEventLog"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


#Local firewall forwards TCP/UDP port 514 to 5140
#tcp syslog stream via 5140
input {
  tcp {
#    type => "syslog"
    port => 5140
  }
}

Here is my filter file:

filter{
    if [type] == "WindowsEventLog" {
        json{
            source => "message"
        }
        if [SourceModuleName] == "eventlog" {
            mutate {
                replace => [ "message", "%{Message}" ]
            }
            mutate {
                remove_field => [ "Message" ]
            }
        }
    }
}

I dont think my Output file is the issue since I am getting data from Syslog. But here it is anyways:

output {
  elasticsearch { hosts => ["<ES1 server IP>", "<ES2 server IP>", "<ES3 server IP>"] }
  }
2

If you use stdout do you see the events?

3

Can you elaborate as to where stout goes? Sorry im new and still learning.

4

In the output section - https://www.elastic.co/guide/en/logstash/2.1/plugins-outputs-stdout.html

5

So modify my output file to something like this:

output {
  elasticsearch { hosts => ["<ES1 server IP>", "<ES2 server IP>", "<ES3 server IP>"] }
  stdout {}
}
6

Yep, you may want to set codec => "rubydebug" to make it easier to read.

7

Ok my output file is not as follows:

output {
  elasticsearch { hosts => ["<ES1 server IP>", "<ES2 server IP>", "<ES3 server IP>"] }
  stdout {codec => "rubydebug"}
  }

So far nothing has showed up within ES.

8

Are you starting LS as a service or a command?
If it's the former then check your LS logs :slight_smile:

9

Its a service. There is nothing in /var/log/logstash/logstash.err

Here is whats in my logstash.log file. Dont see anything thats linked to my Windows conf output

{:timestamp=>"2016-01-24T13:48:37.037000-0800", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-01-24T13:48:37.344000-0800", :message=>"UDP listener died", :exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3682:in `select'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:77:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:206:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:199:in `start_input'"], :level=>:warn}
{:timestamp=>"2016-01-24T14:12:06.578000-0800", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-01-24T14:12:06.860000-0800", :message=>"UDP listener died", :exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3682:in `select'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:77:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:206:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:199:in `start_input'"], :level=>:warn}
{:timestamp=>"2016-01-24T14:25:29.493000-0800", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-01-24T14:25:29.857000-0800", :message=>"UDP listener died", :exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3682:in `select'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:77:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:206:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:199:in `start_input'"], :level=>:warn}
{:timestamp=>"2016-01-24T14:35:24.082000-0800", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-01-24T14:35:24.613000-0800", :message=>"UDP listener died", :exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3682:in `select'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:77:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:206:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:199:in `start_input'"], :level=>:warn}
{:timestamp=>"2016-01-24T14:57:25.001000-0800", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-01-24T14:57:25.319000-0800", :message=>"UDP listener died", :exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3682:in `select'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:77:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:206:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:199:in `start_input'"], :level=>:warn}
10

Still nothing is showing up within ES. Any other thoughts?

11

Stop the service and then run it manually, see what happens.

12

done, no luck so far

13

Then there is no data making it's way to LS.

Remove all the filters and just have inputs and outputs. If still nothing, then it's nxlog.

14

Done, still nothing. This is getting annoying. For all transparancy here is my nxlog config file:

## Please set the ROOT to your nxlog installation directory

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data
LogFile %ROOT%\data\nxlog.log
define CERTDIR %ROOT%\cert

<Extension syslog>
  Module xm_syslog  
</Extension>

# Monitor application log files
#<Input watchfile>
# Module im_file
#  File 'C:\\path\\to\\*.log'
#  Exec $Message = $raw_event;
#  Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1;
#  SavePos TRUE  
#  Recursive TRUE
#</Input>

# Monitor a single application log file
#<Input watchfile2>
#  Module im_file
#  # File 'C:\\path\\to\\a\\single\\file.log'
#  Exec $Message = $raw_event;
#  Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1;
#  SavePos TRUE  
#  Recursive TRUE
#</Input>

# Monitor Windows event logs
<Input eventlog>
  # Uncomment for Windows Vista/2008 or later 
  Module im_msvistalog
  
  # Uncomment for Windows 2000 or later
  # Module im_mseventlog
</Input>

<Processor eventlog_transformer>
  Module pm_transformer
  Exec $Hostname = hostname();
  OutputFormat syslog_rfc5424  
</Processor>

<Processor filewatcher_transformer>
  Module pm_transformer
  
  # Uncomment to override the program name
  # Exec $SourceName = 'PROGRAM NAME';
    
  Exec $Hostname = hostname();
  OutputFormat syslog_rfc5424
</Processor>

<Output syslogout>
  Module om_ssl
  Host <logstash server IP>
  Port 5001
  CAFile %CERTDIR%/logstash-forwarder.crt
  OutputType      Syslog_TLS
  AllowUntrusted TRUE
</Output>

<Route 1>
  Path eventlog => eventlog_transformer => syslogout
</Route>

<Route 2>
  Path watchfile => filewatcher_transformer => syslogout
</Route>

# Replace route 2 with this implementation if watchfile2 is active
#<Route 2>
#  Path watchfile, watchfile2 => filewatcher_transformer => syslogout
#</Route>
15

I have not used NXLog, but was not aware that it supported the lumberjack protocol. In this example it shows NXLog connecting through the TCP input plugin. Can you set it up to connect through the TCP plugin instead and see if that works?

1 Like
16

Argh, good catch, I totally missed that!

17

Thanks guys for the help. I replaced lumberjack protocol with TCP and now the logs from my other inputs are no longer ingesting into ES. Can you only have one TCP input? Here is my current inputs config:

#nxlog-forwarder (Windows Servers)
input {
  tcp {
    port => 5001
    type => "WindowsEventLog"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


#Local firewall forwards TCP/UDP port 514 to 5140
#tcp syslog stream via 5140
input {
  tcp {
#    type => "syslog"
    port => 5140
  }
}
#udp syslogs tream via 5140
input {
  udp {
#    type => "syslog"
    port => 5140
  }
}
18

Ok I made some modifications to my inputs file and not I see data within ES. I decided to just stop using the lumberjack protocol for my Windows server logs. For reference here my updated inputs file:

#Local firewall forwards TCP/UDP port 514 to 5140
#tcp syslog stream via 5140
input {
  #Local firewall forwards TCP/UDP port 514 to 5140
  #tcp syslog stream via 5140
  tcp {
    port => 5140
  }

  tcp {
    port => 5001
    type => "WindowsEventLog"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

#udp syslogs tream via 5140
input {
  udp {
#    type => "syslog"
    port => 5140
  }
}

If im thinking it thru correctly I should be able to add additional TCP ports as needed. For example, ill probably add one for our Linux servers, network switches and storage arrays. That is unless there is a compelling reason not to do so.