Syslog beats separate output

Hi there,
I'm trying to learn logstash. I have three inputs in logstash listening on different ports.

This is what the config looks like

input {
udp {
tags => ["netflow"]
type => "netflow"
port => 5046
codec => netflow {
versions => [9]
}

        }

udp {
tags => ["syslog"]
type => "syslog"
port => 5045
   }

beats {
port => 5044
tags => ["winlogbeat"]
type => "winlogbeat"
   }

}

output {

if [type] == "netflow" {
elasticsearch { hosts => ["http://localhost:9200"] }
stdout { codec => rubydebug }	
file { path => "/home/user/netflow.log" }

}

if [type] == "syslog" {
elasticsearch { hosts => ["http://localhost:9200"] }
stdout { codec => rubydebug }
file { path => "/home/user/syslog.log" }

}

if [type] == "winlogbeat" {
elasticsearch { 
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout { codec => rubydebug }
file { path => "/home/user/winlogbeat.log" }
}

else {
file { path=> "/home/user/unknown.log" }
}

}

The problem that I have is winlogbeat doesnt get recognized properly and it goes to the unknown.log
The netflow and syslog appear to be logging into netflow.log and syslog.log which I've added for testing purposes

Can someone have a look at my syntax and tell me what could be wrong?

./logstash -f /etc/logstash/conf.d/elk.conf -log.level=debug WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console 10:57:55.591 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} 10:57:55.594 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"} log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 10:57:55.660 [[main]-pipeline-manager] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>#} 10:57:55.661 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Using mapping template from {:path=>nil} 10:57:55.800 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}} 10:57:55.803 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#]} 10:57:55.809 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} 10:57:55.810 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"} 10:57:55.812 [[main]-pipeline-manager] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>#} 10:57:55.818 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Using mapping template from {:path=>nil} 10:57:55.821 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}} 10:57:55.825 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#]} 10:57:55.826 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500} 10:57:56.182 [[main]-pipeline-manager] INFO logstash.inputs.beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"} 10:57:56.226 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started 10:57:56.234 [[main]"0.0.0.0:5045"} 10:57:56.235 [[main]"0.0.0.0:5046"} 10:57:56.264 [[main]"0.0.0.0:5046", :receive_buffer_bytes=>"106496", :queue_size=>"2000"} 10:57:56.267 [[main]"0.0.0.0:5045", :receive_buffer_bytes=>"106496", :queue_size=>"2000"} 10:57:56.330 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600} 10:58:46.533 [worker1] INFO logstash.outputs.file - Opening file {:path=>"/home/user/unknown.log"}

Please change

file { path=> "/home/user/unknown.log" }

to

file { path=> "/home/user/unknown.log" codec => json_lines }

and report back what the Winlogbeat events that arrive here look like.

Hello,
thanks for your reply
I've made the change but it still appears to be using plain(?) or same as before inserting the codec.

unknown.log contains windows AD Event Log > Security

Here's a small piece of what it looks like

{"computer_name":"MSAD.something.local","process_id":516,"keywords":["Audit Success"],"log_name":"Security","level":"Information","record_number":"207620581","event_data":{"ProcessName":"-","LogonGuid":"{00000000-0000-0000-0000-000000000000}","LogonType":"3","IpPort":"37018","SubjectLogonId":"0x0","TransmittedServices":"-","KeyLength":"128","LmPackageName":"NTLM V1","TargetLogonId":"0x3c025522","SubjectUserName":"-","WorkstationName":"NDNAGIOS","IpAddress":"192.168.131.150","SubjectDomainName":"-","ImpersonationLevel":"%%1833","TargetUserName":"administrator","ProcessId":"0x0","LogonProcessName":"NtLmSsp ","TargetDomainName":"SOMETHING","SubjectUserSid":"S-1-0-0","TargetUserSid":"S-1-5-21-3192175192-2140367962-1494716559-5839","AuthenticationPackageName":"NTLM"},"message":"An account was successfully logged on.\n\nSubject:\n\tSecurity ID:\t\tS-1-0-0\n\tAccount Name:\t\t-\n\tAccount Domain:\t\t-\n\tLogon ID:\t\t0x0\n\nLogon Type:\t\t\t3\n\nImpersonation Level:\t\tImpersonation\n\nNew Logon:\n\tSecurity ID:\t\tS-1-5-21-3192175192-2140367962-1494716559-5839\n\tAccount Name:\t\tadministrator\n\tAccount Domain:\t\tSOMETHING\n\tLogon ID:\t\t0x3C025522\n\tLogon GUID:\t\t{00000000-0000-0000-0000-000000000000}\n\nProcess Information:\n\tProcess ID:\t\t0x0\n\tProcess Name:\t\t-\n\nNetwork Information:\n\tWorkstation Name:\tNDNAGIOS\n\tSource Network Address:\t192.168.131.150\n\tSource Port:\t\t37018\n\nDetailed Authentication Information:\n\tLogon Process:\t\tNtLmSsp \n\tAuthentication Package:\tNTLM\n\tTransited Services:\t-\n\tPackage Name (NTLM only):\tNTLM V1\n\tKey Length:\t\t128\n\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\n\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\n\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\n\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\n\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\n\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\n\nThe authentication information fields provide detailed information about this specific logon request.\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\n\t- Transited services indicate which intermediate services have participated in this logon request.\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.","type":"wineventlog","opcode":"Info","version":1,"tags":["dc","winlogbeat","beats_input_codec_plain_applied"],"thread_id":6452,"@timestamp":"2017-05-31T05:13:42.829Z","task":"Logon","event_id":4624,"provider_guid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","@version":"1","beat":{"hostname":"NDMSS","name":"NDMSS","version":"5.4.0"},"host":"NDMSS","source_name":"Microsoft-Windows-Security-Auditing"}

{"process_id":516,"computer_name":"MSAD.something.local","keywords":["Audit Success"],"level":"Information","log_name":"Security","record_number":"207620582","event_data":{"TargetLogonId":"0x3c025522","LogonType":"3","TargetUserName":"administrator","TargetDomainName":"SOMETHING","TargetUserSid":"S-1-5-21-3192175192-2140367962-1494716559-5839"},"message":"An account was logged off.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-21-3192175192-2140367962-1494716559-5839\n\tAccount Name:\t\tadministrator\n\tAccount Domain:\t\tSOMETHING\n\tLogon ID:\t\t0x3C025522\n\nLogon Type:\t\t\t3\n\nThis event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.","type":"wineventlog","opcode":"Info","tags":["dc","winlogbeat","beats_input_codec_plain_applied"],"thread_id":2776,"@timestamp":"2017-05-31T05:13:42.830Z","event_id":4634,"task":"Logoff","provider_guid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","@version":"1","beat":{"hostname":"NDMSS","name":"NDMSS","version":"5.4.0"},"host":"NDMSS","source_name":"Microsoft-Windows-Security-Auditing"}

If I change the logstash config and use else to log everything else (including winlogbeat) it does log to winlogbeat.log but I wouldn't mind knowing why its ignoring the type field for winlogbeat

else {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "winlogbeat-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
file { path => "/home/user/winlogbeat.log" }

I've made the change but it still appears to be using plain(?) or same as before inserting the codec.

No, it's clearly emitting JSON as requested.

but I wouldn't mind knowing why its ignoring the type field for winlogbeat

Because Winlogbeat already sets the type to "wineventlog". The type => "winlogbeat" in your input configuration doesn't overwrite that.

Thanks for the explanation. So in the end was there anything wrong with my output section?

Would there be any need to have 3 if statements if I have exactly 3 inputs and is else statement good practise to catch everything else?

So in the end was there anything wrong with my output section?

Correct.

Would there be any need to have 3 if statements if I have exactly 3 inputs and is else statement good practise to catch everything else?

It depends. Do you want to have a catch-all destination or should non-matching events be dropped on the floor?

I was trying to have a catch all destination in the unknown.log but that was the original problem in the first post where winlogbeat was being sent to unknown.

You said the type was already set to winlogbeat so there was no need to have it again in the input however if i use an if tag or if type they don't seem to be matched and sent to the correct test log (winlogbeat.log)

Thanks for your help

You said the type was already set to winlogbeat

No, I said it was set to "wineventlog". You need to either adjust your conditionals to match this reality or use a mutate filter to replace the type contents with "winlogbeat". But I suggest you don't; "wineventlog" describes the data and "winlogbeat" the program that collected the data. Prefer type names that describe the data.

Hi Magnus,
Sorry I wasn't reading it properly. It really does make perfect sense now
Thank you very much

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.