Extract fields from syslog "message" part for visualization in Kibana

Elastic Stack Logstash
1

I am a total beginner when it comes to LogStash and it's filtering mechanisms. I have tried google but I don't seem to get the answer to my question there (or I cannot make sense of the answer by myself). I am using a syslog agent (WinCollect) to send data to LogStash but in Kibana I want to have several fields for visualization based on the syslog "message" part. For example this message:
`

AgentDevice=WindowsLog AgentLogFile=Security PluginVersion=7.2.5.27 Source=Microsoft-Windows-Security-Auditing Computer=domaincontroller1.contoso.com OriginatingComputer=127.0.0.1 User= Domain= EventID=4624 EventIDCode=4624 EventType=8 EventCategory=12544 RecordNumber=595555663 TimeGenerated=1521725859 TimeWritten=1521725859 Level=0 Keywords=0 Task=0 Opcode=0 Message=An account was successfully logged on. Subject: Security ID: NULL SID Account Name: - Account Domain: - Logon ID: 0x0 Logon Type: 3 New Logon: Security ID: CONTOSO\u0111 Account Name: u0111 Account Domain: CONTOSO Logon ID: 0x2a376541 Logon GUID: {300CFB3A-32CF-8207-48C5-00000000000} Process Information: Process ID: 0x0 Process Name: - Network Information: Workstation Name: - Source Network Address: 10.0.0.1 Source Port: 55123 Detailed Authentication Information: Logon Process: Kerberos Authentication Package: Kerberos Transited Services: - Package Name (NTLM only): - Key Length: 0 This event is generated when a logon session is created. It is generated on the computer that was accessed. The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe. The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network). The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on. The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases. The authentication information fields provide detailed information about this specific logon request. - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event. - Transited services indicate which intermediate services have participated in this logon request. - Package name indicates which sub-protocol was used among the NTLM protocols. - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

`

If I would like to have "EventID" and "Computer" as fields in kibana based on this message how would I setup my grok filters? Sorry no code for my current grok filters since none of them work, my config is pretty simple with just syslog as input and elasticsearch as output.

2

I would lean towards using a kv filter rather than grok. kv will not handle "User= Domain=" correctly, so you can use mutate/gsub to adjust the empty values.

  mutate { gsub => [ "message", "= ", "='' " ] }
  kv { field_split => " " value_split => "=" }

You end up with a field called Message (upper case M) that just contains "An". I would delete that and modify the message field.

 mutate { gsub => [ "message", ".*Message=", "" ] }
3

I tested your solution but got very weird results, I think it has to do with my original paste did not reflect that the delimiter after the "value" part is a tab

"message": "AgentDevice=WindowsLog\tAgentLogFile=Security\tPluginVersion=7.2.5.27\tSource=Microsoft-Windows-Security-Auditing\tComputer=dc1.contoso.com\tOriginatingComputer=127.0.0.1\tUser=\tDomain=\tEventID=4624\tEventIDCode=4624\tEventType=8\tEventCategory=12544\tRecordNumber=600938073\tTimeGenerated=1521754051\tTimeWritten=1521754051\tLevel=0\tKeywords=0\tTask=0\tOpcode=0\tMessage=An account was successfully logged on. Subject: Security ID: NULL SID Account Name: - Account Domain: - Logon ID: 0x0 Logon Type: 3 New Logon: Security ID: CONTOSO\COMPUTER1$ Account Name: COMPUTER1$ Account Domain: CONTOSO Logon ID: 0x30a0e717 Logon GUID: {00000000-0DF5-A4C5-992D-000000000} Process Information: Process ID: 0x0 Process Name: - Network Information: Workstation Name: - Source Network Address: 10.0.0.1 Source Port: 54864 Detailed Authentication Information: Logon Process: Kerberos Authentication Package: Kerberos Transited Services: - Package Name (NTLM only): - Key Length: 0 This event is generated when a logon session is created. It is generated on the computer that was accessed. The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe. The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network). The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on. The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases. The authentication information fields provide detailed information about this specific logon request. - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event. - Transited services indicate which intermediate services have participated in this logon request. - Package name indicates which sub-protocol was used among the NTLM protocols. - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.\n",

Here it is pasted from the JSON view in Kibana.

4

Correct. In that case go to the line in the config that sets field_split and change the space between the double quotes to be a tab.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.