Windows Event Log filtering

Elastic Stack Logstash
1

So, I'm trying to upgrade my logstash conf file to filter the windows event logs and make them more useful. At the moment the log dumps loads of information into one field called "message", like so:

{"EventTime":"2016-06-21
08:37:21","Hostname":"WINDOWS-9NOTMBF.etech.local","Keywords":-9214364837600034816,"EventType":"AUDIT_SUCCESS","SeverityValue":2,"Severity":"INFO","EventID":4648,"SourceName":"Microsoft-Windows-Security-Auditing","ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","Version":0,"Task":12544,"OpcodeValue":0,"RecordNumber":5564,"ProcessID":840,"ThreadID":7096,"Channel":"Security","Message":"A
logon was attempted using explicit
credentials.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount
Name:\t\tWINDOWS-9NOTMBF$\r\n\tAccount Domain:\t\tETECH\r\n\tLogon
ID:\t\t0x3E7\r\n\tLogon
GUID:\t\t{00000000-0000-0000-0000-000000000000}\r\n\r\nAccount Whose
Credentials Were Used:\r\n\tAccount Name:\t\tBrandonMcGrath\r\n\tAccount
Domain:\t\tETECH\r\n\tLogon
GUID:\t\t{B66720EA-C940-0098-E15D-E9F999495C98}\r\n\r\nTarget
Server:\r\n\tTarget Server Name:\tlocalhost\r\n\tAdditional
Information:\tlocalhost\r\n\r\nProcess Information:\r\n\tProcess
ID:\t\t0x348\r\n\tProcess
Name:\t\tC:\Windows\System32\lsass.exe\r\n\r\nNetwork
Information:\r\n\tNetwork Address:\t-\r\n\tPort:\t\t\t-\r\n\r\nThis
event is generated when a process attempts to log on an account by
explicitly specifying that account’s credentials. This most commonly
occurs in batch-type configurations such as scheduled tasks, or when
using the RUNAS
command.","Category":"Logon","Opcode":"Info","SubjectUserSid":"S-1-5-18","SubjectUserName":"WINDOWS-9NOTMBF$","SubjectDomainName":"ETECH","SubjectLogonId":"0x3e7","LogonGuid":"{00000000-0000-0000-0000-000000000000}","TargetUserName":"BrandonMcGrath","TargetDomainName":"ETECH","TargetLogonGuid":"{B66720EA-C940-0098-E15D-E9F999495C98}","TargetServerName":"localhost","TargetInfo":"localhost","ProcessName":"C:\Windows\System32\lsass.exe","IpAddress":"-","IpPort":"-","EventReceivedTime":1466494642,"SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog"}

Is there a way to seperate them into seperate lines rather than in one big message?

2

Use a json codec for your beats input.

input {
  beats {
    ...
    codec => json
  }
}
3

that didnt work, it still dumps 90% of the info into "message" :-\

4

That's fishy. Please post the output from a stdout { codec => rubydebug } output. And what if you instead use a json filter for the JSON parsing?

5

I have solved that issue now, thanks. But, there is something else if you can help. Basically, Elasticsearch Logs are getting completely filled up to like 50gb in a day because of an exception error with topbeat. I uinstalled topbeat, filebeat etc because they were causing too many errors. But, I think I uninstalled them wrong and Elasticsearch is still looking for them and its causing so many issues. Any idea on how I can fix it?

6

ES doesn't know about or look for Topbeat or Filebeat. You can use ES's REST APIs to delete the old indexes if you want to clean things up. You're not being very specific about the problems you're experiencing so it's hard to offer specific help.

7

Can you share the solution? It might help someone in the future.