Thanks @Badger . This made things quite clearer. Then I started studying JSON formats correctly. Found some other kind of workaround. (for the benefit of anyone who wants to parse windows eventlogs using json in logstash )-
- Convert the eventlog .evtx file to json using get-winevent . Below command will convert all .evtx files into json with same name (but including .evtx in name, that can be renamed later). I found -Depth 2 the best suitable depth of JSON, higher than this was just nesting unwanted fields in json.
foreach ($file in Get-ChildItem -Path .) {Get-WinEvent -Path $file | ConvertTo-Json -Depth 2 -Compress | Out-File "$($file).json" -Encoding utf8 }
- Now the resulting json file comes as a single line in form of an array, something like-
[{"Id":4726,"Version":0,........,"something":"some"},{"Id":4733,.........,"something":"some"}]
Remove the beginning square brace [ and ending brace ] using SED.
3. The message section in the log contains newline characters \r\n, replace them with , using SED to avoid parts of one log getting splitted into multiple logs in elasticsearch. I need to relook this replacement and do something more useful.
4. Now replace },{ with }\r\n{ using SED.
5. Sometimes there will be repeating json keys with the name Value. Replace them one by one with value1, value2, value3 etc... . You can achieve this by using SED in loop without global substitution.
Thats all, the very basic, clumsy but prsable json logs are ready.
Now I am remaining with one challenge, The Message field I had filled with mutiple , to replace \r\n , the original value of Message key was-
"Message":"A member was added to a security-enabled global group.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-2249204231-554948959-1540642952-1001\r\n\tAccount Name:\t\tkriss\r\n\tAccount Domain:\t\tDESKTOP-E46TUNE............trimmed....
It appears like this in kibana now after replacing \r\n with a ,
Message: A member was added to a security-enabled global group.,,Subject:, Security ID: S-1-5-21-2249204231-554948959-1540642952-1001......trimmed........
The portion of eventlog creating problem is the one containing details in GUI. Can't believe how microsoft ruined the logging system for the sake of nice GUI.
Message key's value has become kind of CSV now. Any ideas on how to make it more parsable?