Custom event parsing

[ohliuw]

I am relatively new to SO. I am trying to understand how to ingest and parse customs logs in ELS.

I currently use the NXLog agent to send json formatted data (coming from the SYSMON windows eventlog) from a host to the logstash port (5044) on the manager. As per the manual, it's now ELS that does the ingestion, correct?

However it is not clear to me what has to happen next? I tried to modify the existing sysmon pipeline without any luck.

First, it is not clear how ELS decides which parser to use when it receives the raw event from logstash?

Then, after the data is parsed, where does it get send?

Thank for clearing out my confusion

Here is an example of what I trying to ingest:

{"EventTime": "2021-11-30T00:12:12.899263-05:00","Hostname":"MYSERVER","Keywords":"9223372036854775808","EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":11,"SourceName":"Microsoft-Windows-Sysmon","ProviderGuid":"{5770385F-C22A-43E0-BF4C-06F5698FFBD9}","Version":2,"TaskValue":11,"OpcodeValue":0,"RecordNumber":6682123,"ExecutionProcessID":9080,"ExecutionThreadID":13784,"Channel":"Microsoft-Windows-Sysmon/Operational","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-18","AccountType":"User","Message":"File created:\r\nRuleName: -\r\nUtcTime: 2021-11-30 05:12:12.899\r\nProcessGuid: {CA401AF7-1F55-6184-4700-000000009600}\r\nProcessId: 4148\r\nImage: C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\MonitoringHost.exe\r\nTargetFilename: C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\Monitoring Host Temporary Files 8\\1470202\\Microsoft.Windows.Server.DNS.WINS.Discovery.vbs\r\nCreationUtcTime: 2021-11-30 05:12:12.899\r\nUser: NT AUTHORITY\\SYSTEM","Category":"File created (rule: FileCreate)","RuleName":"-","UtcTime":"2021-11-30 05:12:12.899","ProcessGuid":"{CA401AF7-1F55-6184-4700-000000009600}","ProcessId":"4148","Image":"C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\MonitoringHost.exe","TargetFilename":"C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\Monitoring Host Temporary Files 8\\1470202\\Microsoft.Windows.Server.DNS.WINS.Discovery.vbs","CreationUtcTime":"2021-11-30 05:12:12.899","User":"NT AUTHORITY\\SYSTEM","EventReceivedTime":"2021-11-30T00:12:14.821147-05:00","SourceModuleName":"sysmon_event","SourceModuleType":"im_msvistalog"}

[warkolm (Mark Walkom)]

Welcome to our community!

What is SO?
What manual are you referring to?

[ohliuw]

SO - Security Onion

Manuals:

https://docs.securityonion.net/en/2.3/elasticsearch.html

Ingest pipelines | Elasticsearch Guide [7.15] | Elastic

thanks

[warkolm (Mark Walkom)]

Ahh ok, you might need to hit up the security onion community there, as they have a bunch of customisations that you are using we aren't directly familiar with sorry.

Otherwise, someone with relevant experience might stop by!

[ohliuw]

Thanks, let me try to simplify my question.

How does Elasticsearch decides which pipeline(parser) to apply to an an incoming event?

[warkolm (Mark Walkom)]

When a request is passed to Elasticsearch it would usually contain the pipeline that is needed to run - Ingest pipelines | Elasticsearch Guide [7.15] | Elastic

[ohliuw]

What if I send the events on port 514 directly to Elastic? Will it use the default pipeline?

[warkolm (Mark Walkom)]

Yes, it may use whatever default pipeline has been defined. But, again, that needs to be set. Elasticsearch doesn't just pick one as default by itself.