Custom event parsing

I am relatively new to SO. I am trying to understand how to ingest and parse customs logs in ELS.

I currently use the NXLog agent to send json formatted data (coming from the SYSMON windows eventlog) from a host to the logstash port (5044) on the manager. As per the manual, it's now ELS that does the ingestion, correct?

However it is not clear to me what has to happen next? I tried to modify the existing sysmon pipeline without any luck.

First, it is not clear how ELS decides which parser to use when it receives the raw event from logstash?

Then, after the data is parsed, where does it get send?

Thank for clearing out my confusion

Here is an example of what I trying to ingest:

{"EventTime": "2021-11-30T00:12:12.899263-05:00","Hostname":"MYSERVER","Keywords":"9223372036854775808","EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":11,"SourceName":"Microsoft-Windows-Sysmon","ProviderGuid":"{5770385F-C22A-43E0-BF4C-06F5698FFBD9}","Version":2,"TaskValue":11,"OpcodeValue":0,"RecordNumber":6682123,"ExecutionProcessID":9080,"ExecutionThreadID":13784,"Channel":"Microsoft-Windows-Sysmon/Operational","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-18","AccountType":"User","Message":"File created:\r\nRuleName: -\r\nUtcTime: 2021-11-30 05:12:12.899\r\nProcessGuid: {CA401AF7-1F55-6184-4700-000000009600}\r\nProcessId: 4148\r\nImage: C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\MonitoringHost.exe\r\nTargetFilename: C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\Monitoring Host Temporary Files 8\\1470202\\Microsoft.Windows.Server.DNS.WINS.Discovery.vbs\r\nCreationUtcTime: 2021-11-30 05:12:12.899\r\nUser: NT AUTHORITY\\SYSTEM","Category":"File created (rule: FileCreate)","RuleName":"-","UtcTime":"2021-11-30 05:12:12.899","ProcessGuid":"{CA401AF7-1F55-6184-4700-000000009600}","ProcessId":"4148","Image":"C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\MonitoringHost.exe","TargetFilename":"C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\Monitoring Host Temporary Files 8\\1470202\\Microsoft.Windows.Server.DNS.WINS.Discovery.vbs","CreationUtcTime":"2021-11-30 05:12:12.899","User":"NT AUTHORITY\\SYSTEM","EventReceivedTime":"2021-11-30T00:12:14.821147-05:00","SourceModuleName":"sysmon_event","SourceModuleType":"im_msvistalog"}

Welcome to our community! :smiley:

What is SO?
What manual are you referring to?

SO - Security Onion


thanks :slight_smile:

Ahh ok, you might need to hit up the security onion community there, as they have a bunch of customisations that you are using we aren't directly familiar with sorry.

Otherwise, someone with relevant experience might stop by!

Thanks, let me try to simplify my question.

How does Elasticsearch decides which pipeline(parser) to apply to an an incoming event?

When a request is passed to Elasticsearch it would usually contain the pipeline that is needed to run - Ingest pipelines | Elasticsearch Guide [7.15] | Elastic

What if I send the events on port 514 directly to Elastic? Will it use the default pipeline?

Yes, it may use whatever default pipeline has been defined. But, again, that needs to be set. Elasticsearch doesn't just pick one as default by itself.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.