Double processor in ingest pipeline

Elastic Stack Elastic Agent
21

Ahh good catch...

The second solution with the grok does not do that.. pretty sure it is what you asked for ... Did you try that?

You might need to adjust if does not work perfectly... escape characters etc.. I ran it against your sample and it works... But the field k8.pod.labels appears more than once I think the way I have it it will ignore the one with the {"app".... etc

22

Hi,
It seems to work correctly even if k8.pod.labels appears twice with different formats.
I have to study the grok processor that seems very powerful!
Thanks a lot!

1 Like
23

ehm...
is it possible that POST /_ingest/pipeline/logs-trend_micro_vision_one.detection@custom/_simulate?verbose=true
does not report an error for the 2 types of expected docs but that the data stream logs-trend_micro_vision_one.detection-default is no longer populated? How can I investigate the error?

24

Apologies I am not clear what you are asking...

I am not sure what 2 types of expected documents

_simulate makes sure

  1. the Ingest Pipeline works
    but
  2. does not write the document to an index... There can be mapping errors when the document is being written and thus can be rekected

If you want to test 2) and make sure the document can actually be written to the datastream

You can POST an actual document to the datastream ...
if there is an error it will show up.

To do this... put the event.original content in the message field you will also need to change the @timestamp I also add a tag test_document so I can find it easily.

It will either the document will be written or there will be an error.

POST logs-trend_micro_vision_one.detection-default/_doc
{
  		"tags": [
			"preserve_original_event",
			"preserve_duplicate_custom_fields",
			"forwarded",
			"trend_micro_vision_one-detection",
			"test_document"
		],
  		"@timestamp": "2024-12-11T17:09:59.000Z",
		"message": "{\"act\":[\"log\"],\"clusterId\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"*************\",\"clusterType\":\"k8s\",\"containerId\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"containerImage\":\"***********.jfrog.io/***********/tomcat:9.0.73-***\",\"containerImageDigest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"containerName\":\"tomcat-container\",\"customTags\":[\"mitre_command_and_control\",\"process\"],\"detectionType\":\"process\",\"endpointHostName\":\"****.eu-south-1.compute.internal\",\"eventId\":\"100119\",\"eventName\":\"SECURITY_RISK_DETECTION\",\"eventSourceType\":3,\"eventTime\":1733936999000,\"eventTimeDT\":\"2024-12-11T17:09:59+00:00\",\"groupId\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"k8sNamespace\":\"sic-iam\",\"k8sPodId\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8sPodName\":\"pnt-test-***-0\",\"logKey\":\"e99c5cff-18f2-2e55-1b2b-ac77279777b5\",\"logReceivedTime\":\"1733937090989\",\"parentCmd\":\"bash ./new-entrypoint.sh\",\"parentName\":\"bash\",\"parentPid\":37520,\"pname\":\"Vision One Container Security\",\"policyId\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"processCmd\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\",\"processName\":\"curl\",\"processPid\":1534304,\"productCode\":\"scs\",\"pver\":\"2021-12-01T00:00:00.0000000Z\",\"rawDataStr\":\"{\\\"clusterID\\\":\\\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\\\",\\\"clusterName\\\":\\\"*************\\\",\\\"customerID\\\":\\\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\\\",\\\"details\\\":{\\\"container.id\\\":\\\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\\\",\\\"container.image.digest\\\":\\\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\\\",\\\"container.image.repository\\\":\\\"****viva.jfrog.io/********/tomcat\\\",\\\"container.image.tag\\\":\\\"9.0.73-***\\\",\\\"container.name\\\":\\\"tomcat-container\\\",\\\"evt.arg.exe\\\":\\\"curl\\\",\\\"evt.arg.filename\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.mode\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.name\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.category\\\":\\\"process\\\",\\\"evt.dir\\\":\\\"\\u003c\\\",\\\"evt.num\\\":\\\"8267962726\\\",\\\"evt.rawtime\\\":\\\"1733936999010100132\\\",\\\"evt.time\\\":\\\"17:09:59.010100132\\\",\\\"evt.type\\\":\\\"execve\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.id\\\":\\\"b777d8d9-441e-46ab-b34f-dac23b316636\\\",\\\"k8s.pod.labels\\\":\\\"app:pnt-test-***,controller-revision-hash:pnt-test-***-7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-***-0\\\",\\\"k8s.pod.name\\\":\\\"pnt-test-***-0\\\",\\\"proc.args\\\":\\\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.cmdline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.exe\\\":\\\"curl\\\",\\\"proc.exeline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.exepath\\\":\\\"/usr/bin/curl\\\",\\\"proc.name\\\":\\\"curl\\\",\\\"proc.pcmdline\\\":\\\"bash ./new-entrypoint.sh\\\",\\\"proc.pid\\\":\\\"1534304\\\",\\\"proc.pid.ts\\\":\\\"1733936999008820540\\\",\\\"proc.pname\\\":\\\"bash\\\",\\\"proc.ppid\\\":\\\"37520\\\",\\\"proc.ppid.ts\\\":\\\"1733427545111053154\\\",\\\"user.loginuid\\\":\\\"-1\\\",\\\"user.name\\\":\\\"root\\\"},\\\"hostname\\\":\\\"****.eu-south-1.compute.internal\\\",\\\"id\\\":\\\"2q50NZGekOuxLvx2iLCOQYHouki\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.labels\\\":{\\\"app\\\":\\\"pnt-test-***\\\",\\\"controller-revision-hash\\\":\\\"pnt-test-***-7668f9d96d\\\",\\\"statefulset.kubernetes.io/pod-name\\\":\\\"pnt-test-***-0\\\"},\\\"k8s.pod.name\\\":\\\"pnt-test-***-0\\\",\\\"mitigation\\\":\\\"log\\\",\\\"name\\\":\\\"(T1105)Launch Ingress Remote File Copy Tools in Container\\\",\\\"orchestration\\\":\\\"k8s\\\",\\\"policyID\\\":\\\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\\\",\\\"policyName\\\":\\\"LogOnlyPolicy\\\",\\\"productCode\\\":\\\"scs\\\",\\\"ruleID\\\":\\\"TM-00000049\\\",\\\"rulesets\\\":[{\\\"id\\\":\\\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\\\",\\\"name\\\":\\\"LogOnlyRuleset\\\"}],\\\"severity\\\":\\\"notice\\\",\\\"tags\\\":[\\\"mitre_command_and_control\\\",\\\"process\\\"],\\\"timestamp\\\":\\\"2024-12-11T17:09:59.010100Z\\\",\\\"type\\\":\\\"syscall\\\",\\\"version\\\":\\\"2021-12-01\\\"}\",\"rt_utc\":\"2024-12-11T17:09:59.0000000Z\",\"ruleIdStr\":\"TM-00000049\",\"ruleName\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"ruleSetId\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"ruleSetName\":\"LogOnlyRuleset\",\"searchDL\":\"DDL\",\"severity\":4,\"sourceType\":\"syscall\",\"uuid\":\"319bf2a2-02c9-4e24-b927-1b78725d7c94\"}"
}
25

But no log?
I don't want to change the index

26

@Cristina_Marletta_Li

Sure you can turn up the agent logging and elasticsearch logging and hunt through them to find the mapping error if that is what is....

or just post to a test namespace and that will put in a different data stream that you can then clean up and not change the current index.

POST logs-trend_micro_vision_one.detection-test/_doc
...........................................^^^^

That will put the document in a separate datastream
logs-trend_micro_vision_one.detection-test
and backing index
.ds-logs-trend_micro_vision_one.detection-test-2024.12.16-000001 (or similar)

which you can clean up... and not change the current / default datastream and index. It will still run all the ingest pipelines and use the correct mappings.

Otherwise, yes you will need to turn up logging on the Elasticsearch cluster and the Agent logging and dig through them to find the error.

27

Ok Stephenb,
I have to practice.
The grok processor is very very interesting.
Very very interesting the grok processor.
Where can I find the keywords of the patterns (and their description): DATA, IP, WORD etc.?

28

Docs

Patterns

Helpfully tool
https://grokconstructor.appspot.com/do/construction

It is just Regex under the covers...

29

Thank you @stephenb but the builtin keywords like WORD, IP...?

30

Found it!
In github!

1 Like
31

It Takes a little practice... but grok is very powerful...

You should also learn dissect and kv together you can do a lot if needed.