# Double processor in ingest pipeline

**URL:** <https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768>\
**Category:** Elastic Agent\
**Created:** [December 10, 2024, 2:22pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768 "2024-12-10T14:22:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cristina\_Marletta\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristina_marletta_li/32/137793_2.png) [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\
**Post date:** [December 10, 2024, 2:22pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/1 "2024-12-10T14:22:05Z")

</div>

Hi all,  
it is possible to create a custom ingest pipeline associated with an integration and a specific Data Stream with 2 processors.  
The second one should process a field created by the first one.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 10, 2024, 3:14pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/2 "2024-12-10T15:14:57Z")

</div>

Hi @Cristina_Marletta_Li

Yes, absolutely

An ingest pipeline executes the processors in sequence so you can create a field in one processor and then use it or modify it in a later processor.  
Sometimes it's a bit tricky. If you give us an example, maybe we can help.

---

<div class="post-metadata">

**Author:** ![Cristina\_Marletta\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristina_marletta_li/32/137793_2.png) [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\
**Post date:** [December 11, 2024, 3:33pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/3 "2024-12-11T15:33:35Z")

</div>

Hi,  
hi,  
the first processor does this:  
[  
{  
"json": {  
"field": "event.original",  
"target\_field": "details"  
}  
}  
]

among the generated fields there is details.rawDataStr which, in turn, has the json structure. I would like to parse it from JSON.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 11, 2024, 3:43pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/4 "2024-12-11T15:43:49Z")

</div>

Hi @Cristina_Marletta_Li

Yes, Just build it like you would procedural language, something like this ... you might need to use some conditional logic or ignore failures etc

There might need to be some need to do some cleaning up etc..

I would suggest looking at the docs [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html)

```auto
{
  "json": {
  "field": "event.original",
  "target_field": "details"
  }
},
{
  "json": {
  "field": "details.rawDataStr",
  "target_field": "rawdetails"
  }
}

```

If you give an example document perhaps we could take a quick look

---

<div class="post-metadata">

**Author:** ![Cristina\_Marletta\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristina_marletta_li/32/137793_2.png) [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\
**Post date:** [December 11, 2024, 5:07pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/5 "2024-12-11T17:07:02Z")

</div>

Thankk you stephenb,  
I will study your document.  
Maybe I should add a condition to check that details.rawDataStr is not empty (this can happen)?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 11, 2024, 5:28pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/6 "2024-12-11T17:28:38Z")

</div>

> [@Cristina\_Marletta\_Li](#):
>
> Maybe I should add a condition to check that details.rawDataStr is not empty (this can happen)?

Absolutely! Or you can just set the ignore failure and it'll just drop through...

I highly recommend reading through the documentation if you're going to start using ingest pipelines. They're very powerful and they're as conditionals and error handling or simple drop-thru. Lots of good stuff

I also would recommend kind of building modular. If you have pipelines you want to reuse you can just call them from other pipelines

But that's a little more advanced. Just get started. Come back with some more questionsBut that's a little more advanced. Just get started. Come back with some more questions

---

<div class="post-metadata">

**Author:** ![Cristina\_Marletta\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristina_marletta_li/32/137793_2.png) [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\
**Post date:** [December 12, 2024, 9:59am UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/7 "2024-12-12T09:59:56Z")

</div>

Hi stephenb,  
unfortunately something is wrong. The pipeline is associated with an integration (logs-trend\_micro\_vision\_one.detection@custom). This pipeline refers to another (add\_field) defined like this:

```auto
[
{
"json": {
"field": "event.original",
"target_field": "details"
}
},
{
"json": {
"field": "details.rawDataStr",
"target_field": "rawdetails",
"ignore_failure": true
}
}
]

```

Attached, I report the field to be processed and the screenshot of the definition of the pipeline logs-trend\_micro\_vision\_one.detection@custom.  
Furthermore, since I inserted the new processor, docs containing details.rawDataStr are no longer loaded.

 ![Screenshot 2024-12-12 105556](https://us1.discourse-cdn.com/elastic/original/3X/1/8/1856554c25f8effd7bfadd497f5eb03d0c2cd723.png)  
 ![Screenshot 1](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5f002791f6387f2174927405fa549ad7ab66ed53.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 12, 2024, 1:25pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/8 "2024-12-12T13:25:52Z")

</div>

Hi @Cristina_Marletta_Li

In order to debug you will need to provide a full sample document in JSON and the full pipeline....

You can test with `_simulate`  
And also use `verbose`

> **[Simulate pipeline API | Elasticsearch Guide \[8.16\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html)**

Then in addition there could be a mapping error...

So get a raw event and try to me manually`POST` it to the data stream

```auto
POST logs-trend_micro_vision_one.detection-default/_doc
{
Raw document 

}

```

---

<div class="post-metadata">

**Author:** ![Cristina\_Marletta\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristina_marletta_li/32/137793_2.png) [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\
**Post date:** [December 12, 2024, 1:33pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/9 "2024-12-12T13:33:35Z")

</div>

I tested from GUI with 'Test pipeline'. No errors were found.  
No doc containing the data I'm interested in is written to datastream.  
How can I send you the full document? I can only attach images and the document is very long.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 12, 2024, 1:37pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/10 "2024-12-12T13:37:09Z")

</div>

Unfortunately the GUI test pipeline that may mean the pipeline works but when the doc is written there is mapping conflict and it is not written because it is rejected.

You need to try to POST a document from the dev tools.

See my instructions above.

That was s how you are going to need to test if you can not prove s sample document and full pipeline

---

<div class="post-metadata">

**Author:** ![Cristina\_Marletta\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristina_marletta_li/32/137793_2.png) [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\
**Post date:** [December 12, 2024, 2:03pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/11 "2024-12-12T14:03:02Z")

</div>

I'll try to post here the original document in JSON after the first processing (that creates the details.\* fields). This was the situation before I added the JSON second processor:

```auto
{
	"_index": ".ds-logs-trend_micro_vision_one.detection-default-2024.12.10-000004",
	"_id": "qNQeFHRFBRO02UQv2jILApKG/oA=",
	"_version": 1,
	"_score": 0,
	"_source": {
		"agent": {
			"name": "Shuffle",
			"id": "5f4edad9-8329-460f-adbf-c4789daa91b3",
			"ephemeral_id": "875a1d57-8cb8-4fed-802f-4e4deb2bef60",
			"type": "filebeat",
			"version": "8.16.1"
		},
		"trend_micro_vision_one": {
			"detection": {
				"event_time_dt": "2024-12-11T17:09:59.000Z",
				"parent": {
					"cmd": "bash ./new-entrypoint.sh"
				},
				"process": {
					"name": "curl",
					"pid": 1534304,
					"cmd": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
				},
				"product": {
					"code": "scs",
					"name": "Vision One Container Security",
					"version": "2021-12-01T00:00:00.0000000Z"
				},
				"search_data_lake": "DDL",
				"uuid": "319bf2a2-02c9-4e24-b927-1b78725d7c94",
				"endpoint": {
					"hostname": " ***********.eu-south-1.compute.internal"
				},
				"event_id": "100119",
				"detection_type": "process",
				"action": [
					"log"
				],
				"event_name": "SECURITY_RISK_DETECTION",
				"rt_utc": "2024-12-11T17:09:59.000Z",
				"policy": {
					"logkey": "e99c5cff-18f2-2e55-1b2b-ac77279777b5",
					"name": "LogOnlyPolicy"
				}
			}
		},
		"process": {
			"name": "curl",
			"pid": 1534304,
			"command_line": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
		},
		"elastic_agent": {
			"id": "5f4edad9-8329-460f-adbf-c4789daa91b3",
			"version": "8.16.1",
			"snapshot": false
		},
		"tags": [
			"preserve_original_event",
			"preserve_duplicate_custom_fields",
			"forwarded",
			"trend_micro_vision_one-detection"
		],
		"input": {
			"type": "httpjson"
		},
		"@timestamp": "2024-12-11T17:09:59.000Z",
		"ecs": {
			"version": "8.11.0"
		},
		"related": {
			"hosts": [
				" **********.eu-south-1.compute.internal"
			]
		},
		"data_stream": {
			"namespace": "default",
			"type": "logs",
			"dataset": "trend_micro_vision_one.detection"
		},
		"host": {
			"name": " ****.eu-south-1.compute.internal"
		},
		"details": {
			"groupId": " ************* -2okw77rZ7bD1Qk0q9bLfTRAk2VR",
			"pver": "2021-12-01T00:00:00.0000000Z",
			"searchDL": "DDL",
			"logReceivedTime": "1733937090989",
			"clusterId": " ************* -2okw77rZ7bD1Qk0q9bLfTRAk2VR",
			"uuid": "319bf2a2-02c9-4e24-b927-1b78725d7c94",
			"k8sNamespace": "sic-iam",
			"act": [
				"log"
			],
			"customTags": [
				"mitre_command_and_control",
				"process"
			],
			"endpointHostName": " ****.eu-south-1.compute.internal",
			"rawDataStr": "{\"clusterID\":\" *************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"************* \",\"customerID\":\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\",\"details\":{\"container.id\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"container.image.digest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"container.image.repository\":\" ****viva.jfrog.io/******** /tomcat\",\"container.image.tag\":\"9.0.73- ***\",\"container.name\":\"tomcat-container\",\"evt.arg.exe\":\"curl\",\"evt.arg.filename\":\"<NA>\",\"evt.arg.mode\":\"<NA>\",\"evt.arg.name\":\"<NA>\",\"evt.category\":\"process\",\"evt.dir\":\"<\",\"evt.num\":\"8267962726\",\"evt.rawtime\":\"1733936999010100132\",\"evt.time\":\"17:09:59.010100132\",\"evt.type\":\"execve\",\"k8s.ns.name\":\"sic-iam\",\"k8s.pod.id\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8s.pod.labels\":\"app:pnt-test-*** ,controller-revision-hash:pnt-test- ***-7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-*** -0\",\"k8s.pod.name\":\" **********\",\"proc.args\":\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/AAA\",\"proc.cmdline\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/*** \",\"proc.exe\":\"curl\",\"proc.exeline\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\",\"proc.exepath\":\"/usr/bin/curl\",\"proc.name\":\"curl\",\"proc.pcmdline\":\"bash ./new-entrypoint.sh\",\"proc.pid\":\"1534304\",\"proc.pid.ts\":\"1733936999008820540\",\"proc.pname\":\"bash\",\"proc.ppid\":\"37520\",\"proc.ppid.ts\":\"1733427545111053154\",\"user.loginuid\":\"-1\",\"user.name\":\"root\"},\"hostname\":\"****.eu-south-1.compute.internal\",\"id\":\"2q50NZGekOuxLvx2iLCOQYHouki\",\"k8s.ns.name\":\"sic-iam\",\"k8s.pod.labels\":{\"app\":\"pnt-test- ***\",\"controller-revision-hash\":\"pnt-test-*** -7668f9d96d\",\"statefulset.kubernetes.io/pod-name\":\"pnt-test- ***-0\"},\"k8s.pod.name\":\"pnt-test-*** -0\",\"mitigation\":\"log\",\"name\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"orchestration\":\"k8s\",\"policyID\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"productCode\":\"scs\",\"ruleID\":\"TM-00000049\",\"rulesets\":[{\"id\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"name\":\"LogOnlyRuleset\"}],\"severity\":\"notice\",\"tags\":[\"mitre_command_and_control\",\"process\"],\"timestamp\":\"2024-12-11T17:09:59.010100Z\",\"type\":\"syscall\",\"version\":\"2021-12-01\"}",
			"processName": "curl",
			"ruleSetName": "LogOnlyRuleset",
			"containerName": "tomcat-container",
			"parentCmd": "bash ./new-entrypoint.sh",
			"clusterName": " *************",
			"eventTime": 1733936999000,
			"eventName": "SECURITY_RISK_DETECTION",
			"ruleName": "(T1105)Launch Ingress Remote File Copy Tools in Container",
			"containerId": "7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe",
			"containerImage": " ****viva.jfrog.io/******** /tomcat:9.0.73-***",
			"ruleSetId": "LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th",
			"severity": 4,
			"clusterType": "k8s",
			"eventId": "100119",
			"processCmd": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
			"policyName": "LogOnlyPolicy",
			"pname": "Vision One Container Security",
			"parentPid": 37520,
			"eventSourceType": 3,
			"k8sPodId": "b777d8d9-441e-46ab-b34f-dac23b316636",
			"ruleIdStr": "TM-00000049",
			"containerImageDigest": "sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563",
			"logKey": "e99c5cff-18f2-2e55-1b2b-ac77279777b5",
			"eventTimeDT": "2024-12-11T17:09:59+00:00",
			"detectionType": "process",
			"parentName": "bash",
			"productCode": "scs",
			"policyId": "LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq",
			"processPid": 1534304,
			"sourceType": "syscall",
			"k8sPodName": "pnt-test-***-0",
			"rt_utc": "2024-12-11T17:09:59.0000000Z"
		},
		"event": {
			"agent_id_status": "auth_metadata_missing",
			"ingested": "2024-12-11T17:12:14Z",
			"original": "{\"act\":[\"log\"],\"clusterId\":\" *************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"************* \",\"clusterType\":\"k8s\",\"containerId\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"containerImage\":\" ***********.jfrog.io/*********** /tomcat:9.0.73-***\",\"containerImageDigest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"containerName\":\"tomcat-container\",\"customTags\":[\"mitre_command_and_control\",\"process\"],\"detectionType\":\"process\",\"endpointHostName\":\"****.eu-south-1.compute.internal\",\"eventId\":\"100119\",\"eventName\":\"SECURITY_RISK_DETECTION\",\"eventSourceType\":3,\"eventTime\":1733936999000,\"eventTimeDT\":\"2024-12-11T17:09:59+00:00\",\"groupId\":\" *************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"k8sNamespace\":\"sic-iam\",\"k8sPodId\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8sPodName\":\"pnt-test-*** -0\",\"logKey\":\"e99c5cff-18f2-2e55-1b2b-ac77279777b5\",\"logReceivedTime\":\"1733937090989\",\"parentCmd\":\"bash ./new-entrypoint.sh\",\"parentName\":\"bash\",\"parentPid\":37520,\"pname\":\"Vision One Container Security\",\"policyId\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"processCmd\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\",\"processName\":\"curl\",\"processPid\":1534304,\"productCode\":\"scs\",\"pver\":\"2021-12-01T00:00:00.0000000Z\",\"rawDataStr\":\"{\\\"clusterID\\\":\\\"************* -2okw77rZ7bD1Qk0q9bLfTRAk2VR\\\",\\\"clusterName\\\":\\\" *************\\\",\\\"customerID\\\":\\\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\\\",\\\"details\\\":{\\\"container.id\\\":\\\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\\\",\\\"container.image.digest\\\":\\\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\\\",\\\"container.image.repository\\\":\\\"**** viva.jfrog.io/ ********/tomcat\\\",\\\"container.image.tag\\\":\\\"9.0.73-*** \\\",\\\"container.name\\\":\\\"tomcat-container\\\",\\\"evt.arg.exe\\\":\\\"curl\\\",\\\"evt.arg.filename\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.mode\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.name\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.category\\\":\\\"process\\\",\\\"evt.dir\\\":\\\"\\u003c\\\",\\\"evt.num\\\":\\\"8267962726\\\",\\\"evt.rawtime\\\":\\\"1733936999010100132\\\",\\\"evt.time\\\":\\\"17:09:59.010100132\\\",\\\"evt.type\\\":\\\"execve\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.id\\\":\\\"b777d8d9-441e-46ab-b34f-dac23b316636\\\",\\\"k8s.pod.labels\\\":\\\"app:pnt-test- ***,controller-revision-hash:pnt-test-*** -7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test- ***-0\\\",\\\"k8s.pod.name\\\":\\\"pnt-test-*** -0\\\",\\\"proc.args\\\":\\\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\\\",\\\"proc.cmdline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/*** \\\",\\\"proc.exe\\\":\\\"curl\\\",\\\"proc.exeline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\\\",\\\"proc.exepath\\\":\\\"/usr/bin/curl\\\",\\\"proc.name\\\":\\\"curl\\\",\\\"proc.pcmdline\\\":\\\"bash ./new-entrypoint.sh\\\",\\\"proc.pid\\\":\\\"1534304\\\",\\\"proc.pid.ts\\\":\\\"1733936999008820540\\\",\\\"proc.pname\\\":\\\"bash\\\",\\\"proc.ppid\\\":\\\"37520\\\",\\\"proc.ppid.ts\\\":\\\"1733427545111053154\\\",\\\"user.loginuid\\\":\\\"-1\\\",\\\"user.name\\\":\\\"root\\\"},\\\"hostname\\\":\\\"****.eu-south-1.compute.internal\\\",\\\"id\\\":\\\"2q50NZGekOuxLvx2iLCOQYHouki\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.labels\\\":{\\\"app\\\":\\\"pnt-test- ***\\\",\\\"controller-revision-hash\\\":\\\"pnt-test-*** -7668f9d96d\\\",\\\"statefulset.kubernetes.io/pod-name\\\":\\\"pnt-test- ***-0\\\"},\\\"k8s.pod.name\\\":\\\"pnt-test-*** -0\\\",\\\"mitigation\\\":\\\"log\\\",\\\"name\\\":\\\"(T1105)Launch Ingress Remote File Copy Tools in Container\\\",\\\"orchestration\\\":\\\"k8s\\\",\\\"policyID\\\":\\\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\\\",\\\"policyName\\\":\\\"LogOnlyPolicy\\\",\\\"productCode\\\":\\\"scs\\\",\\\"ruleID\\\":\\\"TM-00000049\\\",\\\"rulesets\\\":[{\\\"id\\\":\\\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\\\",\\\"name\\\":\\\"LogOnlyRuleset\\\"}],\\\"severity\\\":\\\"notice\\\",\\\"tags\\\":[\\\"mitre_command_and_control\\\",\\\"process\\\"],\\\"timestamp\\\":\\\"2024-12-11T17:09:59.010100Z\\\",\\\"type\\\":\\\"syscall\\\",\\\"version\\\":\\\"2021-12-01\\\"}\",\"rt_utc\":\"2024-12-11T17:09:59.0000000Z\",\"ruleIdStr\":\"TM-00000049\",\"ruleName\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"ruleSetId\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"ruleSetName\":\"LogOnlyRuleset\",\"searchDL\":\"DDL\",\"severity\":4,\"sourceType\":\"syscall\",\"uuid\":\"319bf2a2-02c9-4e24-b927-1b78725d7c94\"}",
			"created": "2024-12-11T17:12:04.128Z",
			"kind": "event",
			"action": [
				"log"
			],
			"id": "100119",
			"category": [
				"intrusion_detection"
			],
			"type": [
				"info"
			],
			"dataset": "trend_micro_vision_one.detection"
		}
	},
	"fields": {
		"details.eventTimeDT": [
			"2024-12-11T17:09:59+00:00"
		],
		"details.sourceType": [
			"syscall"
		],
		"trend_micro_vision_one.detection.process.name": [
			"curl"
		],
		"details.eventTime": [
			1733936999000
		],
		"details.ruleName": [
			"(T1105)Launch Ingress Remote File Copy Tools in Container"
		],
		"trend_micro_vision_one.detection.endpoint.hostname": [
			" ****.eu-south-1.compute.internal"
		],
		"details.k8sPodId": [
			"b777d8d9-441e-46ab-b34f-dac23b316636"
		],
		"elastic_agent.version": [
			"8.16.1"
		],
		"event.category": [
			"intrusion_detection"
		],
		"process.name.text": [
			"curl"
		],
		"host.name.text": [
			" ****.eu-south-1.compute.internal"
		],
		"process.pid": [
			1534304
		],
		"details.clusterType": [
			"k8s"
		],
		"trend_micro_vision_one.detection.search_data_lake": [
			"DDL"
		],
		"details.groupId": [
			" ************* -2okw77rZ7bD1Qk0q9bLfTRAk2VR"
		],
		"details.ruleSetName": [
			"LogOnlyRuleset"
		],
		"details.detectionType": [
			"process"
		],
		"details.productCode": [
			"scs"
		],
		"details.k8sNamespace": [
			"sic-iam"
		],
		"details.endpointHostName": [
			" ****.eu-south-1.compute.internal"
		],
		"details.pver": [
			"2021-12-01T00:00:00.0000000Z"
		],
		"agent.name.text": [
			"Shuffle"
		],
		"details.searchDL": [
			"DDL"
		],
		"trend_micro_vision_one.detection.event_id": [
			"100119"
		],
		"agent.name": [
			"Shuffle"
		],
		"host.name": [
			" ****.eu-south-1.compute.internal"
		],
		"trend_micro_vision_one.detection.policy.name": [
			"LogOnlyPolicy"
		],
		"event.agent_id_status": [
			"auth_metadata_missing"
		],
		"event.kind": [
			"event"
		],
		"trend_micro_vision_one.detection.parent.cmd": [
			"bash ./new-entrypoint.sh"
		],
		"event.original": [
			"{\"act\":[\"log\"],\"clusterId\":\" *************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"************* \",\"clusterType\":\"k8s\",\"containerId\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"containerImage\":\" ****viva.jfrog.io/******** /tomcat:9.0.73-***\",\"containerImageDigest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"containerName\":\"tomcat-container\",\"customTags\":[\"mitre_command_and_control\",\"process\"],\"detectionType\":\"process\",\"endpointHostName\":\"****.eu-south-1.compute.internal\",\"eventId\":\"100119\",\"eventName\":\"SECURITY_RISK_DETECTION\",\"eventSourceType\":3,\"eventTime\":1733936999000,\"eventTimeDT\":\"2024-12-11T17:09:59+00:00\",\"groupId\":\" *************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"k8sNamespace\":\"sic-iam\",\"k8sPodId\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8sPodName\":\"pnt-test-*** -0\",\"logKey\":\"e99c5cff-18f2-2e55-1b2b-ac77279777b5\",\"logReceivedTime\":\"1733937090989\",\"parentCmd\":\"bash ./new-entrypoint.sh\",\"parentName\":\"bash\",\"parentPid\":37520,\"pname\":\"Vision One Container Security\",\"policyId\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"processCmd\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\",\"processName\":\"curl\",\"processPid\":1534304,\"productCode\":\"scs\",\"pver\":\"2021-12-01T00:00:00.0000000Z\",\"rawDataStr\":\"{\\\"clusterID\\\":\\\"************* -2okw77rZ7bD1Qk0q9bLfTRAk2VR\\\",\\\"clusterName\\\":\\\" *************\\\",\\\"customerID\\\":\\\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\\\",\\\"details\\\":{\\\"container.id\\\":\\\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\\\",\\\"container.image.digest\\\":\\\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\\\",\\\"container.image.repository\\\":\\\"**** viva.jfrog.io/ ********/tomcat\\\",\\\"container.image.tag\\\":\\\"9.0.73-*** \\\",\\\"container.name\\\":\\\"tomcat-container\\\",\\\"evt.arg.exe\\\":\\\"curl\\\",\\\"evt.arg.filename\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.mode\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.name\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.category\\\":\\\"process\\\",\\\"evt.dir\\\":\\\"\\u003c\\\",\\\"evt.num\\\":\\\"8267962726\\\",\\\"evt.rawtime\\\":\\\"1733936999010100132\\\",\\\"evt.time\\\":\\\"17:09:59.010100132\\\",\\\"evt.type\\\":\\\"execve\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.id\\\":\\\"b777d8d9-441e-46ab-b34f-dac23b316636\\\",\\\"k8s.pod.labels\\\":\\\"app:pnt-test- ***,controller-revision-hash:pnt-test-*** -7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test- ***-0\\\",\\\"k8s.pod.name\\\":\\\"pnt-test-*** -0\\\",\\\"proc.args\\\":\\\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\\\",\\\"proc.cmdline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/*** \\\",\\\"proc.exe\\\":\\\"curl\\\",\\\"proc.exeline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\\\",\\\"proc.exepath\\\":\\\"/usr/bin/curl\\\",\\\"proc.name\\\":\\\"curl\\\",\\\"proc.pcmdline\\\":\\\"bash ./new-entrypoint.sh\\\",\\\"proc.pid\\\":\\\"1534304\\\",\\\"proc.pid.ts\\\":\\\"1733936999008820540\\\",\\\"proc.pname\\\":\\\"bash\\\",\\\"proc.ppid\\\":\\\"37520\\\",\\\"proc.ppid.ts\\\":\\\"1733427545111053154\\\",\\\"user.loginuid\\\":\\\"-1\\\",\\\"user.name\\\":\\\"root\\\"},\\\"hostname\\\":\\\"****.eu-south-1.compute.internal\\\",\\\"id\\\":\\\"2q50NZGekOuxLvx2iLCOQYHouki\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.labels\\\":{\\\"app\\\":\\\"pnt-test- ***\\\",\\\"controller-revision-hash\\\":\\\"pnt-test-*** -7668f9d96d\\\",\\\"statefulset.kubernetes.io/pod-name\\\":\\\"pnt-test- ***-0\\\"},\\\"k8s.pod.name\\\":\\\"pnt-test-*** -0\\\",\\\"mitigation\\\":\\\"log\\\",\\\"name\\\":\\\"(T1105)Launch Ingress Remote File Copy Tools in Container\\\",\\\"orchestration\\\":\\\"k8s\\\",\\\"policyID\\\":\\\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\\\",\\\"policyName\\\":\\\"LogOnlyPolicy\\\",\\\"productCode\\\":\\\"scs\\\",\\\"ruleID\\\":\\\"TM-00000049\\\",\\\"rulesets\\\":[{\\\"id\\\":\\\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\\\",\\\"name\\\":\\\"LogOnlyRuleset\\\"}],\\\"severity\\\":\\\"notice\\\",\\\"tags\\\":[\\\"mitre_command_and_control\\\",\\\"process\\\"],\\\"timestamp\\\":\\\"2024-12-11T17:09:59.010100Z\\\",\\\"type\\\":\\\"syscall\\\",\\\"version\\\":\\\"2021-12-01\\\"}\",\"rt_utc\":\"2024-12-11T17:09:59.0000000Z\",\"ruleIdStr\":\"TM-00000049\",\"ruleName\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"ruleSetId\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"ruleSetName\":\"LogOnlyRuleset\",\"searchDL\":\"DDL\",\"severity\":4,\"sourceType\":\"syscall\",\"uuid\":\"319bf2a2-02c9-4e24-b927-1b78725d7c94\"}"
		],
		"trend_micro_vision_one.detection.action": [
			"log"
		],
		"details.logReceivedTime": [
			"1733937090989"
		],
		"trend_micro_vision_one.detection.product.name": [
			"Vision One Container Security"
		],
		"details.uuid": [
			"319bf2a2-02c9-4e24-b927-1b78725d7c94"
		],
		"input.type": [
			"httpjson"
		],
		"data_stream.type": [
			"logs"
		],
		"details.act": [
			"log"
		],
		"details.processName": [
			"curl"
		],
		"tags": [
			"preserve_original_event",
			"preserve_duplicate_custom_fields",
			"forwarded",
			"trend_micro_vision_one-detection"
		],
		"details.logKey": [
			"e99c5cff-18f2-2e55-1b2b-ac77279777b5"
		],
		"process.name": [
			"curl"
		],
		"details.containerImage": [
			" ****viva.jfrog.io/******** /tomcat:9.0.73-***"
		],
		"trend_micro_vision_one.detection.detection_type": [
			"process"
		],
		"agent.id": [
			"5f4edad9-8329-460f-adbf-c4789daa91b3"
		],
		"details.containerImageDigest": [
			"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563"
		],
		"details.parentPid": [
			37520
		],
		"ecs.version": [
			"8.11.0"
		],
		"event.created": [
			"2024-12-11T17:12:04.128Z"
		],
		"trend_micro_vision_one.detection.event_name": [
			"SECURITY_RISK_DETECTION"
		],
		"agent.version": [
			"8.16.1"
		],
		"related.hosts": [
			" ****.eu-south-1.compute.internal"
		],
		"details.eventSourceType": [
			3
		],
		"process.command_line.text": [
			"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
		],
		"details.containerId": [
			"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe"
		],
		"details.processPid": [
			1534304
		],
		"details.rt_utc": [
			"2024-12-11T17:09:59.0000000Z"
		],
		"details.ruleIdStr": [
			"TM-00000049"
		],
		"details.severity": [
			4
		],
		"details.parentName": [
			"bash"
		],
		"trend_micro_vision_one.detection.process.pid": [
			1534304
		],
		"details.ruleSetId": [
			"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th"
		],
		"details.eventName": [
			"SECURITY_RISK_DETECTION"
		],
		"trend_micro_vision_one.detection.product.code": [
			"scs"
		],
		"details.customTags": [
			"mitre_command_and_control",
			"process"
		],
		"details.policyName": [
			"LogOnlyPolicy"
		],
		"agent.type": [
			"filebeat"
		],
		"event.module": [
			"trend_micro_vision_one"
		],
		"trend_micro_vision_one.detection.policy.logkey": [
			"e99c5cff-18f2-2e55-1b2b-ac77279777b5"
		],
		"trend_micro_vision_one.detection.uuid": [
			"319bf2a2-02c9-4e24-b927-1b78725d7c94"
		],
		"details.clusterName": [
			" *************"
		],
		"elastic_agent.snapshot": [
			false
		],
		"details.pname": [
			"Vision One Container Security"
		],
		"details.k8sPodName": [
			"pnt-test-***-0"
		],
		"details.parentCmd": [
			"bash ./new-entrypoint.sh"
		],
		"elastic_agent.id": [
			"5f4edad9-8329-460f-adbf-c4789daa91b3"
		],
		"data_stream.namespace": [
			"default"
		],
		"details.rawDataStr": [
			"{\"clusterID\":\" *************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"************* \",\"customerID\":\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\",\"details\":{\"container.id\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"container.image.digest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"container.image.repository\":\" ****viva.jfrog.io/******** /tomcat\",\"container.image.tag\":\"9.0.73- ***\",\"container.name\":\"tomcat-container\",\"evt.arg.exe\":\"curl\",\"evt.arg.filename\":\"<NA>\",\"evt.arg.mode\":\"<NA>\",\"evt.arg.name\":\"<NA>\",\"evt.category\":\"process\",\"evt.dir\":\"<\",\"evt.num\":\"8267962726\",\"evt.rawtime\":\"1733936999010100132\",\"evt.time\":\"17:09:59.010100132\",\"evt.type\":\"execve\",\"k8s.ns.name\":\"sic-iam\",\"k8s.pod.id\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8s.pod.labels\":\"app:pnt-test-*** ,controller-revision-hash:pnt-test- ***-7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-*** -0\",\"k8s.pod.name\":\"pnt-test- ***-0\",\"proc.args\":\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/*** \",\"proc.cmdline\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\",\"proc.exe\":\"curl\",\"proc.exeline\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/*** \",\"proc.exepath\":\"/usr/bin/curl\",\"proc.name\":\"curl\",\"proc.pcmdline\":\"bash ./new-entrypoint.sh\",\"proc.pid\":\"1534304\",\"proc.pid.ts\":\"1733936999008820540\",\"proc.pname\":\"bash\",\"proc.ppid\":\"37520\",\"proc.ppid.ts\":\"1733427545111053154\",\"user.loginuid\":\"-1\",\"user.name\":\"root\"},\"hostname\":\" ****.eu-south-1.compute.internal\",\"id\":\"2q50NZGekOuxLvx2iLCOQYHouki\",\"k8s.ns.name\":\"sic-iam\",\"k8s.pod.labels\":{\"app\":\"pnt-test-*** \",\"controller-revision-hash\":\"pnt-test- ***-7668f9d96d\",\"statefulset.kubernetes.io/pod-name\":\"pnt-test-*** -0\"},\"k8s.pod.name\":\"pnt-test-***-0\",\"mitigation\":\"log\",\"name\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"orchestration\":\"k8s\",\"policyID\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"productCode\":\"scs\",\"ruleID\":\"TM-00000049\",\"rulesets\":[{\"id\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"name\":\"LogOnlyRuleset\"}],\"severity\":\"notice\",\"tags\":[\"mitre_command_and_control\",\"process\"],\"timestamp\":\"2024-12-11T17:09:59.010100Z\",\"type\":\"syscall\",\"version\":\"2021-12-01\"}"
		],
		"trend_micro_vision_one.detection.product.version": [
			"2021-12-01T00:00:00.0000000Z"
		],
		"details.containerName": [
			"tomcat-container"
		],
		"details.clusterId": [
			" ************* -2okw77rZ7bD1Qk0q9bLfTRAk2VR"
		],
		"details.policyId": [
			"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq"
		],
		"event.action": [
			"log"
		],
		"event.ingested": [
			"2024-12-11T17:12:14.000Z"
		],
		"@timestamp": [
			"2024-12-11T17:09:59.000Z"
		],
		"data_stream.dataset": [
			"trend_micro_vision_one.detection"
		],
		"event.type": [
			"info"
		],
		"process.command_line": [
			"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
		],
		"trend_micro_vision_one.detection.process.cmd": [
			"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
		],
		"agent.ephemeral_id": [
			"875a1d57-8cb8-4fed-802f-4e4deb2bef60"
		],
		"trend_micro_vision_one.detection.event_time_dt": [
			"2024-12-11T17:09:59.000Z"
		],
		"details.eventId": [
			"100119"
		],
		"event.id": [
			"100119"
		],
		"details.processCmd": [
			"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
		],
		"trend_micro_vision_one.detection.rt_utc": [
			"2024-12-11T17:09:59.000Z"
		],
		"event.dataset": [
			"trend_micro_vision_one.detection"
		]
	}
}

```

Also, here the screenshots of the ingestion policies:

 ![immagine](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bbd2f2b8dd70301b1de585939c43508c546ec7e0.png)

 ![immagine](https://us1.discourse-cdn.com/elastic/original/3X/2/4/24160c7a90f0227cd593d9bcefe3b4b6cfac532b.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 12, 2024, 2:18pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/12 "2024-12-12T14:18:05Z")

</div>

It would be better if you could post the pipeline text instead of images...

I have some other things to do if I get a chance I will look in the meantime you should try to post the document... Which will call the pipelines and see what happens

---

<div class="post-metadata">

**Author:** ![Cristina\_Marletta\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristina_marletta_li/32/137793_2.png) [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\
**Post date:** [December 12, 2024, 2:21pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/13 "2024-12-12T14:21:49Z")

</div>

add\_field pipeline:

```auto
[
  {
    "json": {
      "field": "event.original",
      "target_field": "details",
      "if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
    }
  },
  {
    "json": {
      "field": "details.rawDataStr",
      "target_field": "rawdetails",
      "if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
    }
  }
]

```

pipeline logs-trend\_micro\_vision\_one.detection@custom pipeline:

```auto

Processors

    [
      {
        "pipeline": {
          "name": "add_field"
        }
      }
    ]

Failure processors

    [
      {
        "set": {
          "field": "pipelineerror",
          "value": "details.rawDataStr"
        }
      }
    ]

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 12, 2024, 2:32pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/14 "2024-12-12T14:32:33Z")

</div>

Is there a other

You can run in Kibana Dev Tool

`GET _ingest/pipeline/logs-trend_micro_vision_one.detection@custom`

And that will show the whole pipeline.

Ok I will take a look later today if I can

And you can try to POST a document to the data stream.. You will probably need to change the time so it does not think it is the same document

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 12, 2024, 5:05pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/15 "2024-12-12T17:05:28Z")

</div>

Hi @Cristina_Marletta_Li

Ok I have found it..... wow that was pretty buried...

The problem is that the data inside the `details.rawDataStr` is invalid JSON... and thus was creating invalid JSON after expanding and causing a mapping issue and dropping the events.

Parts of it the bad JSON looks like this...

```auto
"details": {
....
		"proc.pid": "1534304", << CAN NOT HAVE THIS AND 
		"proc.pid.ts": "1733936999008820540", << THIS IN JSON THEY COLLIDE
		"proc.pname": "bash",
		"proc.ppid": "37520",
		"proc.ppid.ts": "1733427545111053154",
		"user.loginuid": "-1",
		"user.name": "root"

```

So what I did is this... inside the `details.rawDataStr` I replaced all those dots `.` with underscores `_`

```auto
PUT _ingest/pipeline/logs-trend_micro_vision_one.detection@custom
{
  "processors": [
    {
      "json": {
        "field": "event.original",
        "target_field": "details",
        "if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
      }
    },
    {
      "gsub": {
        "field": "details.rawDataStr",
        "pattern": """\.""",
        "replacement": "_"
      }
    },
    {
      "json": {
        "field": "details.rawDataStr",
        "target_field": "rawdetails",
        "if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
      }
    }
  ]
}

```

Then I think it works

```auto
POST _ingest/pipeline/logs-trend_micro_vision_one.detection-1.22.0/_simulate
{
  "docs": [
    {
      "_source" : 
      {
        "tags": [
          "preserve_original_event",
          "preserve_duplicate_custom_fields",
          "forwarded",
          "trend_micro_vision_one-detection"
          ],
          "@timestamp": "2024-12-11T17:09:59.000Z",
          "message": "{\"act\":[\"log\"],\"clusterId\":\" *************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"************* \",\"clusterType\":\"k8s\",\"containerId\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"containerImage\":\" ***********.jfrog.io/*********** /tomcat:9.0.73-***\",\"containerImageDigest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"containerName\":\"tomcat-container\",\"customTags\":[\"mitre_command_and_control\",\"process\"],\"detectionType\":\"process\",\"endpointHostName\":\"****.eu-south-1.compute.internal\",\"eventId\":\"100119\",\"eventName\":\"SECURITY_RISK_DETECTION\",\"eventSourceType\":3,\"eventTime\":1733936999000,\"eventTimeDT\":\"2024-12-11T17:09:59+00:00\",\"groupId\":\" *************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"k8sNamespace\":\"sic-iam\",\"k8sPodId\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8sPodName\":\"pnt-test-*** -0\",\"logKey\":\"e99c5cff-18f2-2e55-1b2b-ac77279777b5\",\"logReceivedTime\":\"1733937090989\",\"parentCmd\":\"bash ./new-entrypoint.sh\",\"parentName\":\"bash\",\"parentPid\":37520,\"pname\":\"Vision One Container Security\",\"policyId\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"processCmd\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\",\"processName\":\"curl\",\"processPid\":1534304,\"productCode\":\"scs\",\"pver\":\"2021-12-01T00:00:00.0000000Z\",\"rawDataStr\":\"{\\\"clusterID\\\":\\\"************* -2okw77rZ7bD1Qk0q9bLfTRAk2VR\\\",\\\"clusterName\\\":\\\" *************\\\",\\\"customerID\\\":\\\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\\\",\\\"details\\\":{\\\"container.id\\\":\\\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\\\",\\\"container.image.digest\\\":\\\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\\\",\\\"container.image.repository\\\":\\\"**** viva.jfrog.io/ ********/tomcat\\\",\\\"container.image.tag\\\":\\\"9.0.73-*** \\\",\\\"container.name\\\":\\\"tomcat-container\\\",\\\"evt.arg.exe\\\":\\\"curl\\\",\\\"evt.arg.filename\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.mode\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.name\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.category\\\":\\\"process\\\",\\\"evt.dir\\\":\\\"\\u003c\\\",\\\"evt.num\\\":\\\"8267962726\\\",\\\"evt.rawtime\\\":\\\"1733936999010100132\\\",\\\"evt.time\\\":\\\"17:09:59.010100132\\\",\\\"evt.type\\\":\\\"execve\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.id\\\":\\\"b777d8d9-441e-46ab-b34f-dac23b316636\\\",\\\"k8s.pod.labels\\\":\\\"app:pnt-test- ***,controller-revision-hash:pnt-test-*** -7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test- ***-0\\\",\\\"k8s.pod.name\\\":\\\"pnt-test-*** -0\\\",\\\"proc.args\\\":\\\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\\\",\\\"proc.cmdline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/*** \\\",\\\"proc.exe\\\":\\\"curl\\\",\\\"proc.exeline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/ ***\\\",\\\"proc.exepath\\\":\\\"/usr/bin/curl\\\",\\\"proc.name\\\":\\\"curl\\\",\\\"proc.pcmdline\\\":\\\"bash ./new-entrypoint.sh\\\",\\\"proc.pid\\\":\\\"1534304\\\",\\\"proc.pid.ts\\\":\\\"1733936999008820540\\\",\\\"proc.pname\\\":\\\"bash\\\",\\\"proc.ppid\\\":\\\"37520\\\",\\\"proc.ppid.ts\\\":\\\"1733427545111053154\\\",\\\"user.loginuid\\\":\\\"-1\\\",\\\"user.name\\\":\\\"root\\\"},\\\"hostname\\\":\\\"****.eu-south-1.compute.internal\\\",\\\"id\\\":\\\"2q50NZGekOuxLvx2iLCOQYHouki\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.labels\\\":{\\\"app\\\":\\\"pnt-test- ***\\\",\\\"controller-revision-hash\\\":\\\"pnt-test-*** -7668f9d96d\\\",\\\"statefulset.kubernetes.io/pod-name\\\":\\\"pnt-test- ***-0\\\"},\\\"k8s.pod.name\\\":\\\"pnt-test-*** -0\\\",\\\"mitigation\\\":\\\"log\\\",\\\"name\\\":\\\"(T1105)Launch Ingress Remote File Copy Tools in Container\\\",\\\"orchestration\\\":\\\"k8s\\\",\\\"policyID\\\":\\\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\\\",\\\"policyName\\\":\\\"LogOnlyPolicy\\\",\\\"productCode\\\":\\\"scs\\\",\\\"ruleID\\\":\\\"TM-00000049\\\",\\\"rulesets\\\":[{\\\"id\\\":\\\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\\\",\\\"name\\\":\\\"LogOnlyRuleset\\\"}],\\\"severity\\\":\\\"notice\\\",\\\"tags\\\":[\\\"mitre_command_and_control\\\",\\\"process\\\"],\\\"timestamp\\\":\\\"2024-12-11T17:09:59.010100Z\\\",\\\"type\\\":\\\"syscall\\\",\\\"version\\\":\\\"2021-12-01\\\"}\",\"rt_utc\":\"2024-12-11T17:09:59.0000000Z\",\"ruleIdStr\":\"TM-00000049\",\"ruleName\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"ruleSetId\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"ruleSetName\":\"LogOnlyRuleset\",\"searchDL\":\"DDL\",\"severity\":4,\"sourceType\":\"syscall\",\"uuid\":\"319bf2a2-02c9-4e24-b927-1b78725d7c94\"}"
      }
    }
    ]
}

```

Results in .....

```auto
          "details": {
              "proc_exepath": "/usr/bin/curl",
              "proc_args": "--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
              "container_image_tag": "9_0_73-***",
              "evt_arg_name": "<NA>",
              "user_name": "root",
              "proc_ppid_ts": "1733427545111053154", << OK NOW
              "proc_ppid": "37520", << OK NOW

```

And when I did a POST of that message into the actual data stream then it worked...

IF you learn to use `_simulate` you can see that but even for me not obvious at first....

Give it a try and let me know

---

<div class="post-metadata">

**Author:** ![Cristina\_Marletta\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristina_marletta_li/32/137793_2.png) [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\
**Post date:** [December 14, 2024, 4:39pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/16 "2024-12-14T16:39:40Z")

</div>

Hi @stephenb,  
I would say it works. Now I would like to simplify everything. Instead of applying the JSON processor twice, I would like to simply extract the field I am interested in from details.rawDataStr. In particular, I would like to extract the value of the field details.k8s.pod.labels from details.rawDataStr and use it to enhance a new target field. How can I do this?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 14, 2024, 6:35pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/17 "2024-12-14T18:35:08Z")

</div>

> [@Cristina\_Marletta\_Li](#):
>
> I would say it works. Now I would like to simplify everything. Instead of applying the JSON processor twice, I would like to simply extract the field I am interested in from details.rawDataStr. In particular, I would like to extract the value of the field details.k8s.pod.labels from details.rawDataStr and use it to enhance a new target field. How can I do this?

hehehe Why did you not say that in the first place 🙂

1. First, there are several fields and details within `raw details`, the expansion of `details.rawDataStr`. Which one(s) do you want? I posted the whole raw details below.

2. The only way to extract the field(s) will be to use a GROK processor, which is Regex under the covers ... which may or may not simplify... or be more efficient... unwrapping the JSON picking out the strings and then cleaning up maybe a couple more lines of code but may be much cleaner / more efficient...

THis is the whole raw details which ones _ **exactly** _ do you want A,B,C,D something other?

```auto
          "rawdetails": {
            "severity": "notice",
            "mitigation": "log",
            "rulesets": [
              {
                "name": "LogOnlyRuleset",
                "id": "LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th"
              }
            ],
            "orchestration": "k8s",
            "k8s.pod.labels": { A) <<< ONE OF THESE? 
              "app": "pnt-test-***",
              "statefulset.kubernetes.io/pod-name": "pnt-test-***-0",
              "controller-revision-hash": "pnt-test-***-7668f9d96d"
            },
            "policyName": "LogOnlyPolicy",
            "clusterID": " ************* -2okw77rZ7bD1Qk0q9bLfTRAk2VR",
            "type": "syscall",
            "k8s.ns.name": "sic-iam",
            "version": "2021-12-01",
            "tags": [
              "mitre_command_and_control",
              "process"
            ],
            "hostname": " ****.eu-south-1.compute.internal",
            "productCode": "scs",
            "k8s.pod.name": "pnt-test-***-0", <<< B) THIS ONE? 

            "policyID": "LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq",
            "clusterName": " *************",
            "customerID": "7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7",
            "name": "(T1105)Launch Ingress Remote File Copy Tools in Container",
            "details": {
              "proc.pname": "bash",
              "container.image.tag": "9.0.73-***",
              "evt.type": "execve",
              "proc.cmdline": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
              "proc.args": "--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
              "proc.ppid": "37520",
              "proc.exe": "curl",
              "user.name": "root",
              "evt.arg.mode": "<NA>",
              "k8s.ns.name": "sic-iam",
              "container.id": "7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe",
              "proc.pid": "1534304",
              "evt.num": "8267962726",
              "container.name": "tomcat-container",

              "k8s.pod.name": "pnt-test-***-0", <<< C) THiS ONE???
              "proc.name": "curl",
              "proc.pcmdline": "bash ./new-entrypoint.sh",
              "evt.time": "17:09:59.010100132",
              "proc.exeline": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
              "evt.arg.filename": "<NA>",

               ### D) THIS WHOLE STRING
              "k8s.pod.labels": "app:pnt-test- ***,controller-revision-hash:pnt-test-*** -7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-***-0",

              "evt.arg.name": "<NA>",
              "evt.dir": "<",
              "proc.exepath": "/usr/bin/curl",
              "proc.pid.ts": "1733936999008820540",
              "user.loginuid": "-1",
              "container.image.repository": " ****viva.jfrog.io/******** /tomcat",
              "k8s.pod.id": "b777d8d9-441e-46ab-b34f-dac23b316636",
              "evt.rawtime": "1733936999010100132",
              "container.image.digest": "sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563",
              "proc.ppid.ts": "1733427545111053154",
              "evt.arg.exe": "curl",
              "evt.category": "process"
            },
            "id": "2q50NZGekOuxLvx2iLCOQYHouki",
            "ruleID": "TM-00000049",
            "timestamp": "2024-12-11T17:09:59.010100Z"
          }

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 14, 2024, 6:42pm UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/18 "2024-12-14T18:42:42Z")

</div>

Here is a pipeline that works... without the GROK... I think this is pretty straightforward and efficient...

```auto
PUT _ingest/pipeline/logs-trend_micro_vision_one.detection@custom
{
  "processors": [
    {
      "json": {
        "field": "event.original",
        "target_field": "details",
        "if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
      }
    },
    {
      "gsub": {
        "field": "details.rawDataStr",
        "pattern": """\.""",
        "replacement": "_"
      }
    },
    {
      "json": {
        "field": "details.rawDataStr",
        "target_field": "rawdetails",
        "if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
      }
    },
    {
      "rename": {
        "field": "rawdetails.details.k8s_pod_labels",
        "target_field": "k8s.pod.labels"
      }
    },
    {
      "remove": {
        "field": ["details", "rawdetails"]
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 15, 2024, 12:26am UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/19 "2024-12-15T00:26:42Z")

</div>

Hi @Cristina_Marletta_Li

After some playing, try this... I am surprised it works...  
This seems easy after the fact but it was not 🙂  
You might need to play with it; perhaps you should look at the [grok](https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html) documentation.

```auto
PUT _ingest/pipeline/logs-trend_micro_vision_one.detection@custom
{
  "processors": [
    {
      "json": {
        "field": "event.original",
        "target_field": "details",
        "if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
      }
    },
    {
      "grok": {
        "field": "details.rawDataStr",
        "patterns": [
            """%{DATA}"k8s.pod.labels":"%{DATA:k8s.pod.labels}"%{DATA}"""
        ]
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Cristina\_Marletta\_Li](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cristina_marletta_li/32/137793_2.png) [@Cristina\_Marletta\_Li](https://discuss.elastic.co/u/Cristina_Marletta_Li)\
**Post date:** [December 15, 2024, 10:20am UTC](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768/20 "2024-12-15T10:20:12Z")

</div>

Hi @stephenb ,  
this way the '.' is replaced with '\_' not only in the field name but also in the value.  
For example,  
10:03:12.496882712 \> 10:03:12\_496882712  
and this is a problem.

[Next page](https://discuss.elastic.co/t/double-processor-in-ingest-pipeline/371768.md?page=2)