Hi all,
it is possible to create a custom ingest pipeline associated with an integration and a specific Data Stream with 2 processors.
The second one should process a field created by the first one.
Yes, absolutely
An ingest pipeline executes the processors in sequence so you can create a field in one processor and then use it or modify it in a later processor.
Sometimes it's a bit tricky. If you give us an example, maybe we can help.
Hi,
hi,
the first processor does this:
[
{
"json": {
"field": "event.original",
"target_field": "details"
}
}
]
among the generated fields there is details.rawDataStr which, in turn, has the json structure. I would like to parse it from JSON.
Yes, Just build it like you would procedural language, something like this ... you might need to use some conditional logic or ignore failures etc
There might need to be some need to do some cleaning up etc..
I would suggest looking at the docs here
{
"json": {
"field": "event.original",
"target_field": "details"
}
},
{
"json": {
"field": "details.rawDataStr",
"target_field": "rawdetails"
}
}
If you give an example document perhaps we could take a quick look
Thankk you stephenb,
I will study your document.
Maybe I should add a condition to check that details.rawDataStr is not empty (this can happen)?
Absolutely! Or you can just set the ignore failure and it'll just drop through...
I highly recommend reading through the documentation if you're going to start using ingest pipelines. They're very powerful and they're as conditionals and error handling or simple drop-thru. Lots of good stuff
I also would recommend kind of building modular. If you have pipelines you want to reuse you can just call them from other pipelines
But that's a little more advanced. Just get started. Come back with some more questionsBut that's a little more advanced. Just get started. Come back with some more questions
Hi stephenb,
unfortunately something is wrong. The pipeline is associated with an integration (logs-trend_micro_vision_one.detection@custom). This pipeline refers to another (add_field) defined like this:
[
{
"json": {
"field": "event.original",
"target_field": "details"
}
},
{
"json": {
"field": "details.rawDataStr",
"target_field": "rawdetails",
"ignore_failure": true
}
}
]
Attached, I report the field to be processed and the screenshot of the definition of the pipeline logs-trend_micro_vision_one.detection@custom.
Furthermore, since I inserted the new processor, docs containing details.rawDataStr are no longer loaded.
In order to debug you will need to provide a full sample document in JSON and the full pipeline....
You can test with _simulate
And also use verbose
Then in addition there could be a mapping error...
So get a raw event and try to me manuallyPOST it to the data stream
POST logs-trend_micro_vision_one.detection-default/_doc
{
Raw document
}
I tested from GUI with 'Test pipeline'. No errors were found.
No doc containing the data I'm interested in is written to datastream.
How can I send you the full document? I can only attach images and the document is very long.
Unfortunately the GUI test pipeline that may mean the pipeline works but when the doc is written there is mapping conflict and it is not written because it is rejected.
You need to try to POST a document from the dev tools.
See my instructions above.
That was s how you are going to need to test if you can not prove s sample document and full pipeline
I'll try to post here the original document in JSON after the first processing (that creates the details.* fields). This was the situation before I added the JSON second processor:
{
"_index": ".ds-logs-trend_micro_vision_one.detection-default-2024.12.10-000004",
"_id": "qNQeFHRFBRO02UQv2jILApKG/oA=",
"_version": 1,
"_score": 0,
"_source": {
"agent": {
"name": "Shuffle",
"id": "5f4edad9-8329-460f-adbf-c4789daa91b3",
"ephemeral_id": "875a1d57-8cb8-4fed-802f-4e4deb2bef60",
"type": "filebeat",
"version": "8.16.1"
},
"trend_micro_vision_one": {
"detection": {
"event_time_dt": "2024-12-11T17:09:59.000Z",
"parent": {
"cmd": "bash ./new-entrypoint.sh"
},
"process": {
"name": "curl",
"pid": 1534304,
"cmd": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
},
"product": {
"code": "scs",
"name": "Vision One Container Security",
"version": "2021-12-01T00:00:00.0000000Z"
},
"search_data_lake": "DDL",
"uuid": "319bf2a2-02c9-4e24-b927-1b78725d7c94",
"endpoint": {
"hostname": "***********.eu-south-1.compute.internal"
},
"event_id": "100119",
"detection_type": "process",
"action": [
"log"
],
"event_name": "SECURITY_RISK_DETECTION",
"rt_utc": "2024-12-11T17:09:59.000Z",
"policy": {
"logkey": "e99c5cff-18f2-2e55-1b2b-ac77279777b5",
"name": "LogOnlyPolicy"
}
}
},
"process": {
"name": "curl",
"pid": 1534304,
"command_line": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
},
"elastic_agent": {
"id": "5f4edad9-8329-460f-adbf-c4789daa91b3",
"version": "8.16.1",
"snapshot": false
},
"tags": [
"preserve_original_event",
"preserve_duplicate_custom_fields",
"forwarded",
"trend_micro_vision_one-detection"
],
"input": {
"type": "httpjson"
},
"@timestamp": "2024-12-11T17:09:59.000Z",
"ecs": {
"version": "8.11.0"
},
"related": {
"hosts": [
"**********.eu-south-1.compute.internal"
]
},
"data_stream": {
"namespace": "default",
"type": "logs",
"dataset": "trend_micro_vision_one.detection"
},
"host": {
"name": "****.eu-south-1.compute.internal"
},
"details": {
"groupId": "*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR",
"pver": "2021-12-01T00:00:00.0000000Z",
"searchDL": "DDL",
"logReceivedTime": "1733937090989",
"clusterId": "*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR",
"uuid": "319bf2a2-02c9-4e24-b927-1b78725d7c94",
"k8sNamespace": "sic-iam",
"act": [
"log"
],
"customTags": [
"mitre_command_and_control",
"process"
],
"endpointHostName": "****.eu-south-1.compute.internal",
"rawDataStr": "{\"clusterID\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"*************\",\"customerID\":\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\",\"details\":{\"container.id\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"container.image.digest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"container.image.repository\":\"****viva.jfrog.io/********/tomcat\",\"container.image.tag\":\"9.0.73-***\",\"container.name\":\"tomcat-container\",\"evt.arg.exe\":\"curl\",\"evt.arg.filename\":\"<NA>\",\"evt.arg.mode\":\"<NA>\",\"evt.arg.name\":\"<NA>\",\"evt.category\":\"process\",\"evt.dir\":\"<\",\"evt.num\":\"8267962726\",\"evt.rawtime\":\"1733936999010100132\",\"evt.time\":\"17:09:59.010100132\",\"evt.type\":\"execve\",\"k8s.ns.name\":\"sic-iam\",\"k8s.pod.id\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8s.pod.labels\":\"app:pnt-test-***,controller-revision-hash:pnt-test-***-7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-***-0\",\"k8s.pod.name\":\"**********\",\"proc.args\":\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/AAA\",\"proc.cmdline\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\",\"proc.exe\":\"curl\",\"proc.exeline\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\",\"proc.exepath\":\"/usr/bin/curl\",\"proc.name\":\"curl\",\"proc.pcmdline\":\"bash ./new-entrypoint.sh\",\"proc.pid\":\"1534304\",\"proc.pid.ts\":\"1733936999008820540\",\"proc.pname\":\"bash\",\"proc.ppid\":\"37520\",\"proc.ppid.ts\":\"1733427545111053154\",\"user.loginuid\":\"-1\",\"user.name\":\"root\"},\"hostname\":\"****.eu-south-1.compute.internal\",\"id\":\"2q50NZGekOuxLvx2iLCOQYHouki\",\"k8s.ns.name\":\"sic-iam\",\"k8s.pod.labels\":{\"app\":\"pnt-test-***\",\"controller-revision-hash\":\"pnt-test-***-7668f9d96d\",\"statefulset.kubernetes.io/pod-name\":\"pnt-test-***-0\"},\"k8s.pod.name\":\"pnt-test-***-0\",\"mitigation\":\"log\",\"name\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"orchestration\":\"k8s\",\"policyID\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"productCode\":\"scs\",\"ruleID\":\"TM-00000049\",\"rulesets\":[{\"id\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"name\":\"LogOnlyRuleset\"}],\"severity\":\"notice\",\"tags\":[\"mitre_command_and_control\",\"process\"],\"timestamp\":\"2024-12-11T17:09:59.010100Z\",\"type\":\"syscall\",\"version\":\"2021-12-01\"}",
"processName": "curl",
"ruleSetName": "LogOnlyRuleset",
"containerName": "tomcat-container",
"parentCmd": "bash ./new-entrypoint.sh",
"clusterName": "*************",
"eventTime": 1733936999000,
"eventName": "SECURITY_RISK_DETECTION",
"ruleName": "(T1105)Launch Ingress Remote File Copy Tools in Container",
"containerId": "7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe",
"containerImage": "****viva.jfrog.io/********/tomcat:9.0.73-***",
"ruleSetId": "LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th",
"severity": 4,
"clusterType": "k8s",
"eventId": "100119",
"processCmd": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
"policyName": "LogOnlyPolicy",
"pname": "Vision One Container Security",
"parentPid": 37520,
"eventSourceType": 3,
"k8sPodId": "b777d8d9-441e-46ab-b34f-dac23b316636",
"ruleIdStr": "TM-00000049",
"containerImageDigest": "sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563",
"logKey": "e99c5cff-18f2-2e55-1b2b-ac77279777b5",
"eventTimeDT": "2024-12-11T17:09:59+00:00",
"detectionType": "process",
"parentName": "bash",
"productCode": "scs",
"policyId": "LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq",
"processPid": 1534304,
"sourceType": "syscall",
"k8sPodName": "pnt-test-***-0",
"rt_utc": "2024-12-11T17:09:59.0000000Z"
},
"event": {
"agent_id_status": "auth_metadata_missing",
"ingested": "2024-12-11T17:12:14Z",
"original": "{\"act\":[\"log\"],\"clusterId\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"*************\",\"clusterType\":\"k8s\",\"containerId\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"containerImage\":\"***********.jfrog.io/***********/tomcat:9.0.73-***\",\"containerImageDigest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"containerName\":\"tomcat-container\",\"customTags\":[\"mitre_command_and_control\",\"process\"],\"detectionType\":\"process\",\"endpointHostName\":\"****.eu-south-1.compute.internal\",\"eventId\":\"100119\",\"eventName\":\"SECURITY_RISK_DETECTION\",\"eventSourceType\":3,\"eventTime\":1733936999000,\"eventTimeDT\":\"2024-12-11T17:09:59+00:00\",\"groupId\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"k8sNamespace\":\"sic-iam\",\"k8sPodId\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8sPodName\":\"pnt-test-***-0\",\"logKey\":\"e99c5cff-18f2-2e55-1b2b-ac77279777b5\",\"logReceivedTime\":\"1733937090989\",\"parentCmd\":\"bash ./new-entrypoint.sh\",\"parentName\":\"bash\",\"parentPid\":37520,\"pname\":\"Vision One Container Security\",\"policyId\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"processCmd\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\",\"processName\":\"curl\",\"processPid\":1534304,\"productCode\":\"scs\",\"pver\":\"2021-12-01T00:00:00.0000000Z\",\"rawDataStr\":\"{\\\"clusterID\\\":\\\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\\\",\\\"clusterName\\\":\\\"*************\\\",\\\"customerID\\\":\\\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\\\",\\\"details\\\":{\\\"container.id\\\":\\\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\\\",\\\"container.image.digest\\\":\\\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\\\",\\\"container.image.repository\\\":\\\"****viva.jfrog.io/********/tomcat\\\",\\\"container.image.tag\\\":\\\"9.0.73-***\\\",\\\"container.name\\\":\\\"tomcat-container\\\",\\\"evt.arg.exe\\\":\\\"curl\\\",\\\"evt.arg.filename\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.mode\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.name\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.category\\\":\\\"process\\\",\\\"evt.dir\\\":\\\"\\u003c\\\",\\\"evt.num\\\":\\\"8267962726\\\",\\\"evt.rawtime\\\":\\\"1733936999010100132\\\",\\\"evt.time\\\":\\\"17:09:59.010100132\\\",\\\"evt.type\\\":\\\"execve\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.id\\\":\\\"b777d8d9-441e-46ab-b34f-dac23b316636\\\",\\\"k8s.pod.labels\\\":\\\"app:pnt-test-***,controller-revision-hash:pnt-test-***-7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-***-0\\\",\\\"k8s.pod.name\\\":\\\"pnt-test-***-0\\\",\\\"proc.args\\\":\\\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.cmdline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.exe\\\":\\\"curl\\\",\\\"proc.exeline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.exepath\\\":\\\"/usr/bin/curl\\\",\\\"proc.name\\\":\\\"curl\\\",\\\"proc.pcmdline\\\":\\\"bash ./new-entrypoint.sh\\\",\\\"proc.pid\\\":\\\"1534304\\\",\\\"proc.pid.ts\\\":\\\"1733936999008820540\\\",\\\"proc.pname\\\":\\\"bash\\\",\\\"proc.ppid\\\":\\\"37520\\\",\\\"proc.ppid.ts\\\":\\\"1733427545111053154\\\",\\\"user.loginuid\\\":\\\"-1\\\",\\\"user.name\\\":\\\"root\\\"},\\\"hostname\\\":\\\"****.eu-south-1.compute.internal\\\",\\\"id\\\":\\\"2q50NZGekOuxLvx2iLCOQYHouki\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.labels\\\":{\\\"app\\\":\\\"pnt-test-***\\\",\\\"controller-revision-hash\\\":\\\"pnt-test-***-7668f9d96d\\\",\\\"statefulset.kubernetes.io/pod-name\\\":\\\"pnt-test-***-0\\\"},\\\"k8s.pod.name\\\":\\\"pnt-test-***-0\\\",\\\"mitigation\\\":\\\"log\\\",\\\"name\\\":\\\"(T1105)Launch Ingress Remote File Copy Tools in Container\\\",\\\"orchestration\\\":\\\"k8s\\\",\\\"policyID\\\":\\\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\\\",\\\"policyName\\\":\\\"LogOnlyPolicy\\\",\\\"productCode\\\":\\\"scs\\\",\\\"ruleID\\\":\\\"TM-00000049\\\",\\\"rulesets\\\":[{\\\"id\\\":\\\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\\\",\\\"name\\\":\\\"LogOnlyRuleset\\\"}],\\\"severity\\\":\\\"notice\\\",\\\"tags\\\":[\\\"mitre_command_and_control\\\",\\\"process\\\"],\\\"timestamp\\\":\\\"2024-12-11T17:09:59.010100Z\\\",\\\"type\\\":\\\"syscall\\\",\\\"version\\\":\\\"2021-12-01\\\"}\",\"rt_utc\":\"2024-12-11T17:09:59.0000000Z\",\"ruleIdStr\":\"TM-00000049\",\"ruleName\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"ruleSetId\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"ruleSetName\":\"LogOnlyRuleset\",\"searchDL\":\"DDL\",\"severity\":4,\"sourceType\":\"syscall\",\"uuid\":\"319bf2a2-02c9-4e24-b927-1b78725d7c94\"}",
"created": "2024-12-11T17:12:04.128Z",
"kind": "event",
"action": [
"log"
],
"id": "100119",
"category": [
"intrusion_detection"
],
"type": [
"info"
],
"dataset": "trend_micro_vision_one.detection"
}
},
"fields": {
"details.eventTimeDT": [
"2024-12-11T17:09:59+00:00"
],
"details.sourceType": [
"syscall"
],
"trend_micro_vision_one.detection.process.name": [
"curl"
],
"details.eventTime": [
1733936999000
],
"details.ruleName": [
"(T1105)Launch Ingress Remote File Copy Tools in Container"
],
"trend_micro_vision_one.detection.endpoint.hostname": [
"****.eu-south-1.compute.internal"
],
"details.k8sPodId": [
"b777d8d9-441e-46ab-b34f-dac23b316636"
],
"elastic_agent.version": [
"8.16.1"
],
"event.category": [
"intrusion_detection"
],
"process.name.text": [
"curl"
],
"host.name.text": [
"****.eu-south-1.compute.internal"
],
"process.pid": [
1534304
],
"details.clusterType": [
"k8s"
],
"trend_micro_vision_one.detection.search_data_lake": [
"DDL"
],
"details.groupId": [
"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR"
],
"details.ruleSetName": [
"LogOnlyRuleset"
],
"details.detectionType": [
"process"
],
"details.productCode": [
"scs"
],
"details.k8sNamespace": [
"sic-iam"
],
"details.endpointHostName": [
"****.eu-south-1.compute.internal"
],
"details.pver": [
"2021-12-01T00:00:00.0000000Z"
],
"agent.name.text": [
"Shuffle"
],
"details.searchDL": [
"DDL"
],
"trend_micro_vision_one.detection.event_id": [
"100119"
],
"agent.name": [
"Shuffle"
],
"host.name": [
"****.eu-south-1.compute.internal"
],
"trend_micro_vision_one.detection.policy.name": [
"LogOnlyPolicy"
],
"event.agent_id_status": [
"auth_metadata_missing"
],
"event.kind": [
"event"
],
"trend_micro_vision_one.detection.parent.cmd": [
"bash ./new-entrypoint.sh"
],
"event.original": [
"{\"act\":[\"log\"],\"clusterId\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"*************\",\"clusterType\":\"k8s\",\"containerId\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"containerImage\":\"****viva.jfrog.io/********/tomcat:9.0.73-***\",\"containerImageDigest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"containerName\":\"tomcat-container\",\"customTags\":[\"mitre_command_and_control\",\"process\"],\"detectionType\":\"process\",\"endpointHostName\":\"****.eu-south-1.compute.internal\",\"eventId\":\"100119\",\"eventName\":\"SECURITY_RISK_DETECTION\",\"eventSourceType\":3,\"eventTime\":1733936999000,\"eventTimeDT\":\"2024-12-11T17:09:59+00:00\",\"groupId\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"k8sNamespace\":\"sic-iam\",\"k8sPodId\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8sPodName\":\"pnt-test-***-0\",\"logKey\":\"e99c5cff-18f2-2e55-1b2b-ac77279777b5\",\"logReceivedTime\":\"1733937090989\",\"parentCmd\":\"bash ./new-entrypoint.sh\",\"parentName\":\"bash\",\"parentPid\":37520,\"pname\":\"Vision One Container Security\",\"policyId\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"processCmd\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\",\"processName\":\"curl\",\"processPid\":1534304,\"productCode\":\"scs\",\"pver\":\"2021-12-01T00:00:00.0000000Z\",\"rawDataStr\":\"{\\\"clusterID\\\":\\\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\\\",\\\"clusterName\\\":\\\"*************\\\",\\\"customerID\\\":\\\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\\\",\\\"details\\\":{\\\"container.id\\\":\\\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\\\",\\\"container.image.digest\\\":\\\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\\\",\\\"container.image.repository\\\":\\\"****viva.jfrog.io/********/tomcat\\\",\\\"container.image.tag\\\":\\\"9.0.73-***\\\",\\\"container.name\\\":\\\"tomcat-container\\\",\\\"evt.arg.exe\\\":\\\"curl\\\",\\\"evt.arg.filename\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.mode\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.name\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.category\\\":\\\"process\\\",\\\"evt.dir\\\":\\\"\\u003c\\\",\\\"evt.num\\\":\\\"8267962726\\\",\\\"evt.rawtime\\\":\\\"1733936999010100132\\\",\\\"evt.time\\\":\\\"17:09:59.010100132\\\",\\\"evt.type\\\":\\\"execve\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.id\\\":\\\"b777d8d9-441e-46ab-b34f-dac23b316636\\\",\\\"k8s.pod.labels\\\":\\\"app:pnt-test-***,controller-revision-hash:pnt-test-***-7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-***-0\\\",\\\"k8s.pod.name\\\":\\\"pnt-test-***-0\\\",\\\"proc.args\\\":\\\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.cmdline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.exe\\\":\\\"curl\\\",\\\"proc.exeline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.exepath\\\":\\\"/usr/bin/curl\\\",\\\"proc.name\\\":\\\"curl\\\",\\\"proc.pcmdline\\\":\\\"bash ./new-entrypoint.sh\\\",\\\"proc.pid\\\":\\\"1534304\\\",\\\"proc.pid.ts\\\":\\\"1733936999008820540\\\",\\\"proc.pname\\\":\\\"bash\\\",\\\"proc.ppid\\\":\\\"37520\\\",\\\"proc.ppid.ts\\\":\\\"1733427545111053154\\\",\\\"user.loginuid\\\":\\\"-1\\\",\\\"user.name\\\":\\\"root\\\"},\\\"hostname\\\":\\\"****.eu-south-1.compute.internal\\\",\\\"id\\\":\\\"2q50NZGekOuxLvx2iLCOQYHouki\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.labels\\\":{\\\"app\\\":\\\"pnt-test-***\\\",\\\"controller-revision-hash\\\":\\\"pnt-test-***-7668f9d96d\\\",\\\"statefulset.kubernetes.io/pod-name\\\":\\\"pnt-test-***-0\\\"},\\\"k8s.pod.name\\\":\\\"pnt-test-***-0\\\",\\\"mitigation\\\":\\\"log\\\",\\\"name\\\":\\\"(T1105)Launch Ingress Remote File Copy Tools in Container\\\",\\\"orchestration\\\":\\\"k8s\\\",\\\"policyID\\\":\\\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\\\",\\\"policyName\\\":\\\"LogOnlyPolicy\\\",\\\"productCode\\\":\\\"scs\\\",\\\"ruleID\\\":\\\"TM-00000049\\\",\\\"rulesets\\\":[{\\\"id\\\":\\\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\\\",\\\"name\\\":\\\"LogOnlyRuleset\\\"}],\\\"severity\\\":\\\"notice\\\",\\\"tags\\\":[\\\"mitre_command_and_control\\\",\\\"process\\\"],\\\"timestamp\\\":\\\"2024-12-11T17:09:59.010100Z\\\",\\\"type\\\":\\\"syscall\\\",\\\"version\\\":\\\"2021-12-01\\\"}\",\"rt_utc\":\"2024-12-11T17:09:59.0000000Z\",\"ruleIdStr\":\"TM-00000049\",\"ruleName\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"ruleSetId\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"ruleSetName\":\"LogOnlyRuleset\",\"searchDL\":\"DDL\",\"severity\":4,\"sourceType\":\"syscall\",\"uuid\":\"319bf2a2-02c9-4e24-b927-1b78725d7c94\"}"
],
"trend_micro_vision_one.detection.action": [
"log"
],
"details.logReceivedTime": [
"1733937090989"
],
"trend_micro_vision_one.detection.product.name": [
"Vision One Container Security"
],
"details.uuid": [
"319bf2a2-02c9-4e24-b927-1b78725d7c94"
],
"input.type": [
"httpjson"
],
"data_stream.type": [
"logs"
],
"details.act": [
"log"
],
"details.processName": [
"curl"
],
"tags": [
"preserve_original_event",
"preserve_duplicate_custom_fields",
"forwarded",
"trend_micro_vision_one-detection"
],
"details.logKey": [
"e99c5cff-18f2-2e55-1b2b-ac77279777b5"
],
"process.name": [
"curl"
],
"details.containerImage": [
"****viva.jfrog.io/********/tomcat:9.0.73-***"
],
"trend_micro_vision_one.detection.detection_type": [
"process"
],
"agent.id": [
"5f4edad9-8329-460f-adbf-c4789daa91b3"
],
"details.containerImageDigest": [
"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563"
],
"details.parentPid": [
37520
],
"ecs.version": [
"8.11.0"
],
"event.created": [
"2024-12-11T17:12:04.128Z"
],
"trend_micro_vision_one.detection.event_name": [
"SECURITY_RISK_DETECTION"
],
"agent.version": [
"8.16.1"
],
"related.hosts": [
"****.eu-south-1.compute.internal"
],
"details.eventSourceType": [
3
],
"process.command_line.text": [
"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
],
"details.containerId": [
"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe"
],
"details.processPid": [
1534304
],
"details.rt_utc": [
"2024-12-11T17:09:59.0000000Z"
],
"details.ruleIdStr": [
"TM-00000049"
],
"details.severity": [
4
],
"details.parentName": [
"bash"
],
"trend_micro_vision_one.detection.process.pid": [
1534304
],
"details.ruleSetId": [
"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th"
],
"details.eventName": [
"SECURITY_RISK_DETECTION"
],
"trend_micro_vision_one.detection.product.code": [
"scs"
],
"details.customTags": [
"mitre_command_and_control",
"process"
],
"details.policyName": [
"LogOnlyPolicy"
],
"agent.type": [
"filebeat"
],
"event.module": [
"trend_micro_vision_one"
],
"trend_micro_vision_one.detection.policy.logkey": [
"e99c5cff-18f2-2e55-1b2b-ac77279777b5"
],
"trend_micro_vision_one.detection.uuid": [
"319bf2a2-02c9-4e24-b927-1b78725d7c94"
],
"details.clusterName": [
"*************"
],
"elastic_agent.snapshot": [
false
],
"details.pname": [
"Vision One Container Security"
],
"details.k8sPodName": [
"pnt-test-***-0"
],
"details.parentCmd": [
"bash ./new-entrypoint.sh"
],
"elastic_agent.id": [
"5f4edad9-8329-460f-adbf-c4789daa91b3"
],
"data_stream.namespace": [
"default"
],
"details.rawDataStr": [
"{\"clusterID\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"*************\",\"customerID\":\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\",\"details\":{\"container.id\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"container.image.digest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"container.image.repository\":\"****viva.jfrog.io/********/tomcat\",\"container.image.tag\":\"9.0.73-***\",\"container.name\":\"tomcat-container\",\"evt.arg.exe\":\"curl\",\"evt.arg.filename\":\"<NA>\",\"evt.arg.mode\":\"<NA>\",\"evt.arg.name\":\"<NA>\",\"evt.category\":\"process\",\"evt.dir\":\"<\",\"evt.num\":\"8267962726\",\"evt.rawtime\":\"1733936999010100132\",\"evt.time\":\"17:09:59.010100132\",\"evt.type\":\"execve\",\"k8s.ns.name\":\"sic-iam\",\"k8s.pod.id\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8s.pod.labels\":\"app:pnt-test-***,controller-revision-hash:pnt-test-***-7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-***-0\",\"k8s.pod.name\":\"pnt-test-***-0\",\"proc.args\":\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\",\"proc.cmdline\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\",\"proc.exe\":\"curl\",\"proc.exeline\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\",\"proc.exepath\":\"/usr/bin/curl\",\"proc.name\":\"curl\",\"proc.pcmdline\":\"bash ./new-entrypoint.sh\",\"proc.pid\":\"1534304\",\"proc.pid.ts\":\"1733936999008820540\",\"proc.pname\":\"bash\",\"proc.ppid\":\"37520\",\"proc.ppid.ts\":\"1733427545111053154\",\"user.loginuid\":\"-1\",\"user.name\":\"root\"},\"hostname\":\"****.eu-south-1.compute.internal\",\"id\":\"2q50NZGekOuxLvx2iLCOQYHouki\",\"k8s.ns.name\":\"sic-iam\",\"k8s.pod.labels\":{\"app\":\"pnt-test-***\",\"controller-revision-hash\":\"pnt-test-***-7668f9d96d\",\"statefulset.kubernetes.io/pod-name\":\"pnt-test-***-0\"},\"k8s.pod.name\":\"pnt-test-***-0\",\"mitigation\":\"log\",\"name\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"orchestration\":\"k8s\",\"policyID\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"productCode\":\"scs\",\"ruleID\":\"TM-00000049\",\"rulesets\":[{\"id\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"name\":\"LogOnlyRuleset\"}],\"severity\":\"notice\",\"tags\":[\"mitre_command_and_control\",\"process\"],\"timestamp\":\"2024-12-11T17:09:59.010100Z\",\"type\":\"syscall\",\"version\":\"2021-12-01\"}"
],
"trend_micro_vision_one.detection.product.version": [
"2021-12-01T00:00:00.0000000Z"
],
"details.containerName": [
"tomcat-container"
],
"details.clusterId": [
"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR"
],
"details.policyId": [
"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq"
],
"event.action": [
"log"
],
"event.ingested": [
"2024-12-11T17:12:14.000Z"
],
"@timestamp": [
"2024-12-11T17:09:59.000Z"
],
"data_stream.dataset": [
"trend_micro_vision_one.detection"
],
"event.type": [
"info"
],
"process.command_line": [
"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
],
"trend_micro_vision_one.detection.process.cmd": [
"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
],
"agent.ephemeral_id": [
"875a1d57-8cb8-4fed-802f-4e4deb2bef60"
],
"trend_micro_vision_one.detection.event_time_dt": [
"2024-12-11T17:09:59.000Z"
],
"details.eventId": [
"100119"
],
"event.id": [
"100119"
],
"details.processCmd": [
"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***"
],
"trend_micro_vision_one.detection.rt_utc": [
"2024-12-11T17:09:59.000Z"
],
"event.dataset": [
"trend_micro_vision_one.detection"
]
}
}
Also, here the screenshots of the ingestion policies:
It would be better if you could post the pipeline text instead of images...
I have some other things to do if I get a chance I will look in the meantime you should try to post the document... Which will call the pipelines and see what happens
add_field pipeline:
[
{
"json": {
"field": "event.original",
"target_field": "details",
"if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
}
},
{
"json": {
"field": "details.rawDataStr",
"target_field": "rawdetails",
"if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
}
}
]
pipeline logs-trend_micro_vision_one.detection@custom pipeline:
Processors
[
{
"pipeline": {
"name": "add_field"
}
}
]
Failure processors
[
{
"set": {
"field": "pipelineerror",
"value": "details.rawDataStr"
}
}
]
Is there a other
You can run in Kibana Dev Tool
GET _ingest/pipeline/logs-trend_micro_vision_one.detection@custom
And that will show the whole pipeline.
Ok I will take a look later today if I can
And you can try to POST a document to the data stream.. You will probably need to change the time so it does not think it is the same document
Ok I have found it..... wow that was pretty buried...
The problem is that the data inside the details.rawDataStr is invalid JSON... and thus was creating invalid JSON after expanding and causing a mapping issue and dropping the events.
Parts of it the bad JSON looks like this...
"details": {
....
"proc.pid": "1534304", << CAN NOT HAVE THIS AND
"proc.pid.ts": "1733936999008820540", << THIS IN JSON THEY COLLIDE
"proc.pname": "bash",
"proc.ppid": "37520",
"proc.ppid.ts": "1733427545111053154",
"user.loginuid": "-1",
"user.name": "root"
So what I did is this... inside the details.rawDataStr I replaced all those dots . with underscores _
PUT _ingest/pipeline/logs-trend_micro_vision_one.detection@custom
{
"processors": [
{
"json": {
"field": "event.original",
"target_field": "details",
"if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
}
},
{
"gsub": {
"field": "details.rawDataStr",
"pattern": """\.""",
"replacement": "_"
}
},
{
"json": {
"field": "details.rawDataStr",
"target_field": "rawdetails",
"if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
}
}
]
}
Then I think it works
POST _ingest/pipeline/logs-trend_micro_vision_one.detection-1.22.0/_simulate
{
"docs": [
{
"_source" :
{
"tags": [
"preserve_original_event",
"preserve_duplicate_custom_fields",
"forwarded",
"trend_micro_vision_one-detection"
],
"@timestamp": "2024-12-11T17:09:59.000Z",
"message": "{\"act\":[\"log\"],\"clusterId\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"clusterName\":\"*************\",\"clusterType\":\"k8s\",\"containerId\":\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\",\"containerImage\":\"***********.jfrog.io/***********/tomcat:9.0.73-***\",\"containerImageDigest\":\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\",\"containerName\":\"tomcat-container\",\"customTags\":[\"mitre_command_and_control\",\"process\"],\"detectionType\":\"process\",\"endpointHostName\":\"****.eu-south-1.compute.internal\",\"eventId\":\"100119\",\"eventName\":\"SECURITY_RISK_DETECTION\",\"eventSourceType\":3,\"eventTime\":1733936999000,\"eventTimeDT\":\"2024-12-11T17:09:59+00:00\",\"groupId\":\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\",\"k8sNamespace\":\"sic-iam\",\"k8sPodId\":\"b777d8d9-441e-46ab-b34f-dac23b316636\",\"k8sPodName\":\"pnt-test-***-0\",\"logKey\":\"e99c5cff-18f2-2e55-1b2b-ac77279777b5\",\"logReceivedTime\":\"1733937090989\",\"parentCmd\":\"bash ./new-entrypoint.sh\",\"parentName\":\"bash\",\"parentPid\":37520,\"pname\":\"Vision One Container Security\",\"policyId\":\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\",\"policyName\":\"LogOnlyPolicy\",\"processCmd\":\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\",\"processName\":\"curl\",\"processPid\":1534304,\"productCode\":\"scs\",\"pver\":\"2021-12-01T00:00:00.0000000Z\",\"rawDataStr\":\"{\\\"clusterID\\\":\\\"*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR\\\",\\\"clusterName\\\":\\\"*************\\\",\\\"customerID\\\":\\\"7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7\\\",\\\"details\\\":{\\\"container.id\\\":\\\"7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe\\\",\\\"container.image.digest\\\":\\\"sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563\\\",\\\"container.image.repository\\\":\\\"****viva.jfrog.io/********/tomcat\\\",\\\"container.image.tag\\\":\\\"9.0.73-***\\\",\\\"container.name\\\":\\\"tomcat-container\\\",\\\"evt.arg.exe\\\":\\\"curl\\\",\\\"evt.arg.filename\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.mode\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.arg.name\\\":\\\"\\u003cNA\\u003e\\\",\\\"evt.category\\\":\\\"process\\\",\\\"evt.dir\\\":\\\"\\u003c\\\",\\\"evt.num\\\":\\\"8267962726\\\",\\\"evt.rawtime\\\":\\\"1733936999010100132\\\",\\\"evt.time\\\":\\\"17:09:59.010100132\\\",\\\"evt.type\\\":\\\"execve\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.id\\\":\\\"b777d8d9-441e-46ab-b34f-dac23b316636\\\",\\\"k8s.pod.labels\\\":\\\"app:pnt-test-***,controller-revision-hash:pnt-test-***-7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-***-0\\\",\\\"k8s.pod.name\\\":\\\"pnt-test-***-0\\\",\\\"proc.args\\\":\\\"--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.cmdline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.exe\\\":\\\"curl\\\",\\\"proc.exeline\\\":\\\"curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***\\\",\\\"proc.exepath\\\":\\\"/usr/bin/curl\\\",\\\"proc.name\\\":\\\"curl\\\",\\\"proc.pcmdline\\\":\\\"bash ./new-entrypoint.sh\\\",\\\"proc.pid\\\":\\\"1534304\\\",\\\"proc.pid.ts\\\":\\\"1733936999008820540\\\",\\\"proc.pname\\\":\\\"bash\\\",\\\"proc.ppid\\\":\\\"37520\\\",\\\"proc.ppid.ts\\\":\\\"1733427545111053154\\\",\\\"user.loginuid\\\":\\\"-1\\\",\\\"user.name\\\":\\\"root\\\"},\\\"hostname\\\":\\\"****.eu-south-1.compute.internal\\\",\\\"id\\\":\\\"2q50NZGekOuxLvx2iLCOQYHouki\\\",\\\"k8s.ns.name\\\":\\\"sic-iam\\\",\\\"k8s.pod.labels\\\":{\\\"app\\\":\\\"pnt-test-***\\\",\\\"controller-revision-hash\\\":\\\"pnt-test-***-7668f9d96d\\\",\\\"statefulset.kubernetes.io/pod-name\\\":\\\"pnt-test-***-0\\\"},\\\"k8s.pod.name\\\":\\\"pnt-test-***-0\\\",\\\"mitigation\\\":\\\"log\\\",\\\"name\\\":\\\"(T1105)Launch Ingress Remote File Copy Tools in Container\\\",\\\"orchestration\\\":\\\"k8s\\\",\\\"policyID\\\":\\\"LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq\\\",\\\"policyName\\\":\\\"LogOnlyPolicy\\\",\\\"productCode\\\":\\\"scs\\\",\\\"ruleID\\\":\\\"TM-00000049\\\",\\\"rulesets\\\":[{\\\"id\\\":\\\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\\\",\\\"name\\\":\\\"LogOnlyRuleset\\\"}],\\\"severity\\\":\\\"notice\\\",\\\"tags\\\":[\\\"mitre_command_and_control\\\",\\\"process\\\"],\\\"timestamp\\\":\\\"2024-12-11T17:09:59.010100Z\\\",\\\"type\\\":\\\"syscall\\\",\\\"version\\\":\\\"2021-12-01\\\"}\",\"rt_utc\":\"2024-12-11T17:09:59.0000000Z\",\"ruleIdStr\":\"TM-00000049\",\"ruleName\":\"(T1105)Launch Ingress Remote File Copy Tools in Container\",\"ruleSetId\":\"LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th\",\"ruleSetName\":\"LogOnlyRuleset\",\"searchDL\":\"DDL\",\"severity\":4,\"sourceType\":\"syscall\",\"uuid\":\"319bf2a2-02c9-4e24-b927-1b78725d7c94\"}"
}
}
]
}
Results in .....
"details": {
"proc_exepath": "/usr/bin/curl",
"proc_args": "--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
"container_image_tag": "9_0_73-***",
"evt_arg_name": "<NA>",
"user_name": "root",
"proc_ppid_ts": "1733427545111053154", << OK NOW
"proc_ppid": "37520", << OK NOW
And when I did a POST of that message into the actual data stream then it worked...
IF you learn to use _simulate you can see that but even for me not obvious at first....
Give it a try and let me know
Hi @stephenb,
I would say it works. Now I would like to simplify everything. Instead of applying the JSON processor twice, I would like to simply extract the field I am interested in from details.rawDataStr. In particular, I would like to extract the value of the field details.k8s.pod.labels from details.rawDataStr and use it to enhance a new target field. How can I do this?
hehehe Why did you not say that in the first place 
-
First, there are several fields and details within
raw details, the expansion ofdetails.rawDataStr. Which one(s) do you want? I posted the whole raw details below. -
The only way to extract the field(s) will be to use a GROK processor, which is Regex under the covers ... which may or may not simplify... or be more efficient... unwrapping the JSON picking out the strings and then cleaning up maybe a couple more lines of code but may be much cleaner / more efficient...
THis is the whole raw details which ones exactly do you want A,B,C,D something other?
"rawdetails": {
"severity": "notice",
"mitigation": "log",
"rulesets": [
{
"name": "LogOnlyRuleset",
"id": "LogOnlyRuleset-2ZQuYrP1rWUXTWC8ln6HQRni7th"
}
],
"orchestration": "k8s",
"k8s.pod.labels": { A) <<< ONE OF THESE?
"app": "pnt-test-***",
"statefulset.kubernetes.io/pod-name": "pnt-test-***-0",
"controller-revision-hash": "pnt-test-***-7668f9d96d"
},
"policyName": "LogOnlyPolicy",
"clusterID": "*************-2okw77rZ7bD1Qk0q9bLfTRAk2VR",
"type": "syscall",
"k8s.ns.name": "sic-iam",
"version": "2021-12-01",
"tags": [
"mitre_command_and_control",
"process"
],
"hostname": "****.eu-south-1.compute.internal",
"productCode": "scs",
"k8s.pod.name": "pnt-test-***-0", <<< B) THIS ONE?
"policyID": "LogOnlyPolicy-2ZQuYzNIRyA8rb5AQo7wyR4ncvq",
"clusterName": "*************",
"customerID": "7c49c1e2-f12c-4f97-9a67-fb2f44ea87a7",
"name": "(T1105)Launch Ingress Remote File Copy Tools in Container",
"details": {
"proc.pname": "bash",
"container.image.tag": "9.0.73-***",
"evt.type": "execve",
"proc.cmdline": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
"proc.args": "--head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
"proc.ppid": "37520",
"proc.exe": "curl",
"user.name": "root",
"evt.arg.mode": "<NA>",
"k8s.ns.name": "sic-iam",
"container.id": "7ad0e152f9b6dd132fdc0d6ec3f4450499dfb476dbaad1dbd492d8f07eae1efe",
"proc.pid": "1534304",
"evt.num": "8267962726",
"container.name": "tomcat-container",
"k8s.pod.name": "pnt-test-***-0", <<< C) THiS ONE???
"proc.name": "curl",
"proc.pcmdline": "bash ./new-entrypoint.sh",
"evt.time": "17:09:59.010100132",
"proc.exeline": "curl --head --location --connect-timeout 5 --max-time 5 --write-out %{http_code} --silent --output /dev/null http://localhost:8080/***",
"evt.arg.filename": "<NA>",
### D) THIS WHOLE STRING
"k8s.pod.labels": "app:pnt-test-***,controller-revision-hash:pnt-test-***-7668f9d96d,statefulset.kubernetes.io/pod-name:pnt-test-***-0",
"evt.arg.name": "<NA>",
"evt.dir": "<",
"proc.exepath": "/usr/bin/curl",
"proc.pid.ts": "1733936999008820540",
"user.loginuid": "-1",
"container.image.repository": "****viva.jfrog.io/********/tomcat",
"k8s.pod.id": "b777d8d9-441e-46ab-b34f-dac23b316636",
"evt.rawtime": "1733936999010100132",
"container.image.digest": "sha256:f9859cbc865035a55922648dbe394a34b36afc238399e561760d3dad86516563",
"proc.ppid.ts": "1733427545111053154",
"evt.arg.exe": "curl",
"evt.category": "process"
},
"id": "2q50NZGekOuxLvx2iLCOQYHouki",
"ruleID": "TM-00000049",
"timestamp": "2024-12-11T17:09:59.010100Z"
}
Here is a pipeline that works... without the GROK... I think this is pretty straightforward and efficient...
PUT _ingest/pipeline/logs-trend_micro_vision_one.detection@custom
{
"processors": [
{
"json": {
"field": "event.original",
"target_field": "details",
"if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
}
},
{
"gsub": {
"field": "details.rawDataStr",
"pattern": """\.""",
"replacement": "_"
}
},
{
"json": {
"field": "details.rawDataStr",
"target_field": "rawdetails",
"if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
}
},
{
"rename": {
"field": "rawdetails.details.k8s_pod_labels",
"target_field": "k8s.pod.labels"
}
},
{
"remove": {
"field": ["details", "rawdetails"]
}
}
]
}
After some playing, try this... I am surprised it works...
This seems easy after the fact but it was not
You might need to play with it; perhaps you should look at the grok documentation.
PUT _ingest/pipeline/logs-trend_micro_vision_one.detection@custom
{
"processors": [
{
"json": {
"field": "event.original",
"target_field": "details",
"if": "ctx?.trend_micro_vision_one.detection.product.name == 'Vision One Container Security'"
}
},
{
"grok": {
"field": "details.rawDataStr",
"patterns": [
"""%{DATA}"k8s.pod.labels":"%{DATA:k8s.pod.labels}"%{DATA}"""
]
}
}
]
}
Hi @stephenb ,
this way the '.' is replaced with '_' not only in the field name but also in the value.
For example,
10:03:12.496882712 > 10:03:12_496882712
and this is a problem.