Extracting and mapping fields from event.original in Trend Micro Vision One Integration

Elastic Stack Elastic Agent
1

Hi everyone,

I’m trying to properly configure the Vision One integration in Kibana to collect and manage logs. Here’s what I’ve done so far:

  1. I’ve set up the Vision One integration in Kibana.
  2. I’ve configured the policy on the agent installed on the same machine, which collects logs via API.

The setup works partially—I’m receiving various logs from Vision One. However, I’ve noticed an issue with some important data found in the event.original field. This field contains a full JSON object that isn’t mapped, but it has information that would be very useful for me.

My goal is to extract specific information from event.original and map it to other fields so I can manage it better. I’m considering using an ingest pipeline to do this.

Additionally, I’m having trouble handling logs related to container security with this setup.

Has anyone dealt with a similar situation? Could you share advice or examples of how to set up an ingest pipeline to extract and map fields from event.original?

Any help or tips would be greatly appreciated!

Thanks a lot!

2

Can you provide any example? Probably this kind of data is not being parsed by the built-in integration, so you would need to build your own parsers by creating a custom ingest pipeline.

1 Like
3

Hi @cyberm Welcome to the community.

What version Elastic and Kibana?
What are you using to collect the Elastic Agent integrations?
Are you speaking of the Trend Micro Vision One?
Which Logs do you want to add the extra parsing to?

Yes if you want to work with event.original make sure you set Preserve original event looks like you are doing that.

So I would do a quick read of this (you do need to understand it all but good background)

Then, here is how you add a custom pipeline to the end of the Vision One Processing

Then, in the custom pipeline, you will probably want to use the JSON processor which should extract the fields... then you can rename drop, move the fields etc if you like.

Pro tip: put the expanded JSON field under a field like details so you can see it first ... if you write them to the root of the document, you may overwrite key /processed fields.

1 Like
4

Hi everyone,

This image shows the part of the document we are focusing on, extracted from the index populated through the integration with Trend Micro Vision One:

image001
image001517×365 18.1 KB

The complete document looks like this and includes the event.original attribute, which is a complete JSON:

{
    "_index": "\"***\"",
    "_id": "\"***\"",
    "_version": 1,
    "_score": 0,
    "_source": {
        "agent": {
            "name": "\"***\"",
            "id": "\"***\"",
            "ephemeral_id": "\"***\"",
            "type": "\"***\"",
            "version": "\"***\""
        },
        "trend_micro_vision_one": {
            "alert": {
                "impact_scope": {
                    "email_address_count": 0,
                    "entities": [
                        {
                            "provenance": [
                                "\"***\""
                            ],
                            "managementScopeGroupId": "\"***\"",
                            "related_indicator_id": [
                                1,
                                2,
                                3,
                                4
                            ],
                            "id": "\"***\"",
                            "type": "\"***\"",
                            "value": {
                                "name": "\"***\"",
                                "guid": "\"***\"",
                                "ips": [
                                    "\"***\""
                                ]
                            }
                        },
                        {
                            "provenance": [
                                "\"***\""
                            ],
                            "managementScopeGroupId": "\"***\"",
                            "related_indicator_id": [
                                2,
                                4,
                                5,
                                6,
                                7,
                                8,
                                9,
                                10,
                                11,
                                12,
                                13,
                                14,
                                15,
                                16
                            ],
                            "id": "\"***\"",
                            "type": "\"***\"",
                            "value": {
                                "account_value": "\"***\""
                            }
                        }
                    ],
                    "desktop_count": 1,
                    "server_count": 0,
                    "account_count": 0
                },
                "description": "\"***\"",
                "indicators": [
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\"",
                            "\"***\""
                        ],
                        "id": 1,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\"",
                            "\"***\""
                        ],
                        "id": 2,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\"",
                            "\"***\""
                        ],
                        "id": 3,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\"",
                            "\"***\""
                        ],
                        "id": 4,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 5,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 6,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 7,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 8,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 9,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 10,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 11,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 12,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 13,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 14,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 15,
                        "type": "\"***\"",
                        "value": "\"***\""
                    },
                    {
                        "provenance": [
                            "\"***\""
                        ],
                        "field": "\"***\"",
                        "filter_id": [
                            "\"***\""
                        ],
                        "related_entities": [
                            "\"***\""
                        ],
                        "id": 16,
                        "type": "\"***\"",
                        "value": "\"***\""
                    }
                ],
                "schema_version": "\"***\"",
                "investigation_status": "\"***\"",
                "alert_provider": "\"***\"",
                "model": "\"***\"",
                "matched_rule": [
                    {
                        "filter": [
                            {
                                "date": "\"***\"",
                                "name": "\"***\"",
                                "mitre_technique_id": [
                                    "\"***\""
                                ],
                                "id": "\"***\"",
                                "events": [
                                    {
                                        "date": "\"***\"",
                                        "type": "\"***\"",
                                        "uuid": "\"***\""
                                    }
                                ]
                            },
                            {
                                "date": "\"***\"",
                                "name": "\"***\"",
                                "mitre_technique_id": [
                                    "\"***\""
                                ],
                                "id": "\"***\"",
                                "events": [
                                    {
                                        "date": "\"***\"",
                                        "type": "\"***\"",
                                        "uuid": "\"***\""
                                    }
                                ]
                            }
                        ],
                        "name": "\"***\"",
                        "id": "\"***\""
                    }
                ],
                "created_date": "\"***\"",
                "workbench_link": "\"***\""
            }
        },
        "log": {
            "level": "\"***\""
        },
        "elastic_agent": {
            "id": "\"***\"",
            "version": "\"***\"",
            "snapshot": false
        },
        "url": {
            "path": "\"***\"",
            "fragment": "\"***\"",
            "extension": "\"***\"",
            "original": "\"***\"",
            "scheme": "\"***\"",
            "domain": "\"***\""
        },
        "tags": [
            "\"***\"",
            "\"***\"",
            "\"***\""
        ],
        "cloud": {
            "availability_zone": "\"***\"",
            "instance": {
                "name": "\"***\"",
                "id": "\"***\""
            },
            "provider": "\"***\"",
            "service": {
                "name": "\"***\""
            },
            "machine": {
                "type": "\"***\""
            }
        },
        "input": {
            "type": "\"***\""
        },
        "@timestamp": "\"***\"",
        "ecs": {
            "version": "\"***\""
        },
        "related": {
            "ip": [
                "\"***\""
            ]
        },
        "data_stream": {
            "namespace": "\"***\"",
            "type": "\"***\"",
            "dataset": "\"***\""
        },
        "event": {
            "severity": 30,
            "agent_id_status": "\"***\"",
            "ingested": "\"***\"",
            "original": "\"eventOriginalinJSON\"",
            "created": "\"***\"",
            "kind": "\"***\"",
            "id": "\"***\"",
            "category": [
                "\"***\""
            ],
            "type": [
                "\"***\""
            ],
            "dataset": "\"***\""
        }
    },
    "fields": {
        "elastic_agent.version": [
            "\"***\""
        ],
        "event.category": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.entities.value.account_value": [
            "\"***\""
        ],
        "url.original.text": [
            "\"***\""
        ],
        "cloud.availability_zone": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.schema_version": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.description": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.entities.value.guid": [
            "\"***\""
        ],
        "log.level": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.entities.value.name": [
            "\"***\""
        ],
        "agent.name": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.indicators.id": [
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\""
        ],
        "event.agent_id_status": [
            "\"***\""
        ],
        "event.kind": [
            "\"***\""
        ],
        "url.fragment": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.email_address_count": [
            0
        ],
        "event.severity": [
            30
        ],
        "trend_micro_vision_one.alert.matched_rule.filter.id": [
            "\"***\"",
            "\"***\""
        ],
        "trend_micro_vision_one.alert.matched_rule.filter.mitre_technique_id": [
            "\"***\"",
            "\"***\""
        ],
        "event.original": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.entities.provenance": [
            "\"***\"",
            "\"***\""
        ],
        "trend_micro_vision_one.alert.matched_rule.id": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.matched_rule.name": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.model": [
            "\"***\""
        ],
        "url.extension": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.entities.managementScopeGroupId": [
            "\"***\"",
            "\"***\""
        ],
        "input.type": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.alert_provider": [
            "\"***\""
        ],
        "data_stream.type": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.created_date": [
            "\"***\""
        ],
        "tags": [
            "\"***\"",
            "\"***\"",
            "\"***\""
        ],
        "cloud.machine.type": [
            "\"***\""
        ],
        "cloud.provider": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.investigation_status": [
            "\"***\""
        ],
        "url.path": [
            "\"***\""
        ],
        "agent.id": [
            "\"***\""
        ],
        "cloud.service.name": [
            "\"***\""
        ],
        "ecs.version": [
            "\"***\""
        ],
        "event.created": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.matched_rule.filter.name": [
            "\"***\"",
            "\"***\""
        ],
        "agent.version": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.desktop_count": [
            1
        ],
        "trend_micro_vision_one.alert.indicators.field": [
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.entities.value.ips": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.indicators.related_entities": [
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.account_count": [
            0
        ],
        "url.scheme": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.indicators.provenance": [
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\""
        ],
        "cloud.instance.id": [
            "\"***\""
        ],
        "agent.type": [
            "\"***\""
        ],
        "event.module": [
            "\"***\""
        ],
        "related.ip": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.indicators.filter_id": [
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\""
        ],
        "elastic_agent.snapshot": [
            false
        ],
        "trend_micro_vision_one.alert.impact_scope.server_count": [
            0
        ],
        "trend_micro_vision_one.alert.matched_rule.filter.events.uuid": [
            "\"***\"",
            "\"***\""
        ],
        "trend_micro_vision_one.alert.indicators.value": [
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.entities.related_indicator_id": [
            1,
            2,
            3,
            4,
            2,
            4,
            5,
            6,
            7,
            8,
            9,
            10,
            11,
            12,
            13,
            14,
            15,
            16
        ],
        "trend_micro_vision_one.alert.matched_rule.filter.date": [
            "\"***\"",
            "\"***\""
        ],
        "trend_micro_vision_one.alert.matched_rule.filter.events.date": [
            "\"***\"",
            "\"***\""
        ],
        "elastic_agent.id": [
            "\"***\""
        ],
        "data_stream.namespace": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.matched_rule.filter.events.type": [
            "\"***\"",
            "\"***\""
        ],
        "trend_micro_vision_one.alert.indicators.type": [
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\"",
            "\"***\""
        ],
        "trend_micro_vision_one.alert.workbench_link": [
            "\"***\""
        ],
        "event.ingested": [
            "\"***\""
        ],
        "url.original": [
            "\"***\""
        ],
        "@timestamp": [
            "\"***\""
        ],
        "data_stream.dataset": [
            "\"***\""
        ],
        "event.type": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.entities.id": [
            "\"***\"",
            "\"***\""
        ],
        "url.domain": [
            "\"***\""
        ],
        "agent.ephemeral_id": [
            "\"***\""
        ],
        "trend_micro_vision_one.alert.impact_scope.entities.type": [
            "\"***\"",
            "\"***\""
        ],
        "event.id": [
            "\"***\""
        ],
        "event.dataset": [
            "\"***\""
        ],
        "cloud.instance.name": [
            "\"***\""
        ]
    }
}

Within this document, here is the specific content of the event.original field:

{
    "alertProvider": "\"***\"",
    "createdDateTime": "\"***\"",
    "description": "\"***\"",
    "id": "\"***\"",
    "impactScope": {
        "accountCount": 0,
        "cloudIdentityCount": 0,
        "containerCount": 1,
        "desktopCount": 1,
        "emailAddressCount": 0,
        "entities": [
            {
                "entityId": "\"***\"",
                "entityType": "\"***\"",
                "entityValue": {
                    "guid": "\"***\"",
                    "ips": [
                        "\"***\""
                    ],
                    "name": "\"***\""
                },
                "managementScopeGroupId": "\"***\"",
                "provenance": [
                    "\"***\""
                ],
                "relatedEntities": [],
                "relatedIndicatorIds": [
                    1,
                    2,
                    3,
                    4
                ]
            },
            {
                "entityId": "\"***\"",
                "entityType": "\"***\"",
                "entityValue": "\"***\"",
                "managementScopeGroupId": "\"***\"",
                "provenance": [
                    "\"***\""
                ],
                "relatedEntities": [],
                "relatedIndicatorIds": [
                    2,
                    4,
                    5,
                    6,
                    7,
                    8,
                    9,
                    10,
                    11,
                    12,
                    13,
                    14,
                    15,
                    16
                ]
            }
        ],
        "serverCount": 0
    },
    "indicators": [
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 1,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\"",
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 2,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\"",
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 3,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\"",
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 4,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\"",
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 5,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 6,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 7,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 8,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 9,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 10,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 11,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 12,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 13,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 14,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"***\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 15,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"***\""
        },
        {
            "field": "\"valueTobeExtract\"",
            "filterIds": [
                "\"***\""
            ],
            "id": 16,
            "provenance": [
                "\"***\""
            ],
            "relatedEntities": [
                "\"***\""
            ],
            "type": "\"***\"",
            "value": "\"valueTobeExtract\""
        }
    ],
    "investigationResult": "\"***\"",
    "investigationStatus": "\"***\"",
    "matchedRules": [
        {
            "id": "\"***\"",
            "matchedFilters": [
                {
                    "id": "\"***\"",
                    "matchedDateTime": "\"***\"",
                    "matchedEvents": [
                        {
                            "matchedDateTime": "\"***\"",
                            "type": "\"***\"",
                            "uuid": "\"***\""
                        }
                    ],
                    "mitreTechniqueIds": [
                        "\"***\""
                    ],
                    "name": "\"***\""
                },
                {
                    "id": "\"***\"",
                    "matchedDateTime": "\"***\"",
                    "matchedEvents": [
                        {
                            "matchedDateTime": "\"***\"",
                            "type": "\"***\"",
                            "uuid": "\"***\""
                        }
                    ],
                    "mitreTechniqueIds": [
                        "\"***\""
                    ],
                    "name": "\"***\""
                }
            ],
            "name": "\"***\""
        }
    ],
    "model": "\"***\"",
    "modelId": "\"***\"",
    "modelType": "\"***\"",
    "ownerIds": [],
    "schemaVersion": "\"***\"",
    "score": 30,
    "severity": "\"***\"",
    "status": "\"***\"",
    "updatedDateTime": "\"***\"",
    "workbenchLink": "\"***\""
}

My goal is to extract the values associated with the key valueToBeExtract from this nested JSON (event.original).

I also noticed that when I perform a search using KQL on the attribute trend_micro_vision_one.alert.indicators.value, it only evaluates the first value in the JSON object, not the subsequent ones. I’m not sure why this happens.

Thanks in advance for any suggestions or help!

5

I’m using Elasticsearch and Kibana version 8.16.0. To collect Elastic Agent integrations, I have configured the Trend Micro Vision One integration. Specifically, there is an agent that forwards the requested logs via API and writes them to an Elasticsearch index.

Regarding the logs I want to parse, I need to extract and parse the values of field and value from the following object:

{
    "field": "\"valueTobeExtract\"",
    "filterIds": ["\"***\""],
    "id": 16,
    "provenance": ["\"***\""],
    "relatedEntities": ["\"***\""],
    "type": "\"***\"",
    "value": "\"valueTobeExtract\""
}

I can also confirm that I enabled the option to preserve the original event (Preserve original event) during the integration setup. I hope this clarifies my setup and requirements!

6

You redacted so many thing that is pretty hard to compare the parsed event and the original event.

But from what I was able to understand your message is already being parsed.

What you want is inside the field trend_micro_vision_one.indicators, the main issue is that field is an array of json objects, the individual fields here will not be parsed.

You still can access them.

If you want to parse some individual fields you will need to use an ingest pipeline to do that, probably will need to use the script processor to traverse the array and get the value of the field you want, but I do not have much experience with this.

7

What exactly do you want to do with this field? Maybe you can access it in the way it is already parsed.

8

Querying nested objects like this in KQL requires special syntax, see:

9

The integration with VisionOne includes a mapping, but it does not cover all the fields we are interested in, only a subset. Some attributes in event.original are relevant to us, but they are not searchable because they are internal objects that cannot be queried.

10

The example you shared is of a field that it is already parsed.

The indicators object is parsed into the field trend_micro_vision_one.alert.indicators.

As mentioned, this is originally an array of json objects.

You can search on it by using trend_micro_vision_one.alert.indicators.field for example.

Can you provide some example of what you mean with this?

If you search for trend_micro_vision_one.alert.indicators.value: "valueTobeExtract" it will return documents where at least one of the items in the indicators array has the field named value equal to valueTobeExtract.

11

Hi Stephenb,
It would not appear possible to add a custom pipeline to the Trend Micro Vision One integration.
There is no option to add a custom pipeline in Edit integration.

12

It is possible, where did you try to add it?

You need to add the integration into the policy, and then edit it, then in the advanced options for each data set you will have something like this:

image
image606×616 87.5 KB

And when you click in Add custom pipeline it will directs you to the pipeline creation page:

image
image3027×523 47.9 KB

Also, if you want to create a custom ingest pipeline that will apply to all data sets, you just need to create an ingest pipeline with the following name:

logs-trend_micro_vision_one.integration@custom

This will then be applied to every data set.

1 Like
13

Thank you.
Now the problem is extract a nested json.
I mean, I have a field (event.original) that contains in a field a stringfy json. If I apply the JSON processor through custom pipeline I get the parsing of all the fields in event.original (in custom field details). Now I would like to recursively apply the processor on the field details.rawDataStr that in turn contains a stringfy json.