The tags field in the message conflicts with the built-in tags field in Logstash

Elastic Stack Logstash
1

The version:
logstash:8.12.2

I have the following data structure in kafka.

{
  "contents": {
    "content": "2024-03-13 12:17:18,614 [aecd9c48e39fb9fe,aecd9c48e39fb9fe,,false] INFO  com.td.mc.gate.util...."
  },
  "tags": {
    "host.ip": "192.168.1.1",
    "host.name": "work01",
    "ilogtail_platform": "custom",
    "ilogtail_project": "p1",
    "ilogtail_svcname": "gate",
    "ilogtail_topic": "custom-gate",
    "k8s.namespace.name": "default",
    "k8s.node.ip": "192.168.1.1",
    "k8s.node.name": "work01",
    "k8s.pod.name": "gate-5fb8865d5b-rccq5",
    "k8s.pod.uid": "a77dda33-fab8-405d-9421-0c290e94a2e7",
    "log.file.path": "/data/logs/gate_info.log"
  },
  "time": 1710312022
}

then logstash consume this topic, i wish the result is

{
  "host.ip": "192.168.1.1",
  "host.name": "work01",
  "ilogtail_platform": "custom",
  "ilogtail_project": "p1",
  "ilogtail_svcname": "gate",
  "ilogtail_topic": "custom-gate",
  "k8s.namespace.name": "default",
  "k8s.node.ip": "192.168.1.1",
  "k8s.node.name": "work01",
  "k8s.pod.name": "gate-5fb8865d5b-rccq5",
  "k8s.pod.uid": "a77dda33-fab8-405d-9421-0c290e94a2e7",
  "log.file.path": "/data/logs/gate_info.log",
  "@timestamp": "2024-03-13T06:43:04.976389522Z",
  "message": "2024-03-13 12:17:18,614 [aecd9c48e39fb9fe,aecd9c48e39fb9fe,,false] IN......",
  "event": {
    "original": "{....}"
  },
  "@version": "1",
  "contents": {},
  "time": 1710312022
}

but tags field in the topic conflicts with the built-in tags field in Logstash,so it was renamed to _tags,so i changed the configure:

  mutate {
    copy => { "[contents][content]" => "message" }
    remove_field => [ "[contents][content]" ]
  }

  json {
    source => "_tags"
    target => "parsedMsg"
  }

  mutate {
    add_field => { 
      "topic" => "%{[parsedMsg][ilogtail_topic]}"
    }
  }

but the reuslt was:

{
  "topic": "%{[parsedMsg][ilogtail_topic]}",
  "tags": ["_tagsparsefailure","_jsonparsefailure"],
  "message": "2024-03-13 12:17:18,614 [aecd9c48e39fb9fe,aecd9c48e39fb9fe,,false] IN......",
  "event": {
    "original": "....."
  },
  "@version": "1",
  "_tags": [
    {
      "host.ip": "192.168.1.1",
      "host.name": "work01",
      "ilogtail_platform": "custom",
      "ilogtail_project": "p1",
      "ilogtail_svcname": "gate",
      "ilogtail_topic": "custom-gate",
      "k8s.namespace.name": "default",
      "k8s.node.ip": "192.168.1.1",
      "k8s.node.name": "work01",
      "k8s.pod.name": "gate-5fb8865d5b-rccq5",
      "k8s.pod.uid": "a77dda33-fab8-405d-9421-0c290e94a2e7",
      "log.file.path": "/data/logs/gate_info.log"
    }
  ],
  "@timestamp": "2024-03-13T06:19:57.781429973Z",
  "contents": {},
  "time": 1710312022
}

so,how to process the _tags?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.