How to use JSON filter correctly to parse my data?

Elastic Stack Logstash
1

Below is the JSON format of the current data and I'm using JSON filter to handle the nested JSON construct but it is not working as expected.

log field is again a JSON field, which I would like to expand further as Message_Details.Handler : reforce-nfs-to-partition, Message_Details.Partition so on..

Can you please point out, where am I going wrong ? thanks.

{
  "_index": "nodejs-2023.04.08",
  "_type": "_doc",
  "_id": "kfjnd7fhgjkgjg8dnsgs",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "@timestamp": "2023.04.08T17:37:41.994Z",
    "stream": "stdout",
    "log": "{\"Message_Details\":{\"Handler\":\"reforce-nfs-to-partition\",\"Partition\":0,\"Offset\":1787015,\"Metadata\":null,\"Error\":null},\"file\":\"/home/jenkins/agent/workspace/nodejs_main/deploy.java\",\"func\":\"deploy.main\",\"level\":\"info\",\"msg\":\"Processed\",\"time\":\"2023-06-08T17:37:40Z\"}",
    "kubernetes": {
      "container_image": "docker.org/nodejs:2.0.0.eedf4b0e.2022.08.08",
      "host": "k8s-uat-depl-worker-pool-1-fgv2n-7c67f68b88-rhpvt",
      "namespace_id": "4c447696-1813-47e5-9c4f-fa1fa1820d54",
      "pod_id": "389e2f16-637a-4bf3-a96b-c3af9a3615eb",
      "pod_ip": "192.168.30.235",
      "namespace_name": "avi-name-check",
      "master_url": "https://10.5.0.10:443/api",
      "labels": {
        "pod-template-hash": "58668b675d",
        "app": "nodejs"
      },
      "pod_name": "nodejs-58668b675d-hxcqk",
      "namespace_labels": {
        "kubernetes_io/metadata_name": "avi-name-check"
      },
      "container_image_id": "docker.org/nodejs@sha256:e9f72ffa02ec2d806c29d756b570b4ece9a14c9bb7d9bb0efdd028917578b689",
      "container_name": "nodejs"
    },
    "docker": {
      "container_id": "52985122463922d79c7d864c8kohbj8543dbji90993388hhgfd"
    },
    "headers": {
      "http_accept": "*/*",
      "http_user_agent": "elasticsearch-ruby/7.13.3 (RUBY_VERSION: 2.6.8; linux x86_64; Faraday v1.5.1)",
      "http_version": "HTTP/1.1",
      "request_path": "/_bulk",
      "content_type": "application/x-ndjson",
      "request_method": "POST",
      "http_host": "drpelk-01-p-dur.cec.lab.emc.com:9182",
      "content_length": "202711"
    }
  },
  "fields": {
    "@timestamp": [
      "2023-06-08T17:37:41.994Z"
    ]
  },
  "sort": [
    1686245861994
  ]
}

My current config looks like this

input {...}
filter {
    json {
       source => "message"
    }
    if [kubernetes][labels][app] in ["nodejs", "cart", "payment"] {
      json {
        source => "log"
      }
    }
}
output {...}

Table format of the current data looks like:

Fluentd
Fluentd1208×688 64.8 KB

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.