How to construct the customized fields from the fluentd output

Elastic Stack Logstash
1

I'm capturing the logs from fluentd output onto Logstash using a basic config.

input {
  http {
    port => 8080
  }
}

output {
  elasticsearch {
      hosts => ["<%= @ipaddress%>:9200"]
      index => "fluentd-%{+YYYY.MM.dd}"
  }
}

I see the logs on the Kibana as well, but the format is very non user friendly.

Below is a message in JSON format

{
  "_index": "fluentd-index-2023.05.04",
  "_type": "_doc",
  "_id": "A-AXl4gBG5MKGg2rSPz9",
  "_version": 1,
  "_score": null,
  "_source": {
    "host": "10.3.7.44",
    "@version": "1",
    "@timestamp": "2023-05-04T18:21:05.558Z",
    "message": "{\"index\":{\"_index\":\"fluentd\",\"_type\":\"_doc\"}}\n{\"stream\":\"stderr\",\"character\":\"F\",\"log\":\"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2023-05-04T18:20:02.274129245Z\\\",\\\"caller\\\":\\\"service/node.go:232\\\",\\\"msg\\\":\\\"NodeGetVolumeStats: called with args {VolumeId:13b6f8ee-7bfe-4225-b465-770a77493c77-c31801dc-646c-4e24-a9b4-e1b4d754cfc2 VolumePath:/var/lib/kubelet/pods/ed2cadb7-4fcd-4e79-b2a5-454f0c7f9c4e/volumes/kubernetes.io~csi/pvc-c31801dc-646c-4e24-a9b4-e1b4d754cfc2/mount StagingTargetPath: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}\\\",\\\"TraceId\\\":\\\"d34c63f8-9387-455e-a854-4c6791dd8c5e\\\"}\",\"docker\":{\"container_id\":\"8ad15cce1823fd30bbf903440a18527efa175c38ad2614431c5f483040cd8b2d\"},\"kubernetes\":{\"container_name\":\"vsphere-csi-node\",\"namespace_name\":\"vmware-system-csi\",\"pod_name\":\"vsphere-csi-node-pwpsd\",\"container_image\":\"localhost:5000/vmware.io/vsphere-csi:v2.5.2-8dee76b\",\"container_image_id\":\"sha256:98ac00c31243e8f9da9d774c1b751c650d76897c17ecafd1deaa2dbe01b78527\",\"pod_id\":\"883eefef-ff85-435b-b3ce-132039d200bc\",\"pod_ip\":\"172.8.0.7\",\"host\":\"node-1-env-trellix-1-wkzqx-6565d7d5fb-q8zwd\",\"labels\":{\"app\":\"vsphere-csi-node\",\"controller-revision-hash\":\"6975b6cf94\",\"pod-template-generation\":\"1\",\"role\":\"vsphere-csi\"},\"master_url\":\"https://10.96.03.22:443/api\",\"namespace_id\":\"9c190e48-83e1-45bd-b672-53375ddcccf9\",\"namespace_labels\":{\"kubernetes_io/metadata_name\":\"vmware-system-csi\"}}}\n{\"index\":{\"_index\":\"fluentd\",\"_type\":\"_doc\"}}\n{\"stream\":\"stderr\",\"character\":\"F\",\"log\":\"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2023-05-04T18:20:02.281888498Z\\\",\\\"caller\\\":\\\"service/node.go:232\\\",\\\"msg\\\":\\\"NodeGetVolumeStats: called with args {VolumeId:13b6f8ee-7bfe-4225-b465-770a77493c77-c2997991-244d-4d14-8035-7fed87da92a9 VolumePath:/var/lib/kubelet/pods/ed2cadb7-4fcd-4e79-b2a5-454f0c7f9c4e/volumes/kubernetes.io~csi/pvc-c2997991-244d-4d14-8035-7fed87da92a9/mount StagingTargetPath: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}\\\",\\\"TraceId\\\":\\\"7a4c997a-3421-4a1b-9d62-f7508295ef1e\\\"}\",\"docker\":{\"container_id\":\"8ad15cce1823fd30bbf903440a18527efa175c38ad2614431c5f483040cd8b2d\"},\"kubernetes\":{\"container_name\":\"vsphere-csi-node\",\"namespace_name\":\"vmware-system-csi\",\"pod_name\":\"vsphere-csi-node-pwpsd\",\"container_image\":\"localhost:5000/vmware.io/vsphere-csi:v2.5.2-8dee76b\",\"container_image_id\":\"sha256:98ac00c31243e8f9da9d774c1b751c650d76897c17ecafd1deaa2dbe01b78527\",\"pod_id\":\"883eefef-ff85-435b-b3ce-132039d200bc\",\"pod_ip\":\"172.8.0.7\",\"host\":\"node-1-env-trellix-1-wkzqx-6565d7d5fb-q8zwd\",\"labels\":{\"app\":\"vsphere-csi-node\",\"controller-revision-hash\":\"6975b6cf94\",\"pod-template-generation\":\"1\",\"role\":\"vsphere-csi\"},\"master_url\":\"https://10.96.03.22:443/api\",\"namespace_id\":\"9c190e48-83e1-45bd-b672-53375ddcccf9\",\"namespace_labels\":{\"kubernetes_io/metadata_name\":\"vmware-system-csi\"}}}\n{\"index\":{\"_index\":\"fluentd\",\"_type\":\"_doc\"}}\n{\"stream\":\"stderr\",\"character\":\"F\",\"log\":\"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2023-05-04T18:20:02.288497541Z\\\",\\\"caller\\\":\\\"service/node.go:232\\\",\\\"msg\\\":\\\"NodeGetVolumeStats: called with args {VolumeId:13b6f8ee-7bfe-4225-b465-770a77493c77-f496b0c7-78c9-4633-8031-21b9f374710e VolumePath:/var/lib/kubelet/pods/ed2cadb7-4fcd-4e79-b2a5-454f0c7f9c4e/volumes/kubernetes.io~csi/pvc-f496b0c7-78c9-4633-8031-21b9f374710e/mount StagingTargetPath: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}\\\",\\\"TraceId\\\":\\\"c32e8a9d-3ba7-4a27-8848-2fe1145d3e9f\\\"}\",\"docker\":{\"container_id\":\"8ad15cce1823fd30bbf903440a18527efa175c38ad2614431c5f483040cd8b2d\"},\"kubernetes\":{\"container_name\":\"vsphere-csi-node\",\"namespace_name\":\"vmware-system-csi\",\"pod_name\":\"vsphere-csi-node-pwpsd\",\"container_image\":\"localhost:5000/vmware.io/vsphere-csi:v2.5.2-8dee76b\",\"container_image_id\":\"sha256:98ac00c31243e8f9da9d774c1b751c650d76897c17ecafd1deaa2dbe01b78527\",\"pod_id\":\"883eefef-ff85-435b-b3ce-132039d200bc\",\"pod_ip\":\"172.8.0.7\",\"host\":\"node-1-env-trellix-1-wkzqx-6565d7d5fb-q8zwd\",\"labels\":{\"app\":\"vsphere-csi-node\",\"controller-revision-hash\":\"6975b6cf94\",\"pod-template-generation\":\"1\",\"role\":\"vsphere-csi\"},\"master_url\":\"https://10.96.03.22:443/api\",\"namespace_id\":\"9c190e48-83e1-45bd-b672-53375ddcccf9\",\"namespace_labels\":{\"kubernetes_io/metadata_name\":\"vmware-system-csi\"}}}\n{\"index\":{\"_index\":\"fluentd\",\"_type\":\"_doc\"}}\n{\"stream\":\"stderr\",\"character\":\"F\",\"log\":\"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2023-05-04T18:20:17.54786454Z\\\",\\\"caller\\\":\\\"service/node.go:232\\\",\\\"msg\\\":\\\"NodeGetVolumeStats: called with args {VolumeId:13b6f8ee-7bfe-4225-b465-770a77493c77-77b56295-ab8d-47a7-a306-434609f01e5d VolumePath:/var/lib/kubelet/pods/ddf707ff-9412-4cad-9793-fb4420d29968/volumes/kubernetes.io~csi/pvc-77b56295-ab8d-47a7-a306-434609f01e5d/mount StagingTargetPath: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}\\\",\\\"TraceId\\\":\\\"91b86bf4-60f8-4bb4-a5d4-ea23204962e0\\\"}\",\"docker\":{\"container_id\":\"8ad15cce1823fd30bbf903440a18527efa175c38ad2614431c5f483040cd8b2d\"},\"kubernetes\":{\"container_name\":\"vsphere-csi-node\",\"namespace_name\":\"vmware-system-csi\",\"pod_name\":\"vsphere-csi-node-pwpsd\",\"container_image\":\"localhost:5000/vmware.io/vsphere-csi:v2.5.2-8dee76b\",\"container_image_id\":\"sha256:98ac00c31243e8f9da9d774c1b751c650d76897c17ecafd1deaa2dbe01b78527\",\"pod_id\":\"883eefef-ff85-435b-b3ce-132039d200bc\",\"pod_ip\":\"172.8.0.7\",\"host\":\"node-1-env-trellix-1-wkzqx-6565d7d5fb-q8zwd\",\"labels\":{\"app\":\"vsphere-csi-node\",\"controller-revision-hash\":\"6975b6cf94\",\"pod-template-generation\":\"1\",\"role\":\"vsphere-csi\"},\"master_url\":\"https://10.96.03.22:443/api\",\"namespace_id\":\"9c190e48-83e1-45bd-b672-53375ddcccf9\",\"namespace_labels\":{\"kubernetes_io/metadata_name\":\"vmware-system-csi\"}}}\n{\"index\":{\"_index\":\"fluentd\",\"_type\":\"_doc\"}}\n{\"stream\":\"stderr\",\"character\":\"F\",\"log\":\"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2023-05-04T18:20:18.744131774Z\\\",\\\"caller\\\":\\\"service/node.go:232\\\",\\\"msg\\\":\\\"NodeGetVolumeStats: called with args {VolumeId:13b6f8ee-7bfe-4225-b465-770a77493c77-b98d9674-084c-485f-a92c-0453b89c9a70 VolumePath:/var/lib/kubelet/pods/9afd8ef3-ff86-42a2-9556-cf5a7bf513cc/volumes/kubernetes.io~csi/pvc-b98d9674-084c-485f-a92c-0453b89c9a70/mount StagingTargetPath: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}\\\",\\\"TraceId\\\":\\\"ed8afcca-32e1-41f8-a4d9-723f7edebe6a\\\"}\",\"docker\":{\"container_id\":\"8ad15cce1823fd30bbf903440a18527efa175c38ad2614431c5f483040cd8b2d\"},\"kubernetes\":{\"container_name\":\"vsphere-csi-node\",\"namespace_name\":\"vmware-system-csi\",\"pod_name\":\"vsphere-csi-node-pwpsd\",\"container_image\":\"localhost:5000/vmware.io/vsphere-csi:v2.5.2-8dee76b\",\"container_image_id\":\"sha256:98ac00c31243e8f9da9d774c1b751c650d76897c17ecafd1deaa2dbe01b78527\",\"pod_id\":\"883eefef-ff85-435b-b3ce-132039d200bc\",\"pod_ip\":\"172.8.0.7\",\"host\":\"node-1-env-trellix-1-wkzqx-6565d7d5fb-q8zwd\",\"labels\":{\"app\":\"vsphere-csi-node\",\"controller-revision-hash\":\"6975b6cf94\",\"pod-template-generation\":\"1\",\"role\":\"vsphere-csi\"},\"master_url\":\"https://10.96.03.22:443/api\",\"namespace_id\":\"9c190e48-83e1-45bd-b672-53375ddcccf9\",\"namespace_labels\":{\"kubernetes_io/metadata_name\":\"vmware-system-csi\"}}}\n{\"index\":{\"_index\":\"fluentd\",\"_type\":\"_doc\"}}\n{\"stream\":\"stderr\",\"character\":\"F\",\"log\":\"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2023-05-04T18:20:23.71954069Z\\\",\\\"caller\\\":\\\"service/node.go:232\\\",\\\"msg\\\":\\\"NodeGetVolumeStats: called with args {VolumeId:13b6f8ee-7bfe-4225-b465-770a77493c77-d2c52bb0-d20f-44b7-bab6-668d8a0c8ac8 VolumePath:/var/lib/kubelet/pods/50ef8bb3-b513-45db-8bf8-85be446f1681/volumes/kubernetes.io~csi/pvc-d2c52bb0-d20f-44b7-bab6-668d8a0c8ac8/mount StagingTargetPath: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}\\\",\\\"TraceId\\\":\\\"6c8c26c4-6e1c-42b4-bace-9d159859f331\\\"}\",\"docker\":{\"container_id\":\"8ad15cce1823fd30bbf903440a18527efa175c38ad2614431c5f483040cd8b2d\"},\"kubernetes\":{\"container_name\":\"vsphere-csi-node\",\"namespace_name\":\"vmware-system-csi\",\"pod_name\":\"vsphere-csi-node-pwpsd\",\"container_image\":\"localhost:5000/vmware.io/vsphere-csi:v2.5.2-8dee76b\",\"container_image_id\":\"sha256:98ac00c31243e8f9da9d774c1b751c650d76897c17ecafd1deaa2dbe01b78527\",\"pod_id\":\"883eefef-ff85-435b-b3ce-132039d200bc\",\"pod_ip\":\"172.8.0.7\",\"host\":\"node-1-env-trellix-1-wkzqx-6565d7d5fb-q8zwd\",\"labels\":{\"app\":\"vsphere-csi-node\",\"controller-revision-hash\":\"6975b6cf94\",\"pod-template-generation\":\"1\",\"role\":\"vsphere-csi\"},\"master_url\":\"https://10.96.03.22:443/api\",\"namespace_id\":\"9c190e48-83e1-45bd-b672-53375ddcccf9\",\"namespace_labels\":{\"kubernetes_io/metadata_name\":\"vmware-system-csi\"}}}\n",
    "headers": {
      "request_path": "/_bulk",
      "request_method": "POST",
      "content_length": "8464",
      "http_user_agent": "elasticsearch-ruby/7.13.3 (RUBY_VERSION: 2.6.8; linux x86_64; Faraday v1.5.1)",
      "http_host": "elk.my.org:8080",
      "http_accept": "*/*",
      "http_version": "HTTP/1.1",
      "content_type": "application/x-ndjson"
    }
  },
  "fields": {
    "@timestamp": [
      "2023-05-04T18:21:05.558Z"
    ]
  },
  "sort": [
    1686162065558
  ]
}

What I'm looking ideally is, something like this. Is this possible to construct the below one from the fluentd logs ?

Fluentd
Fluentd1208×688 64.8 KB

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.