Foreign "fields" in JSON document in Kibana Discover

thesn · February 22, 2021, 3:09am

Hi,

my setup:
.log file --> filebeat --> logstash --> elasticsearch <-- kibana

sample of .log file content:

    {
   "@timestamp":"2021-02-22T09:40:32.533+07:00",
   "@version":"1",
   "message":"mappingReloadDTreeError Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type=TCP-ISO8583%2FECHANNEL%2FRES&lastUpdate=2021-02-21T20:37:24.260Z",
   "logger_name":"GenericMapping",
   "thread_name":"Thread-25",
   "level":"DEBUG",
   "level_value":10000,
   "HOSTNAME":"ace-router-echannel-iso8583",
   "logType":"mappingReloadDTreeError",
   "error":[
      {
         "Class":"java.io.IOException",
         "Text":"Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type\u003dTCP-ISO8583%2FECHANNEL%2FRES\u0026lastUpdate\u003d2021-02-21T20:37:24.260Z",
         "StackTrace":[
            {
               "ClassName":"sun.reflect.GeneratedConstructorAccessor4",
               "LineNumber":-1,
               "MethodName":"newInstance"
            },
            {
               "ClassName":"sun.reflect.DelegatingConstructorAccessorImpl",
               "FileName":"DelegatingConstructorAccessorImpl.java",
               "LineNumber":57,
               "MethodName":"newInstance"
            },
            {
               "ClassName":"java.lang.reflect.Constructor",
               "FileName":"Constructor.java",
               "LineNumber":437,
               "MethodName":"newInstance"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$4",
               "FileName":"MbMappingCompute.java",
               "LineNumber":361,
               "MethodName":"apply"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$4",
               "FileName":"MbMappingCompute.java",
               "LineNumber":339,
               "MethodName":"apply"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute",
               "FileName":"MbMappingCompute.java",
               "LineNumber":339,
               "MethodName":"getDTree"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$5",
               "FileName":"MbMappingCompute.java",
               "LineNumber":1088,
               "MethodName":"run"
            }
         ]
      },
      {
         "Class":"java.io.IOException",
         "Text":"Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type\u003dTCP-ISO8583%2FECHANNEL%2FRES\u0026lastUpdate\u003d2021-02-21T20:37:24.260Z",
         "StackTrace":[
            {
               "ClassName":"sun.net.www.protocol.http.HttpURLConnection",
               "FileName":"HttpURLConnection.java",
               "LineNumber":1912,
               "MethodName":"getInputStream0"
            },
            {
               "ClassName":"sun.net.www.protocol.http.HttpURLConnection",
               "FileName":"HttpURLConnection.java",
               "LineNumber":104,
               "MethodName":"access$200"
            },
            {
               "ClassName":"sun.net.www.protocol.http.HttpURLConnection$9",
               "FileName":"HttpURLConnection.java",
               "LineNumber":1502,
               "MethodName":"run"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$4",
               "FileName":"MbMappingCompute.java",
               "LineNumber":360,
               "MethodName":"apply"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$4",
               "FileName":"MbMappingCompute.java",
               "LineNumber":339,
               "MethodName":"apply"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute",
               "FileName":"MbMappingCompute.java",
               "LineNumber":339,
               "MethodName":"getDTree"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$5",
               "FileName":"MbMappingCompute.java",
               "LineNumber":1088,
               "MethodName":"run"
            }
         ]
      }
   ]
}

inserted JSON document in Kibana Discover:

    {
  "_index": "bjb-dev-tcp-000005",
  "_type": "_doc",
  "_id": "yjGcx3cBUmNrAslYLnwZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-02-22T02:40:32.533Z",
    "@version": "1",
    "log": {
      "file": {
        "path": "/opt/hanoman/dev-ace/log/ace-router-echannel-iso8583.log"
      },
      "offset": 456654
    },
    "tags": [
      "_jsonparsefailure",
      "beats_input_codec_json_applied"
    ],
    "container": {
      "id": "ace-router-echannel-iso8583.log"
    },
    "dev": "true",
    "input": {
      "type": "log"
    },
    "thread_name": "Thread-25",
    "error": [
      {
        "Text": "Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type=TCP-ISO8583%2FECHANNEL%2FRES&lastUpdate=2021-02-21T20:37:24.260Z",
        "StackTrace": [
          {
            "LineNumber": -1,
            "ClassName": "sun.reflect.GeneratedConstructorAccessor4",
            "MethodName": "newInstance"
          },
          {
            "LineNumber": 57,
            "MethodName": "newInstance",
            "ClassName": "sun.reflect.DelegatingConstructorAccessorImpl",
            "FileName": "DelegatingConstructorAccessorImpl.java"
          },
          {
            "LineNumber": 437,
            "MethodName": "newInstance",
            "ClassName": "java.lang.reflect.Constructor",
            "FileName": "Constructor.java"
          },
          {
            "LineNumber": 361,
            "MethodName": "apply",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$4",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 339,
            "MethodName": "apply",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$4",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 339,
            "MethodName": "getDTree",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 1088,
            "MethodName": "run",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$5",
            "FileName": "MbMappingCompute.java"
          }
        ],
        "Class": "java.io.IOException"
      },
      {
        "Text": "Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type=TCP-ISO8583%2FECHANNEL%2FRES&lastUpdate=2021-02-21T20:37:24.260Z",
        "StackTrace": [
          {
            "LineNumber": 1912,
            "ClassName": "sun.net.www.protocol.http.HttpURLConnection",
            "MethodName": "getInputStream0",
            "FileName": "HttpURLConnection.java"
          },
          {
            "LineNumber": 104,
            "ClassName": "sun.net.www.protocol.http.HttpURLConnection",
            "MethodName": "access$200",
            "FileName": "HttpURLConnection.java"
          },
          {
            "LineNumber": 1502,
            "MethodName": "run",
            "ClassName": "sun.net.www.protocol.http.HttpURLConnection$9",
            "FileName": "HttpURLConnection.java"
          },
          {
            "LineNumber": 360,
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$4",
            "MethodName": "apply",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 339,
            "MethodName": "apply",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$4",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 339,
            "MethodName": "getDTree",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 1088,
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$5",
            "MethodName": "run",
            "FileName": "MbMappingCompute.java"
          }
        ],
        "Class": "java.io.IOException"
      }
    ],
    "host": {
      "hostname": "bjbmwacedev",
      "name": "bjbmwacedev",
      "containerized": false,
      "id": "740a378d956340ce96a0782348ca00ea"
    },
    "level_value": 10000,
    "logger_name": "GenericMapping",
    "logType": "mappingReloadDTreeError",
    "HOSTNAME": "ace-router-echannel-iso8583",
    "level": "DEBUG",
    "msg": "mappingReloadDTreeError Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type=TCP-ISO8583%2FECHANNEL%2FRES&lastUpdate=2021-02-21T20:37:24.260Z"
  },
  "fields": {
    "dt.Message.RC.0099.$.updatedAt": [],
    "path.path.SET_HEADER.$.createdAt": [],
    "path.path.COPY_REQUEST_DATA.$.createdAt": [],
    "response.expiresAt": [],
    "path.createdAt": [],
    "data.JMSTimestamp": [],
    "jms.Expiration": [],
    "dt.Message.RC.0000.$.updatedAt": [],
    "path.path.TO_ECHANNEL.$.updatedAt": [],
    "Time": [],
    "data.MQMD.PutDate": [],
    "data.JMS.Expiration": [],
    "DATA.ACCESS.updatedAt": [],
    "path.path.TO_GSS.$.updatedAt": [],
    "path.path.SET_HEADER.$.updatedAt": [],
    "data.DELETE.createdAt": [],
    "jms.Timestamp": [],
    "data.Instansi.validations.createdAt": [],
    "data.Instansi.validations.updatedAt": [],
    "data.Message.Time": [],
    "path.updatedAt": [],
    "response.lastUpdate": [],
    "dt.Message.RC.0099.$.createdAt": [],
    "data.ACCESS.Token.api.createdAt": [],
    "data.DELETE.updatedAt": [],
    "data.ACCESS.updatedAt": [],
    "dt.Message.RC.0000.$.createdAt": [],
    "data.ACCESS.Token.api.updatedAt": [],
    "data.JMS.Timestamp": [],
    "data.Time": [],
    "data.UPDATE.original.createdAt": [],
    "DATA.ACCESS.createdAt": [],
    "DATA.MQMD.PutDate": [],
    "path.path.TO_ECHANNEL.$.createdAt": [],
    "data.ACCESS.Token.time": [],
    "data.UPDATE.original.updatedAt": [],
    "data.JMSExpiration": [],
    "path.path.createdAt": [],
    "@timestamp": [
      "2021-02-22T02:40:32.533Z"
    ],
    "data.ACCESS.apiSecret.updatedAt": [],
    "data.ACCESS.createdAt": [],
    "path.path.TO_GSS.$.createdAt": [],
    "path.path.COPY_REQUEST_DATA.$.updatedAt": [],
    "data.Logger.Time": [],
    "path.path.updatedAt": [],
    "data.ACCESS.apiSecret.createdAt": []
  },
  "sort": [
    1613961632533
  ]
}

Questions:

Where is this part coming from? Because I cannot find it in the source log file. Who add this part to the JSON document (Elasticsearch, Logstash, Kibana, or Filebeat)? Why is it being added?

    "fields": {
    "dt.Message.RC.0099.$.updatedAt": [],
    "path.path.SET_HEADER.$.createdAt": [],
    "path.path.COPY_REQUEST_DATA.$.createdAt": [],
    "response.expiresAt": [],
    "path.createdAt": [],
    "data.JMSTimestamp": [],

How can I remove that part?

Please enlighten me.
Thank you.

warkolm · February 22, 2021, 3:29am

I'm moving this to the Beats category as it's related to ingestion.

Can you share your Filebeat config?

thesn · February 22, 2021, 6:44am

Hi,

Thanks for your reply.
Here is my filebeat config:

    filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /opt/hanoman/dev-ace/log/ace*.log
  fields.dev: "true"
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true
  fields_under_root: true

  exclude_files: ['.gz$']

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.logstash:
  hosts: ["10.6.226.80:5046"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

system · March 22, 2021, 6:45am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.