Foreign "fields" in JSON document in Kibana Discover

thesn · February 22, 2021, 3:09am

Hi,

my setup:
.log file --> filebeat --> logstash --> elasticsearch <-- kibana

sample of .log file content:

    {
   "@timestamp":"2021-02-22T09:40:32.533+07:00",
   "@version":"1",
   "message":"mappingReloadDTreeError Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type=TCP-ISO8583%2FECHANNEL%2FRES&lastUpdate=2021-02-21T20:37:24.260Z",
   "logger_name":"GenericMapping",
   "thread_name":"Thread-25",
   "level":"DEBUG",
   "level_value":10000,
   "HOSTNAME":"ace-router-echannel-iso8583",
   "logType":"mappingReloadDTreeError",
   "error":[
      {
         "Class":"java.io.IOException",
         "Text":"Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type\u003dTCP-ISO8583%2FECHANNEL%2FRES\u0026lastUpdate\u003d2021-02-21T20:37:24.260Z",
         "StackTrace":[
            {
               "ClassName":"sun.reflect.GeneratedConstructorAccessor4",
               "LineNumber":-1,
               "MethodName":"newInstance"
            },
            {
               "ClassName":"sun.reflect.DelegatingConstructorAccessorImpl",
               "FileName":"DelegatingConstructorAccessorImpl.java",
               "LineNumber":57,
               "MethodName":"newInstance"
            },
            {
               "ClassName":"java.lang.reflect.Constructor",
               "FileName":"Constructor.java",
               "LineNumber":437,
               "MethodName":"newInstance"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$4",
               "FileName":"MbMappingCompute.java",
               "LineNumber":361,
               "MethodName":"apply"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$4",
               "FileName":"MbMappingCompute.java",
               "LineNumber":339,
               "MethodName":"apply"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute",
               "FileName":"MbMappingCompute.java",
               "LineNumber":339,
               "MethodName":"getDTree"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$5",
               "FileName":"MbMappingCompute.java",
               "LineNumber":1088,
               "MethodName":"run"
            }
         ]
      },
      {
         "Class":"java.io.IOException",
         "Text":"Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type\u003dTCP-ISO8583%2FECHANNEL%2FRES\u0026lastUpdate\u003d2021-02-21T20:37:24.260Z",
         "StackTrace":[
            {
               "ClassName":"sun.net.www.protocol.http.HttpURLConnection",
               "FileName":"HttpURLConnection.java",
               "LineNumber":1912,
               "MethodName":"getInputStream0"
            },
            {
               "ClassName":"sun.net.www.protocol.http.HttpURLConnection",
               "FileName":"HttpURLConnection.java",
               "LineNumber":104,
               "MethodName":"access$200"
            },
            {
               "ClassName":"sun.net.www.protocol.http.HttpURLConnection$9",
               "FileName":"HttpURLConnection.java",
               "LineNumber":1502,
               "MethodName":"run"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$4",
               "FileName":"MbMappingCompute.java",
               "LineNumber":360,
               "MethodName":"apply"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$4",
               "FileName":"MbMappingCompute.java",
               "LineNumber":339,
               "MethodName":"apply"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute",
               "FileName":"MbMappingCompute.java",
               "LineNumber":339,
               "MethodName":"getDTree"
            },
            {
               "ClassName":"id.co.hanoman.mapping.MbMappingCompute$5",
               "FileName":"MbMappingCompute.java",
               "LineNumber":1088,
               "MethodName":"run"
            }
         ]
      }
   ]
}

inserted JSON document in Kibana Discover:

    {
  "_index": "bjb-dev-tcp-000005",
  "_type": "_doc",
  "_id": "yjGcx3cBUmNrAslYLnwZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-02-22T02:40:32.533Z",
    "@version": "1",
    "log": {
      "file": {
        "path": "/opt/hanoman/dev-ace/log/ace-router-echannel-iso8583.log"
      },
      "offset": 456654
    },
    "tags": [
      "_jsonparsefailure",
      "beats_input_codec_json_applied"
    ],
    "container": {
      "id": "ace-router-echannel-iso8583.log"
    },
    "dev": "true",
    "input": {
      "type": "log"
    },
    "thread_name": "Thread-25",
    "error": [
      {
        "Text": "Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type=TCP-ISO8583%2FECHANNEL%2FRES&lastUpdate=2021-02-21T20:37:24.260Z",
        "StackTrace": [
          {
            "LineNumber": -1,
            "ClassName": "sun.reflect.GeneratedConstructorAccessor4",
            "MethodName": "newInstance"
          },
          {
            "LineNumber": 57,
            "MethodName": "newInstance",
            "ClassName": "sun.reflect.DelegatingConstructorAccessorImpl",
            "FileName": "DelegatingConstructorAccessorImpl.java"
          },
          {
            "LineNumber": 437,
            "MethodName": "newInstance",
            "ClassName": "java.lang.reflect.Constructor",
            "FileName": "Constructor.java"
          },
          {
            "LineNumber": 361,
            "MethodName": "apply",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$4",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 339,
            "MethodName": "apply",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$4",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 339,
            "MethodName": "getDTree",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 1088,
            "MethodName": "run",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$5",
            "FileName": "MbMappingCompute.java"
          }
        ],
        "Class": "java.io.IOException"
      },
      {
        "Text": "Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type=TCP-ISO8583%2FECHANNEL%2FRES&lastUpdate=2021-02-21T20:37:24.260Z",
        "StackTrace": [
          {
            "LineNumber": 1912,
            "ClassName": "sun.net.www.protocol.http.HttpURLConnection",
            "MethodName": "getInputStream0",
            "FileName": "HttpURLConnection.java"
          },
          {
            "LineNumber": 104,
            "ClassName": "sun.net.www.protocol.http.HttpURLConnection",
            "MethodName": "access$200",
            "FileName": "HttpURLConnection.java"
          },
          {
            "LineNumber": 1502,
            "MethodName": "run",
            "ClassName": "sun.net.www.protocol.http.HttpURLConnection$9",
            "FileName": "HttpURLConnection.java"
          },
          {
            "LineNumber": 360,
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$4",
            "MethodName": "apply",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 339,
            "MethodName": "apply",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$4",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 339,
            "MethodName": "getDTree",
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute",
            "FileName": "MbMappingCompute.java"
          },
          {
            "LineNumber": 1088,
            "ClassName": "id.co.hanoman.mapping.MbMappingCompute$5",
            "MethodName": "run",
            "FileName": "MbMappingCompute.java"
          }
        ],
        "Class": "java.io.IOException"
      }
    ],
    "host": {
      "hostname": "bjbmwacedev",
      "name": "bjbmwacedev",
      "containerized": false,
      "id": "740a378d956340ce96a0782348ca00ea"
    },
    "level_value": 10000,
    "logger_name": "GenericMapping",
    "logType": "mappingReloadDTreeError",
    "HOSTNAME": "ace-router-echannel-iso8583",
    "level": "DEBUG",
    "msg": "mappingReloadDTreeError Server returned HTTP response code: 500 for URL: http://proxy:10080/api/v1/mapping/list?type=TCP-ISO8583%2FECHANNEL%2FRES&lastUpdate=2021-02-21T20:37:24.260Z"
  },
  "fields": {
    "dt.Message.RC.0099.$.updatedAt": [],
    "path.path.SET_HEADER.$.createdAt": [],
    "path.path.COPY_REQUEST_DATA.$.createdAt": [],
    "response.expiresAt": [],
    "path.createdAt": [],
    "data.JMSTimestamp": [],
    "jms.Expiration": [],
    "dt.Message.RC.0000.$.updatedAt": [],
    "path.path.TO_ECHANNEL.$.updatedAt": [],
    "Time": [],
    "data.MQMD.PutDate": [],
    "data.JMS.Expiration": [],
    "DATA.ACCESS.updatedAt": [],
    "path.path.TO_GSS.$.updatedAt": [],
    "path.path.SET_HEADER.$.updatedAt": [],
    "data.DELETE.createdAt": [],
    "jms.Timestamp": [],
    "data.Instansi.validations.createdAt": [],
    "data.Instansi.validations.updatedAt": [],
    "data.Message.Time": [],
    "path.updatedAt": [],
    "response.lastUpdate": [],
    "dt.Message.RC.0099.$.createdAt": [],
    "data.ACCESS.Token.api.createdAt": [],
    "data.DELETE.updatedAt": [],
    "data.ACCESS.updatedAt": [],
    "dt.Message.RC.0000.$.createdAt": [],
    "data.ACCESS.Token.api.updatedAt": [],
    "data.JMS.Timestamp": [],
    "data.Time": [],
    "data.UPDATE.original.createdAt": [],
    "DATA.ACCESS.createdAt": [],
    "DATA.MQMD.PutDate": [],
    "path.path.TO_ECHANNEL.$.createdAt": [],
    "data.ACCESS.Token.time": [],
    "data.UPDATE.original.updatedAt": [],
    "data.JMSExpiration": [],
    "path.path.createdAt": [],
    "@timestamp": [
      "2021-02-22T02:40:32.533Z"
    ],
    "data.ACCESS.apiSecret.updatedAt": [],
    "data.ACCESS.createdAt": [],
    "path.path.TO_GSS.$.createdAt": [],
    "path.path.COPY_REQUEST_DATA.$.updatedAt": [],
    "data.Logger.Time": [],
    "path.path.updatedAt": [],
    "data.ACCESS.apiSecret.createdAt": []
  },
  "sort": [
    1613961632533
  ]
}

Questions:

Where is this part coming from? Because I cannot find it in the source log file. Who add this part to the JSON document (Elasticsearch, Logstash, Kibana, or Filebeat)? Why is it being added?

    "fields": {
    "dt.Message.RC.0099.$.updatedAt": [],
    "path.path.SET_HEADER.$.createdAt": [],
    "path.path.COPY_REQUEST_DATA.$.createdAt": [],
    "response.expiresAt": [],
    "path.createdAt": [],
    "data.JMSTimestamp": [],

How can I remove that part?

Please enlighten me.
Thank you.

warkolm · February 22, 2021, 3:29am

I'm moving this to the Beats category as it's related to ingestion.

Can you share your Filebeat config?

thesn · February 22, 2021, 6:44am

Hi,

Thanks for your reply.
Here is my filebeat config:

    filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /opt/hanoman/dev-ace/log/ace*.log
  fields.dev: "true"
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true
  fields_under_root: true

  exclude_files: ['.gz$']

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.logstash:
  hosts: ["10.6.226.80:5046"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

system · March 22, 2021, 6:45am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
'Index' json 'message' field without Logstash but within Filebeat? Filebeat > ES > Kibana Beats filebeat	2	292	April 1, 2022
Filebeat send json data to Elastiksearch isuue Logs	6	516	September 18, 2020
Just one unknown field - Filebeat json parsing Beats filebeat	5	1071	July 2, 2020
Filebeat 5.0.0-alpha3 not parsing the JSON messages, it shows up as a string Beats filebeat	4	1386	July 13, 2016
Parse json data from log file into Kibana via Filebeat and Logstash Beats filebeat	10	9548	May 19, 2020

Foreign "fields" in JSON document in Kibana Discover

Related topics