Pretty JSON format of recieved (dot separated) log data fields

Elastic Stack Kibana
1

Hi all,
can you help me, please? I want to see my whole log entries in pretty JSON format on Kibana Discovery page.

What I received from logstash:

  • log entries with "dot case format" fields like: http.host, http.body, http.headers

What I can see now - log entry in Expanded document - JSON view on Discovery page:

  • pretty printed agent (id, type, version....), host (id, name, os.name....), log (file.path, flags, offset) and other fields
  • not pretty printed http and other custom fields
    obrazek
    obrazek718×713 45.8 KB

What I want to see:

  • pretty structured JSON showing all http information together like agent.version, agent.id, agent.type, agent.hostname and other dot separated "default" kibana fields.
    Example of pretty printed JSON:

      {     
  "agent": {
      	"version": "7.5.1",
      	"id": "dd8dbc43-aa66-4934-b889-dfe2ff3c7c73",
      	"type": "filebeat",
      	"ephemeral_id": "ec51746e-460e-433f-9e4c-5d8fe1a9a9e7",
      	"hostname": "crkonkol"
      },
  "http": {
      	"host": "mchcon.clance.local",
      	"method": "POST",
      	"headers": "{accept-encoding=[gzip,deflate], content-type=[application/json;charset=UTF-8], authorization=[Bearer 3*****], accept=[application/json], accept-language=[cs], content-length=[80], host=\\[mchcon.clance.local\\], connection=[Keep-Alive], user-agent=[Apache-HttpClient/4.1.1 (java 1.5)]}",
      	"path": "/v1/user/get",
      	"body": "{\n    \"mut\": \"dc1b89e39dda7df8eca553dc44f21fe287463853b6c6c92856c93a5e7eb44d7c\"\n}",
      },
  "severity": "DEBUG",
      "@version": "1",
      "logger": "c.c.s.c.json.InboundTracing.received",
      "input": {
      	"type": "log"
      },
      "nodeid": "crkonkol",
      "tags": [
      	"beats_input_codec_plain_applied"
      ],
      "timestamp": "2020-01-29 12:09:37,632",
      "log": {
      	"offset": 201372,
      	"file": {
      		"path": "/opt/mch/logs/j2ee/mub-crkonkol-technical.log"
      	}
      }
  }

Why Kibana shows log, agent and other fields differently than my http fields (or fields separated from log message)? What can I do for pretty JSON view of my fields?

We are using last stable Kibana, Logstash, Filebeat versions (Kibana 7.5.2 - new installation and configuration without plugins)
Thanks a lot, best regards
RK

2

Hi @Roman_Konkol. Can you share the mapping for your index? I wonder if the http mapping is missing or incorrect?

3

I hope that I found correct mapping. (GET /filebeat-7.5.1-2020.02.04/_mapping or Management - Index management - filebeat index and Mapping in the detail screen)

"http": {
        "properties": {
          "body": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "code": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "headers": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "host": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "method": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "path": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "query": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
4

Thanks. Unfortunately, I'm not able to recreate the issue on v7.5.2. Because I am seeing properly nested JSON in Discover under my http object [1]. But I may be using a different module (nginx) than you, so maybe that's why my mapping [2] is different than yours.

Are you connecting Filebeat directly to Elasticsearch or through Logstash?

[1] Snippet from Discover

"http": {
      "request": {
        "referrer": "https://kibana-proxy.local/tfg/login?next=%2Ftfg%2F",
        "method": "GET"
      },
      "response": {
        "status_code": 200,
        "body": {
          "bytes": 6371
        }
      },
      "version": "1.1"
    },

[2] My mapping:

"http": {
        "properties": {
          "request": {
            "properties": {
              "body": {
                "properties": {
                  "bytes": {
                    "type": "long"
                  },
                  "content": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "bytes": {
                "type": "long"
              },
              "method": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "referrer": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "response": {
            "properties": {
              "body": {
                "properties": {
                  "bytes": {
                    "type": "long"
                  },
                  "content": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "bytes": {
                "type": "long"
              },
              "status_code": {
                "type": "long"
              }
            }
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.