Filebeat decoded JSON fields aren't searchable

Elastic Stack Kibana
1

Sending logs (JSON nested) using filebeat, the fields are getting created in (confirming from index pattern) , but the decoded fields are not visible in Kibana Discover tab.

Sample data

{"id":"5f5a0206-c431-4d76-9e8a-06095a2c2317","name":"Mavent-EPPS","date":"2020-11-22 21:55:33 PST","apmModuleDetailViewData":[{"licenseModuleType":"APM","peakUsage":172,"licenseProvisioned":220}],"nonApmModuleDetailViewData":[{"licenseModuleType":"MACHINE_AGENT","peakUsage":31,"licenseProvisioned":220},{"licenseModuleType":"SIM_MACHINE_AGENT","peakUsage":18,"licenseProvisioned":40}],"apmStackGraphViewData":[{"licenseModuleType":"JAVA","peakUsage":142,"licenseProvisioned":0},{"licenseModuleType":"DOT_NET","peakUsage":30,"licenseProvisioned":0}]}

Fields getting created -

image
image1170×616 102 KB

But not searchable

image
image1114×877 213 KB

Filebeat Inputs along with processors -

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /opt/logs/LicenseRulesUtil.log
  #json.keys_under_root: true
  #json.add_error_key: true
  #json.overwrite_keys: false
  processors:
    - decode_json_fields:
        fields: ["message"]
        process_array: true
        target: ""
        max_depth: 5
        overwrite_keys: true
    - timestamp:
        field: date
        layouts:
          - '2006-01-02 15:04:05 PST'
        test:
          - '2020-11-22 21:55:33 PST'
        timezone: "America/Los_Angeles"

  exclude_files: ['.gz$']
  ignore_older: 72h

  fields:
    name: appd_license_logs
    index: appd_license
    environment: prod
  fields_under_root: true

Any help appreciated ..
Thanks

2

I'm guessing that's because either these fields are not indexed correctly or because the Kibana index pattern needs refreshing.

Can you check your Filebeat index mappings for the missing fields and post their mappings here? https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html

Also, could you try refreshing your Filebeat index pattern in Kibana? https://www.elastic.co/guide/en/kibana/current/index-patterns.html#_refresh_the_data_fields

Shaunak

3

I tried refreshing the index pattern in Kibana, but it did not help .

I also believe the fields are indexed, because those fields appear in the index pattern (I've attached the snapshot earlier).

After much trials, I found, when trying to search the data using the search Bar in Discover (Kibana), the fields are usable/searchable. I could also use them in visualizations, but they still appear to not decoded in the Kibana fields list.

4

@shaunak

Field index Mappings

{
  "appd_license-prod-2020.11.23" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "agent" : {
          "properties" : {
            "ephemeral_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "apmModuleDetailViewData" : {
          "properties" : {
            "licenseModuleType" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "licenseProvisioned" : {
              "type" : "long"
            },
            "peakUsage" : {
              "type" : "long"
            }
          }
        },
        "apmStackGraphViewData" : {
          "properties" : {
            "licenseModuleType" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "licenseProvisioned" : {
              "type" : "long"
            },
            "peakUsage" : {
              "type" : "long"
            }
          }
        },
        "date" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "environment" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "host" : {
          "properties" : {
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "input" : {
          "properties" : {
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "offset" : {
              "type" : "long"
            }
          }
        },
        "name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "nonApmModuleDetailViewData" : {
          "properties" : {
            "licenseModuleType" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "licenseProvisioned" : {
              "type" : "long"
            },
            "peakUsage" : {
              "type" : "long"
            }
          }
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "top" : {
          "properties" : {
            "ingest_method" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    }
  }
}
5

Ah, sorry I missed the index patterns screenshot earlier.

Hmm, so it looks like the fields are being mapped and the index pattern in Kibana "sees" them too, e.g. apmModuleDetailViewData.licenseModuleType, apmModuleDetailViewData.licenseModuleType.keyword, etc.

I'd expect these to show up in the Discover fields list on the left but I'm not sure why they're not showing up. I wonder if it has something to do with the field values being arrays (not nested). I'm going to transfer this post to the Kibana forums so hopefully the right developers can take a look.

6

Thanks @shaunak.

Hope Someone from the team will be able to help and yes, it looks like Array of values, looking deeper into the data,

"apmStackGraphViewData":[{"licenseModuleType":"JAVA","peakUsage":142,"licenseProvisioned":0},{"licenseModuleType":"DOT_NET","peakUsage":30,"licenseProvisioned":0}]

While building the visualizations, I realized the data is getting overwritten for the following (maybe) -

apmStackGraphViewData.licenseModuleType
apmStackGraphViewData.peakUsage
apmStackGraphViewData.licenseProvisioned

Not sure, how to process and not lose it.

Original Event Example -

{"id":"5f5a0206-c431-4d76-9e8a-06095a2c2317","name":"Mavent-EPPS","date":"2020-11-22 21:55:33 PST","apmModuleDetailViewData":[{"licenseModuleType":"APM","peakUsage":172,"licenseProvisioned":220}],"nonApmModuleDetailViewData":[{"licenseModuleType":"MACHINE_AGENT","peakUsage":31,"licenseProvisioned":220},{"licenseModuleType":"SIM_MACHINE_AGENT","peakUsage":18,"licenseProvisioned":40}],"apmStackGraphViewData":[{"licenseModuleType":"JAVA","peakUsage":142,"licenseProvisioned":0},{"licenseModuleType":"DOT_NET","peakUsage":30,"licenseProvisioned":0}]}
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.