In Kibana 7.6.2 discover tab, I have an index that displays all json nested field data as a hyphen/dash. However, I'm expecting data to display. Here's an example of one field:
payments.payer.payerIdentification.name
payments.payer.payerIdentification.name.keyword
If I drill into the row (or view data in _source) the json data exists. Simplified json example:
{
"_index": "payment",
"_type": "_doc",
"_id": "wOFdXHEB3lhSxZ8rk2wc",
"_version": 1,
"_score": 0,
"_source": {
...
"payments": [{
"payer": {
"payerIdentification": {
"name": "abc"
}
}
}]
}
}
Index auto mapping creates the following mapping (for the field):
{
"mapping": {
"_doc": {
"properties": {
"payments": {
"properties": {
...
"payer": {
"properties": {
"payerIdentification": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
}
}
}
}
}
I can "search" and "filter" on this field successfully (so the field does have data): e.g. payments.payer.payerIdentification.name: "abc"
I can add the payments (top level property) to discover and it displays json data correctly.
The Elasticsearch index was created using the "Elasticsearch for Apache Hadoop" library with default settings: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/reference.html
The source Spark dataframe structure looks good (for the field):
root
|-- id: string (nullable = true)
|-- ...
|-- payments: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- payer: struct (nullable = true)
| | | |-- payerIdentification: struct (nullable = true)
| | | | |-- name: string (nullable = true)
In comparison, there is another similar Elasticsearch index (created by another developer) that works as expected. I don't know how the data was loaded. But the mapping and index patterns are the same. The only differences I notice when working with the index:
a. All fields are displayed in Discover > Available Fields. In the problem index above, only top level fields display in available fields e.g. payment. Note, I can turn off "hide missing fields" and see more fields including payments.payer.payerIdentification.name.
b. When viewing the _source field, all fields display as top level fields e.g. payments.payer.payerIdentification.name exists as a top level field. In the problem index above, only the "payments" field shows top level (but drilling into payments does show all the correct json data).
c. The good index "health" displays as green while the problem index above shows as "yellow.
If I had to guess, I'd guess the good index added two different fields to _source (payments and payments.payer.payerIdentification.name) which is why both fields appear at the top level.
Additional Update
I tried changing the Spark dataframe to include a top level field "payments.payer.payerIdentification.name" and removed the "payments" field (i.e. chunk of nested json). This new field does behave as expected and shows in Discover > Available fields and the correct data when added to the grid.
Spark dataframe structure :
root
|-- id: string (nullable = true)
|-- ...
|-- payments.payer.payerIdentification.name: array (nullable = true)
| |-- element: string (containsNull = true)
However, it's not clear if this is the correct approach. It's a lot more work to parse out each field from nested json (prior to import into Elasticsearch). I would have thought there was a way of getting this correct behavior by importing only one deeply nested json field and using mappings somehow.
Thanks!