Objects showing in Discover as json, not discrete fields

(Dave Martin) #1

After deleting and recreating my .kibana index to fix the API/FORBIDDEN issue, all of my structured fields are showing at top level json blobs.

Refreshing the index pattern doesn't help.

Mapping:
` "meta": {
"dynamic": "true",
"properties": {
"delay": {
"type": "integer"
},
"filter": {
"type": "keyword"
},
"filterhost": {
"type": "keyword"
},
"filtertime": {
"type": "long"
},

etc...
`

In Discover:
'{ "type": "syslog", "target": { "topic": "logqueue-pprd", "index": "log_syslog-2019.05.07" }, "queuehost": [ "logqueue-prod-03", "logqueue-pprd-01" ], "queuetime": 1557256848, "filter": [ "20-syslog-base", "78-category" ], "input": [ "beats", "prodsample" ], "size": 107, "filtertime": 1557256850, "delay": 8, "filterhost": "logfilter-pprd-02.cas-1.cls.vt.edu" }'

(Nathan Reese) #2

Can you provide a screen shot. I am not sure what you mean by "all of my structured fields are showing at top level json blobs". Also, could you provide the complete index mapping.

(Dave Martin) #3

I've got things disassembled at the moment, but I'll get the files shortly.

By "all of my structured fields are showing at top level json blobs", I mean my top level fields show a single value containing all their sub fields.

In the above example, my 'meta' field contains this blob: '{ "type": "syslog", "target": { "topic": "logqueue-pprd", "index": "log_syslog-2019.05.07" }, "queuehost": [ "logqueue-prod-03", "logqueue-pprd-01" ], "queuetime": 1557256848, "filter": [ "20-syslog-base", "78-category" ], "input": [ "beats", "prodsample" ], "size": 107, "filtertime": 1557256850, "delay": 8, "filterhost": "logfilter-pprd-02.cas-1.cls.vt.edu" }'

Rather than having a meta.type, meta.target.topic, meta.target.index, meta.queuehost, meta.queuetime, meta.filter, meta.input, meta.size, meta.filtertime, meta.delay, and meta.filterhost fields, it just displays the 'meta' field.

It displays the 'meta' field as an unknown type, but defaults to string, despite having the proper mappings for meta and it's member fields, as shown above.

This has been working for years.

(Dave Martin) #4

Here's the screen shot.

The index pattern (just the meta fields - note that the sub fields show properly):

And the relevant part of the mapping (sorry, the forum won't let me post non-image attachments) mapping:
indent preformatted text by 4 spaces {
"mapping": {
"doc": {
"dynamic_templates": [
{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"doc_values": true,
"index": "true",
"type": "keyword"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "integer"
},
"beat": {
"properties": {
"hostname": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"version": {
"type": "keyword"
}
}
},
"category": {
"type": "keyword"
},
"event": {
"dynamic": "true",
"properties": {
"hostname": {
"type": "keyword"
},
"id": {
"type": "keyword"
},
"level": {
"type": "keyword"
},
"message": {
"type": "keyword"
},
"pid": {
"type": "keyword"
},
"program": {
"type": "keyword"
},
"reason": {
"type": "keyword"
}
}
},
"fields": {
"properties": {
"group": {
"type": "keyword"
},
"host": {
"type": "keyword"
},
"json": {
"type": "boolean"
},
"name": {
"type": "keyword"
},
"tier": {
"type": "keyword"
},
"type": {
"type": "keyword"
}
}
},
"message": {
"type": "text"
},
"meta": {
"dynamic": "true",
"properties": {
"delay": {
"type": "integer"
},
"filter": {
"type": "keyword"
},
"filterhost": {
"type": "keyword"
},
"filtertime": {
"type": "long"
},
"input": {
"type": "keyword"
},
"queuehost": {
"type": "keyword"
},
"queuetime": {
"type": "long"
},
"relayhost": {
"type": "keyword"
},
"size": {
"type": "long"
},
"target": {
"properties": {
"index": {
"type": "keyword"
},
"topic": {
"type": "keyword"
}
}
},
"type": {
"type": "keyword"
}
}
},
"offset": {
"type": "long"
},
"session": {
"properties": {
"duration": {
"type": "keyword"
},
"input_bytes": {
"type": "keyword"
},
"output_bytes": {
"type": "keyword"
},
"start_time": {
"type": "keyword"
}
}
},
"source": {
"type": "keyword"
},
"tags": {
"type": "keyword"
},
"type": {
"type": "keyword"
},
"user": {
"properties": {
"sponsor": {
"type": "keyword"
},
"sponsor_profile": {
"type": "keyword"
}
}
},
"username": {
"type": "keyword"
}
}
}
}
}

(Dave Martin) #5

Any thoughts?

All of my indices are doing this.

If I manually edit the index pattern definition to remove the entry for the 'meta' field, all of the sub fields display properly. Of course, I need to do that for all objects.

This is clearly a Kibana bug (incorrect index pattern creation). Can I get a bug number and ETR, please?

(Cristiano Ferreira) #6

Hi,

Have you tried to map the fields "beat", "event", "fields", and "meta" with type: nested like:

"properties": {
  "beat": {
    "type": "nested" 
  }
}

https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

Also, remove the

"dynamic_templates": [ {
    "string_fields": {
        "match": "*",
        "match_mapping_type": "string"
      }
 }]

That is matching all fields, thus making all fields of type string.

(Dave Martin) #7

Thanks for replying.

No, we aren't looking for nested documents. Just structured fields. The elastic mapping is fine. Kibana just isn't building the correct index pattern.

I'll try removing it, but we are getting dynamic numeric fields. (meta.size for example is of type 'long')

(Dave Martin) #8

Sorry, removing the match_mapping_type failed to load. Removing the entire dynamic_templates block put the *.keyword fields back. We went to some effort to fix that. It also didn't fix the base problem of structured fields showing as json. (rewriting the index pattern does, but I'd rather not have to do that every time someone refreshes a pattern.)

(Dave Martin) #9

I tried setting the mapping for those objects to 'nested', rather than 'object' and Kibana didn't even show them anymore.

Anyone have any ideas on how to tell Kibana to treat these as objects, please?

(Dave Martin) #10

https://www.elastic.co/guide/en/kibana/6.7/document-data.html#adding-columns

You can see an example here of what should be happening.

Here's the documentation for the elastic mapping:
https://www.elastic.co/guide/en/elasticsearch/reference/6.7/object.html