Objects showing in Discover as json, not discrete fields

Elastic Stack Kibana
1

After deleting and recreating my .kibana index to fix the API/FORBIDDEN issue, all of my structured fields are showing at top level json blobs.

Refreshing the index pattern doesn't help.

Mapping:
` "meta": {
"dynamic": "true",
"properties": {
"delay": {
"type": "integer"
},
"filter": {
"type": "keyword"
},
"filterhost": {
"type": "keyword"
},
"filtertime": {
"type": "long"
},

etc...
`

In Discover:
'{ "type": "syslog", "target": { "topic": "logqueue-pprd", "index": "log_syslog-2019.05.07" }, "queuehost": [ "logqueue-prod-03", "logqueue-pprd-01" ], "queuetime": 1557256848, "filter": [ "20-syslog-base", "78-category" ], "input": [ "beats", "prodsample" ], "size": 107, "filtertime": 1557256850, "delay": 8, "filterhost": "logfilter-pprd-02.cas-1.cls.vt.edu" }'

2

Can you provide a screen shot. I am not sure what you mean by "all of my structured fields are showing at top level json blobs". Also, could you provide the complete index mapping.

3

I've got things disassembled at the moment, but I'll get the files shortly.

By "all of my structured fields are showing at top level json blobs", I mean my top level fields show a single value containing all their sub fields.

In the above example, my 'meta' field contains this blob: '{ "type": "syslog", "target": { "topic": "logqueue-pprd", "index": "log_syslog-2019.05.07" }, "queuehost": [ "logqueue-prod-03", "logqueue-pprd-01" ], "queuetime": 1557256848, "filter": [ "20-syslog-base", "78-category" ], "input": [ "beats", "prodsample" ], "size": 107, "filtertime": 1557256850, "delay": 8, "filterhost": "logfilter-pprd-02.cas-1.cls.vt.edu" }'

Rather than having a meta.type, meta.target.topic, meta.target.index, meta.queuehost, meta.queuetime, meta.filter, meta.input, meta.size, meta.filtertime, meta.delay, and meta.filterhost fields, it just displays the 'meta' field.

It displays the 'meta' field as an unknown type, but defaults to string, despite having the proper mappings for meta and it's member fields, as shown above.

This has been working for years.

4

Here's the screen shot.

Screenshot_20190509
Screenshot_20190509.png1112×804 65.3 KB

The index pattern (just the meta fields - note that the sub fields show properly):

Screenshot_20190509_1941
Screenshot_20190509_1941.png1443×622 33.8 KB

And the relevant part of the mapping (sorry, the forum won't let me post non-image attachments) mapping:
indent preformatted text by 4 spaces {
"mapping": {
"doc": {
"dynamic_templates": [
{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"doc_values": true,
"index": "true",
"type": "keyword"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "integer"
},
"beat": {
"properties": {
"hostname": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"version": {
"type": "keyword"
}
}
},
"category": {
"type": "keyword"
},
"event": {
"dynamic": "true",
"properties": {
"hostname": {
"type": "keyword"
},
"id": {
"type": "keyword"
},
"level": {
"type": "keyword"
},
"message": {
"type": "keyword"
},
"pid": {
"type": "keyword"
},
"program": {
"type": "keyword"
},
"reason": {
"type": "keyword"
}
}
},
"fields": {
"properties": {
"group": {
"type": "keyword"
},
"host": {
"type": "keyword"
},
"json": {
"type": "boolean"
},
"name": {
"type": "keyword"
},
"tier": {
"type": "keyword"
},
"type": {
"type": "keyword"
}
}
},
"message": {
"type": "text"
},
"meta": {
"dynamic": "true",
"properties": {
"delay": {
"type": "integer"
},
"filter": {
"type": "keyword"
},
"filterhost": {
"type": "keyword"
},
"filtertime": {
"type": "long"
},
"input": {
"type": "keyword"
},
"queuehost": {
"type": "keyword"
},
"queuetime": {
"type": "long"
},
"relayhost": {
"type": "keyword"
},
"size": {
"type": "long"
},
"target": {
"properties": {
"index": {
"type": "keyword"
},
"topic": {
"type": "keyword"
}
}
},
"type": {
"type": "keyword"
}
}
},
"offset": {
"type": "long"
},
"session": {
"properties": {
"duration": {
"type": "keyword"
},
"input_bytes": {
"type": "keyword"
},
"output_bytes": {
"type": "keyword"
},
"start_time": {
"type": "keyword"
}
}
},
"source": {
"type": "keyword"
},
"tags": {
"type": "keyword"
},
"type": {
"type": "keyword"
},
"user": {
"properties": {
"sponsor": {
"type": "keyword"
},
"sponsor_profile": {
"type": "keyword"
}
}
},
"username": {
"type": "keyword"
}
}
}
}
}

5

Any thoughts?

All of my indices are doing this.

If I manually edit the index pattern definition to remove the entry for the 'meta' field, all of the sub fields display properly. Of course, I need to do that for all objects.

This is clearly a Kibana bug (incorrect index pattern creation). Can I get a bug number and ETR, please?

6

Hi,

Have you tried to map the fields "beat", "event", "fields", and "meta" with type: nested like:

"properties": {
  "beat": {
    "type": "nested" 
  }
}

https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

Also, remove the

"dynamic_templates": [ {
    "string_fields": {
        "match": "*",
        "match_mapping_type": "string"
      }
 }]

That is matching all fields, thus making all fields of type string.

7

Thanks for replying.

No, we aren't looking for nested documents. Just structured fields. The elastic mapping is fine. Kibana just isn't building the correct index pattern.

I'll try removing it, but we are getting dynamic numeric fields. (meta.size for example is of type 'long')

8

Sorry, removing the match_mapping_type failed to load. Removing the entire dynamic_templates block put the *.keyword fields back. We went to some effort to fix that. It also didn't fix the base problem of structured fields showing as json. (rewriting the index pattern does, but I'd rather not have to do that every time someone refreshes a pattern.)

9

I tried setting the mapping for those objects to 'nested', rather than 'object' and Kibana didn't even show them anymore.

Anyone have any ideas on how to tell Kibana to treat these as objects, please?

10

https://www.elastic.co/guide/en/kibana/6.7/document-data.html#adding-columns

You can see an example here of what should be happening.

Here's the documentation for the elastic mapping:
https://www.elastic.co/guide/en/elasticsearch/reference/6.7/object.html

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.