Fields in Table Do Not Match Fields in JSON

[ddodge (Dustyn Dodge)]

Under the Discover tab in Kibana I can see entries in the Table listing that say "No cached mapping for this field. Refresh field list from the Management > Index Patters page"

Conversely in Discover, I can see fields in the JSON that do not show up in the Table view.

I've clicked Management > Index Patterns then I go to my index "logstash-beats-*" and click refresh (multiple times). This does not help. In the management view, it recognizes the JSON fields (which are the ones I want). For some reason Discover (and in my Dashboards) do not show the correct JSON fields.

Is there something that I'm missing?

[matw (Matthias Wilhelm)]

Hi and welcome to our community! We tried to reproduce your issue, so far we were not successful. When you refresh the index pattern, the warning you've mentioned, is still there right?

Which version of kibana are you using? Could you provide us a sample of your data and an export of the index mapping? That would be great.

When you select your index pattern in management, the id of your pattern is part of the URL, e.g.

http://localhost:5601/oan/app/kibana#/management/kibana/index_patterns/ff959d40-b880-11e8-a6d9-e546fe2bba5f?_g=()&_a=(tab:indexedFields)

With this id you can export the pattern in Console of Dev Tools by:

GET .kibana/_doc/index-pattern:ff959d40-b880-11e8-a6d9-e546fe2bba5

This would be a great help to solve this problem, thank you very much

[ddodge (Dustyn Dodge)]

Sure thing, I'm using Kibana 6.7.2. I've attached a screenshot of the pattern. It's pretty long. Let me know if you need to see more. Thank you!

[ddodge (Dustyn Dodge)]

Sorry, I forgot to mention that yes, the warning is still there when I refresh.

[matw (Matthias Wilhelm)]

Thanks, could you paste the textual output here, and maybe 1-2 datasets you're using? thx a lot!

[ddodge (Dustyn Dodge)]

{
"_index" : ".kibana_1",
"_type" : "_doc",
"_id" : "index-pattern:AWBLHZaBRuBloj96jvrD",
"_version" : 15,
"_seq_no" : 2532,
"_primary_term" : 12,
"found" : true,
"_source" : {
"index-pattern" : {
"title" : ":logstash-beats-",
"timeFieldName" : "@timestamp",
"notExpandable" : true,
"fields" : """[{"name":"@timestamp","type":"date","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"@version","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"_id","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":false},{"name":"_index","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":false},{"name":"_score","type":"number","count":0,"scripted":false,"searchable":false,"aggregatable":false,"readFromDocValues":false},{"name":"_source","type":"_source","count":0,"scripted":false,"searchable":false,"aggregatable":false,"readFromDocValues":false},{"name":"_type","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":false},{"name":"aa","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"aa.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"ack","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"ack.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"action","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"action.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"activity_id","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"additional_info","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"additional_info.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"age","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"age.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"alert","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"alert.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"alert_level","type":"number","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"alert_level.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"analyzer","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"analyzer.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"answers","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"answers.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"apache2.access.agent","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"apache2.access.body_sent.bytes","type":"number","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"apache2.access.geoip.city_name","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"apache2.access.geoip.continent_name","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"apache2.access.geoip.country_iso_code","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"apache2.access.geoip.location","type":"geo_point","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"apache2.access.geoip.region_name","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},

[ddodge (Dustyn Dodge)]

Here is what shows in the JSON tab for a windows log (with some data masking). Notice that the event_id field, for example, shows up in the JSON file.

"winlog": {
"api": "wineventlog",
"event_data": {
"SubjectLogonId": "0x3e7",
"SubjectUserSid": "S-1-5-18",
"PrivilegeList": "SeAssignPrimaryTokenPrivilege\n\t\t\tSeTcbPrivilege\n\t\t\tSeSecurityPrivilege\n\t\t\tSeTakeOwnershipPrivilege\n\t\t\tSeLoadDriverPrivilege\n\t\t\tSeBackupPrivilege\n\t\t\tSeRestorePrivilege\n\t\t\tSeDebugPrivilege\n\t\t\tSeAuditPrivilege\n\t\t\tSeSystemEnvironmentPrivilege\n\t\t\tSeImpersonatePrivilege\n\t\t\tSeDelegateSessionUserImpersonatePrivilege",
"SubjectDomainName": "NT AUTHORITY",
"SubjectUserName": "SYSTEM"
},
"activity_id": "{E65959B9-40BA-0000-F959-59E6BA40D501}",
"task": "Special Logon",
"provider_name": "Microsoft-Windows-Security-Auditing",
"channel": "Security",
"event_id": 4672,
"computer_name": "/DATA MASKED/",
"provider_guid": "{54849625-5478-4994-A5BA-3E3B0328C30D}",
"record_id": 159180,
"process": {
"thread": {
"id": 824
},
"pid": 776
},
"keywords": [
"Audit Success"
],
"opcode": "Info"
},

[ddodge (Dustyn Dodge)]

Now notice that the field event_id doesn't show up in the Table view. It shows up as event.code which is not compatible with any of my dashboards. This is one example. The same thing applies for computer_name, process_id, user.name, etc.

[ddodge (Dustyn Dodge)]

Console view for the event_id field:

......
{"name":"event_id","type":"number","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"event_timestamp","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},{"name":"event_timestamp.keyword","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":true,"readFromDocValues":true},{"name":"event_type","type":"string","count":0,"scripted":false,"searchable":true,"aggregatable":false,"readFromDocValues":false},
........

event.code field in console view is not found.

[matw (Matthias Wilhelm)]

thanks a lot, I will try to reproduce your problem

[matw (Matthias Wilhelm)]

There's an odd thing about your index pattern, according to your screenshot, it's *:logstash-beats-*. How was this index pattern created? Was it migrated? Could you try adding another index pattern named logstash-beats-*? I can't manually create an index pattern with such a name.

thx a lot!

[Christian_Dahlqvist]

Isn’t that the index pattern format when using cross-cluster search?

[matw (Matthias Wilhelm)]

@Christian_Dahlqvist yes, you're right. thank you very much, I wasn't aware of this

https://www.elastic.co/guide/en/kibana/current/management-cross-cluster-search.html

So maybe there's an issue with refreshing indices for cross cluster searches ? I'll continue investigating in this direction

[matw (Matthias Wilhelm)]

Sorry for the long time since my last answer. So I couldn't reproduce it, but I have an idea what might cause the problem. It would be interesting what's the origin of the data. If e.g. beats were upgraded, your indices could now indices use different mappings.

So maybe the source of your ingested data are different beat versions? Could you provide some details about this? Furthermore it would be interesting, since you're using cross cluster search, how many clusters du you have?

thx!