wwalker
(Walker)
January 10, 2022, 10:00pm
1
I use Logstash to process our logs and organize relevant fields into objects so that they display together in Kibana.
log.source
log.format
log.time
However, after upgrading, I THINK to 7.16, they're now displaying as an object under the root field log
in JSON.
{
"format": "",
"source": "",
"time": ""
}
When did this happen? Is there a setting I can change or do I have to completely re-write my Logstash pipelines to put them in separate fields?
flash1293
(Joe Reuter)
January 11, 2022, 8:01am
2
Could you share the mapping of your index?
wwalker
(Walker)
January 11, 2022, 3:05pm
3
"log": {
"properties": {
"format": {
"type": "keyword"
},
"time": {
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
flash1293
(Joe Reuter)
January 11, 2022, 3:06pm
4
This is should work fine, can you share a screenshot of the problem?
wwalker
(Walker)
January 11, 2022, 3:23pm
5
The fields in my previous example aren't real data, but below is the same problem with real data.
"action": {
"properties": {
"detail": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"endpoint": {
"type": "keyword"
},
"method": {
"type": "keyword"
}
}
},
Kibana Screenshot
Tomo_M
(Tomohiro Mitani)
January 11, 2022, 4:19pm
6
What happen when you reload the kibana page?
The mappings are the same across all the indices contained in the index pattern?
wwalker
(Walker)
January 11, 2022, 4:21pm
7
Same behavior reloading the page.
I've deleted all the related indices to eliminate the possibility of data type mismatch.
warkolm
(Mark Walkom)
January 11, 2022, 9:54pm
8
For Elasticsearch, this;
log.source
log.format
log.time
Is the same as this;
{
"log":
{
"format": "",
"source": "",
"time": ""
}
}
So Kibana is just displaying them differently, but to Elasticsearch there's no change.
flash1293
(Joe Reuter)
January 12, 2022, 8:49am
9
wwalker:
"action": {
"properties": {
"detail": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"endpoint": {
"type": "keyword"
},
"method": {
"type": "keyword"
}
}
}
Are you sure all indices in your index pattern have this mapping? I'm guessing there is a conflict and some index has another mapping. I tested the following on 7.16:
PUT actionindex
{
"mappings": {
"properties": {
"action": {
"properties": {
"detail": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"endpoint": {
"type": "keyword"
},
"method": {
"type": "keyword"
}
}
}
}
}
}
POST actionindex/_doc
{
"action": {
"detail": "a test",
"endpoint": "the_endpoint",
"method": "the_method"
}
}
This is how it looks like in discover (all fields are recognized correctly):
1 Like
wwalker
(Walker)
January 12, 2022, 10:44pm
10
I stopped Filebeat on the remote server that pushes the logs, deleted all indices, the index template, the index pattern from Kibana/Elasticsearch, and restarted the pipeline to push a fresh copy of the template into Elasticsearch. Still getting the same appearance in Kibana.
JSON snippet of event
{
"_source": {
"timestamp": "2021-12-30t11:11:39.722296",
"action": {
"detail": "",
"endpoint": "/validate/check",
"method": "post"
},
Table view of event
Index Template Configuration
"action": {
"type": "object",
"properties": {
"detail": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"endpoint": {
"type": "keyword"
},
"method": {
"type": "keyword"
}
}
},
wwalker
(Walker)
January 12, 2022, 10:47pm
11
I just discovered the cause of the issue, which may be a regression bug. When the option "Discover fields from source" is enabled, it causes the unwanted behavior. Once I disabled this setting, the fields started displaying properly. However, I enabled this option in the past because not all fields were being displayed in my indices in the past. Hoping that problem has been fixed...
matw
(Matthias Wilhelm)
January 17, 2022, 3:25pm
12
2 Likes
system
(system)
Closed
February 14, 2022, 3:26pm
13
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.