In Kibana I see that for a particular index, I have all my fields available as expected, I can search for them and I see they show up as fields under Management_>Index Patterns->Index.
But when I go to Discover tab and select a particular message, and select the JSON output, I see that all my data is stored under _source, and not under fields. Does this mean my data is stored only in _source and the fields are not stored as actual fields?
In JSON output, when I am looking at a message from a particular index that contains fields that are dates, these seem to be added to fields as well as _source. Is this behavior expected, that all date fields are stored as fields automatically?
3)If I remove _source from my index, will all my data be stored under fields automatically?
_source contains the source document that you sent to elasticsearch. We store that as a whole for various reasons. I recommend doing this!!!! (you can opt out but that has a bunch of implications). Now you can also mark certain fields as stored for instance if you don't request _source as part of your result that might reduce the amout of data that we need to fetch for you.
I think for dates you see it markes as stored since it's organized as a doc-value (column oriented storage) so no, it's not stored by default. I think it's just a naming issue and you are fine. we store all your data by default. to mark a field stored is an optimization for certain cases.
Thanks for the info, very helpful. So the "available fields" I see in Kibana are actually based on what is in _source, that is good to know. I will check out which fields are worth storing as "stored", for optimization reasons.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.