Storing Data in ElasticSearch


I have noticed by default all my data is stored under _source in ElasticSearch.
I have some questions, I would like to clarify as the article below is not clear to me:

  1. In Kibana I see that for a particular index, I have all my fields available as expected, I can search for them and I see they show up as fields under Management_>Index Patterns->Index.
    But when I go to Discover tab and select a particular message, and select the JSON output, I see that all my data is stored under _source, and not under fields. Does this mean my data is stored only in _source and the fields are not stored as actual fields?

  2. In JSON output, when I am looking at a message from a particular index that contains fields that are dates, these seem to be added to fields as well as _source. Is this behavior expected, that all date fields are stored as fields automatically?

3)If I remove _source from my index, will all my data be stored under fields automatically?

(Simon Willnauer) #2

_source contains the source document that you sent to elasticsearch. We store that as a whole for various reasons. I recommend doing this!!!! (you can opt out but that has a bunch of implications). Now you can also mark certain fields as stored for instance if you don't request _source as part of your result that might reduce the amout of data that we need to fetch for you.
I think for dates you see it markes as stored since it's organized as a doc-value (column oriented storage) so no, it's not stored by default. I think it's just a naming issue and you are fine. we store all your data by default. to mark a field stored is an optimization for certain cases.

hope that helps


Thanks for the info, very helpful. So the "available fields" I see in Kibana are actually based on what is in _source, that is good to know. I will check out which fields are worth storing as "stored", for optimization reasons.

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.