Does kibana need _source? can we just store file&byte offset and load doc at query time?


(Dieter Plaetinck) #1

Hi,
we have a lot of docs, and keeping all docs in ES would consume too much
space, but we keep the data in text files, and would like to use ES for
indexing and
kibana for analytics.
kibana needs docs with _source / raw body, right? especially the table
widget?

since our workload is very heavy on writes and very light on reads, could
we instead of _source just store a filename and byte offset pointer?
and have ES just read the document from a given file & byte offset when it
needs them (assuming file is available on ES node). or
i could write a network service that receives a filename & byte offset and
returns the corresponding document from a file.

that way the raw doc wouldn't have to be in ES, but ES could still return
the full docs at query time, and hand them off to kibana?
is that possible?

thanks,
Dieter

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0c074659-42a8-41c1-8d36-f9003162b9cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #2