First post here and hopefully an easy question. I was asked: if Elastic is accepting streaming data, does that file have to be closed before it can be fed into Lucene to be indexed?
I'm in my second week researching Elasticsearch and this question doesn't entirely make sense. Here's my answer (quoted from the documentation):
Accepting streaming data would the job of the Bulk API. If the document is changed and saved the timestamp/date would be differ, therefore the document will be replaced with the latest version and a higher version number. The latest version will be indexed when the Update action is called.
When using update action _retry_on_conflict can be used as field in the action itself (not in the extra payload line), to specify how many times an update should be retried in the case of a version conflict.
The update action payload, supports the following options: doc (partial document), upsert,doc_as_upsert, script, params (for script), lang (for script) and fields.
I would like to know if there is a better answer to the question.
I think a first step to a better answer would be to gather more data on what the actual use case is behind "if Elastic is accepting streaming data...".
(Also, as a general hint: If a question you get doesn't entirely make sense, don't be afraid to ask for clarifications until it makes sense. Often there's small things missing that are obvious to the one asking the question but way less obvious to the one trying to come up with an answer.)
Glad I could help. Good luck with your Elasticsearch project. Would love to hear more about what you are actually doing with the project if you care to share.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
