Is the document a file that is ‘closed’?


#1

Hello All,

First post here and hopefully an easy question. I was asked: if Elastic is accepting streaming data, does that file have to be closed before it can be fed into Lucene to be indexed?

I'm in my second week researching Elasticsearch and this question doesn't entirely make sense. Here's my answer (quoted from the documentation):

Accepting streaming data would the job of the Bulk API. If the document is changed and saved the timestamp/date would be differ, therefore the document will be replaced with the latest version and a higher version number. The latest version will be indexed when the Update action is called.

When using update action _retry_on_conflict can be used as field in the action itself (not in the extra payload line), to specify how many times an update should be retried in the case of a version conflict.
The update action payload, supports the following options: doc (partial document), upsert,doc_as_upsert, script, params (for script), lang (for script) and fields.

I would like to know if there is a better answer to the question.


(Isabel Drost-Fromm) #2

I think a first step to a better answer would be to gather more data on what the actual use case is behind "if Elastic is accepting streaming data...".

If we are talking about log data to be streamed to Elasticsearch, you might want to look at https://www.elastic.co/products/beats/filebeat or https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html on ways to stream lines from logfiles to Elasticsearch as they come in.

(Also, as a general hint: If a question you get doesn't entirely make sense, don't be afraid to ask for clarifications until it makes sense. Often there's small things missing that are obvious to the one asking the question but way less obvious to the one trying to come up with an answer.)

Hope this helps,
Isabel


#3

That's great information and advice. It helped me direct my searches better.

Thanks Isabel.


(Isabel Drost-Fromm) #4

Glad I could help. Good luck with your Elasticsearch project. Would love to hear more about what you are actually doing with the project if you care to share.

Isabel


#5

I'm not doing anything specific yet, just researching and learning as much as I can. I do have a lot of log data that would be nice to visualize.


(Isabel Drost-Fromm) #6

If you end up doing the visualization with Elasticsearch (and friends like Beats, Logstash, Kibana) let us know :slight_smile:


(system) #7