Index getting delayed?

Elastic Stack Elasticsearch
1

I'm writing a javascript program that will inject test data records into Elasticsearch, several thousand at a time.

It's using a hot sleep to make the program slow down so the timestamp on each record is at least a few ms different.

What I appear to be seeing is that none of the index calls are made/passed to elastic until the program finishes, at which point all of them are passed in. Almost like something is buffering them until my script has finished.

Is there any way to get a client.index() request processed immediately? I'd like to be able to look at the data in kibana while it's still being generated so I can simulate near real time processing.

Mik

2

If I understand correctly, you'd like to get the data immediately available just after you call index operation, right?

You need to pass the refresh=wait_for to your index/bulk request or call explicitly the refresh endpoint.

Note that you should not do that in production but it's fine doing that in tests.

3

Thanks dude. With a combination of your answer and restructuring my code to use timeouts it's working a treat.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.