ES Python API behaves strangely when used for search right after ingestion

Elastic Stack Elasticsearch
1

I run a search right after ingesting documents through my Python client and get an empty response even though in Kibana I can see that everything is ingested correctly. Here's a simple demo code:

from elasticsearch import Elasticsearch, helpers

es_host = 'localhost'
es_port = '9200'

es = Elasticsearch([{'host': es_host, 'port': es_port}])

bulk_data = []

for i in range(5):
    obj = {}

    obj['_index'] = 'test_ind'
    obj['_type'] = 'my_type'
    obj['a'] = i

    bulk_data.append(obj)

bulk_data = iter(bulk_data)
helpers.bulk(es, bulk_data)

body = {
  "query": {
    "match_all": {}
  }
}

print es.search('test_ind', body)

The above code does the ingestion but the search returns nothing. However, when I run my search in a separate code, it works as expected.

from elasticsearch import Elasticsearch

es_host = 'localhost'
es_port = '9200'

es = Elasticsearch([{'host': es_host, 'port': es_port}])

body = {
  "query": {
    "match_all": {}
  }
}

print es.search('test_ind', body=body)

Is there a conceptual reason as to why the first approach won't work or is this just a newbie goof on my part?

PS. Not sure if this matters but I'm using Python 2.7 and Elastic 5.5 (I know!!).

2

After you do helpers.bulk(es, bulk_data), you either need to wait a full second (settings.refresh_interval) docs, send in an explicit refresh with the bulk request, or use the Refresh API to have those docs returned from a search so soon after you index them.

3

Hey thanks for your reply!!!
I had never looked into this but now it makes a lot of sense.
I tried to follow your pointers and added

time.sleep(2)
es.indices.refresh('test_ind')
time.sleep(2)

after the ingestion step but it's not making a difference. I'll figure it out with try and error though.
I remember I had similar issues with Rollover too ...

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.