Real-time vs. Near real-time searching

dkullmann · July 21, 2012, 3:14pm

Hello,

I'm writing unit tests for an ElasticSearch datasource, however, I am
getting mixed results. The problem is that a match_all query isn't finding
records that I submitted, however, when I run the commands by hand using
CURL in the same order the unit test does I am able to find the records.

I believe that perhaps the index isn't refreshed, so, I started running the
"refresh" api command after submitting records, however, this didn't work
either. Here is my list of commands - it would be helpful if anyone had any
suggestions on how to make sure these commands worked even if they were run
in immediate succession.

Commands the unit test runs:

curl -XGET 'http://localhost:9200/test_index/_mapping'

curl -XDELETE 'http://localhost:9200/test_index/test_models'

curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d '{}'

curl -XPUT 'http://localhost:9200/test_index/test_models/_mapping' -d
'{"test_models":{"properties":{"TestModel":{"properties":{"id":{"type":"string","index":"not_analyzed"},"string":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd
HH:mm:ss"},"modified":{"type":"date","format":"yyyy-MM-dd
HH:mm:ss"}},"type":"object"}}}}'

curl -XPOST 'http://localhost:9200/test_index/test_models/_bulk' -d
'{"index":{"_index":"test_index","_type":"test_models","_id":"test-model"}}
{"TestModel":{"id":"test-model","string":"Analyzed for
terms","created":"2012-01-01 00:00:00","modified":"2012-02-01 00:00:00"}}
'

curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d '{}'

curl -XGET 'http://localhost:9200/test_index/_mapping'

curl -XGET 'http://localhost:9200/test_index/test_models/_search' -d
'{"query":{"match_all":{}},"size":10}'

Thanks,
DK

dkullmann · July 21, 2012, 3:15pm

Note:

Adding "sleep 3" in my unit test between each command also worked,
supporting the idea that the index isn't up to date.

Thanks!
DK

On Saturday, July 21, 2012 11:14:10 AM UTC-4, David Kullmann wrote:

Hello,

I'm writing unit tests for an Elasticsearch datasource, however, I am
getting mixed results. The problem is that a match_all query isn't finding
records that I submitted, however, when I run the commands by hand using
CURL in the same order the unit test does I am able to find the records.

I believe that perhaps the index isn't refreshed, so, I started running
the "refresh" api command after submitting records, however, this didn't
work either. Here is my list of commands - it would be helpful if anyone
had any suggestions on how to make sure these commands worked even if they
were run in immediate succession.

Commands the unit test runs:

curl -XGET 'http://localhost:9200/test_index/_mapping'

curl -XDELETE 'http://localhost:9200/test_index/test_models'

curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d
'{}'

curl -XPUT 'http://localhost:9200/test_index/test_models/_mapping' -d
'{"test_models":{"properties":{"TestModel":{"properties":{"id":{"type":"string","index":"not_analyzed"},"string":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd
HH:mm:ss"},"modified":{"type":"date","format":"yyyy-MM-dd
HH:mm:ss"}},"type":"object"}}}}'

curl -XPOST 'http://localhost:9200/test_index/test_models/_bulk' -d
'{"index":{"_index":"test_index","_type":"test_models","_id":"test-model"}}
{"TestModel":{"id":"test-model","string":"Analyzed for
terms","created":"2012-01-01 00:00:00","modified":"2012-02-01 00:00:00"}}
'

curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d
'{}'

curl -XGET 'http://localhost:9200/test_index/_mapping'

curl -XGET 'http://localhost:9200/test_index/test_models/_search' -d
'{"query":{"match_all":{}},"size":10}'

Thanks,
DK

dkullmann · July 21, 2012, 3:23pm

The problem is with the _refresh command.

You can't refresh a type, only an index. I changed the refresh command to:

curl -XPOST 'http://localhost:9200/test_index/_refresh http://localhost:9200/test_index/test_models/_refresh
'

And it is now fixed!

Thanks,
DK

On Saturday, July 21, 2012 11:14:10 AM UTC-4, David Kullmann wrote:

Hello,

I'm writing unit tests for an Elasticsearch datasource, however, I am
getting mixed results. The problem is that a match_all query isn't finding
records that I submitted, however, when I run the commands by hand using
CURL in the same order the unit test does I am able to find the records.

I believe that perhaps the index isn't refreshed, so, I started running
the "refresh" api command after submitting records, however, this didn't
work either. Here is my list of commands - it would be helpful if anyone
had any suggestions on how to make sure these commands worked even if they
were run in immediate succession.

Commands the unit test runs:

curl -XGET 'http://localhost:9200/test_index/_mapping'

curl -XDELETE 'http://localhost:9200/test_index/test_models'

curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d
'{}'

curl -XPUT 'http://localhost:9200/test_index/test_models/_mapping' -d
'{"test_models":{"properties":{"TestModel":{"properties":{"id":{"type":"string","index":"not_analyzed"},"string":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd
HH:mm:ss"},"modified":{"type":"date","format":"yyyy-MM-dd
HH:mm:ss"}},"type":"object"}}}}'

curl -XPOST 'http://localhost:9200/test_index/test_models/_bulk' -d
'{"index":{"_index":"test_index","_type":"test_models","_id":"test-model"}}
{"TestModel":{"id":"test-model","string":"Analyzed for
terms","created":"2012-01-01 00:00:00","modified":"2012-02-01 00:00:00"}}
'

curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d
'{}'

curl -XGET 'http://localhost:9200/test_index/_mapping'

curl -XGET 'http://localhost:9200/test_index/test_models/_search' -d
'{"query":{"match_all":{}},"size":10}'

Thanks,
DK