Index not refreshed. Query count result not accurate

If anybody can shed a light on the following strange behavior:
My index is updated every minute using a bulk processor with tens of
documents per cycle. After 40 cycles or so, I noticed that my query doesn't
count the new added documents (I'm using a simple HTTP GET req with the
_search endpoint). It is stuck with the same count of documents until I run
an optimize on the index. The optimization seems to "refresh" the index and
then my query count is accurate again.... until the next cycle kicks in and
then I have to optimize the index again to see the latest docs counted in.
I also noticed that if I query a specific document using its _id in the
HTTP GET request, then ES returns me the document details without having to
use optimize.
I currently using v 0.20.6 with a single shard no replica.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Did you somehow disable refresh (-1 value)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 mai 2013 à 21:43, Franck garcia.franck@gmail.com a écrit :

If anybody can shed a light on the following strange behavior:
My index is updated every minute using a bulk processor with tens of documents per cycle. After 40 cycles or so, I noticed that my query doesn't count the new added documents (I'm using a simple HTTP GET req with the _search endpoint). It is stuck with the same count of documents until I run an optimize on the index. The optimization seems to "refresh" the index and then my query count is accurate again.... until the next cycle kicks in and then I have to optimize the index again to see the latest docs counted in.
I also noticed that if I query a specific document using its _id in the HTTP GET request, then ES returns me the document details without having to use optimize.
I currently using v 0.20.6 with a single shard no replica.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

No I left the default value (1s). BTW I saw your DEVOXX presentation on ES
that inspired me a lot! Thanks for that.

On Wed, May 22, 2013 at 4:54 PM, David Pilato david@pilato.fr wrote:

Did you somehow disable refresh (-1 value)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 mai 2013 à 21:43, Franck garcia.franck@gmail.com a écrit :

If anybody can shed a light on the following strange behavior:
My index is updated every minute using a bulk processor with tens of
documents per cycle. After 40 cycles or so, I noticed that my query doesn't
count the new added documents (I'm using a simple HTTP GET req with the
_search endpoint). It is stuck with the same count of documents until I run
an optimize on the index. The optimization seems to "refresh" the index and
then my query count is accurate again.... until the next cycle kicks in and
then I have to optimize the index again to see the latest docs counted in.
I also noticed that if I query a specific document using its _id in the
HTTP GET request, then ES returns me the document details without having to
use optimize.
I currently using v 0.20.6 with a single shard no replica.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/wTbgqF-traM/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Franck

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Out of topic: Cool! Was that this year or last year?
About refresh issue, I don't understand how this can happen. Can you print your elasticsearch settings?

curl -XGET 'http://localhost:9200/_nodes/settings?pretty'

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 23 mai 2013 à 12:40, Franck Garcia garcia.franck@gmail.com a écrit :

No I left the default value (1s). BTW I saw your DEVOXX presentation on ES that inspired me a lot! Thanks for that.

On Wed, May 22, 2013 at 4:54 PM, David Pilato david@pilato.fr wrote:
Did you somehow disable refresh (-1 value)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 mai 2013 à 21:43, Franck garcia.franck@gmail.com a écrit :

If anybody can shed a light on the following strange behavior:
My index is updated every minute using a bulk processor with tens of documents per cycle. After 40 cycles or so, I noticed that my query doesn't count the new added documents (I'm using a simple HTTP GET req with the _search endpoint). It is stuck with the same count of documents until I run an optimize on the index. The optimization seems to "refresh" the index and then my query count is accurate again.... until the next cycle kicks in and then I have to optimize the index again to see the latest docs counted in.
I also noticed that if I query a specific document using its _id in the HTTP GET request, then ES returns me the document details without having to use optimize.
I currently using v 0.20.6 with a single shard no replica.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/wTbgqF-traM/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Franck

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Out of Topic: it was the one from last year, saw it on Parleys, excellent.

Ok this is my settings:
{

"ok" : true,
"cluster_name" : "cluster_prod",
"nodes" : {
"oM9TizIFT3CFN59cfoyJDA" : {
"name" : "AS400 PROD",
"transport_address" : "inet[/172.17.54.81:9300]",
"hostname" : "PROD.mycomp.com",
"version" : "0.20.6",
"http_address" : "inet[/172.17.54.81:9200]",
"settings" : {
"foreground" : "yes",
"logger.prefix" : "",
"path.home" : "/opt/elasticsearch",
"pidfile" : "/home/jboss/es.pid",
"compress.lzf.decoder" : "safe",
"index.number_of_shards" : "1",
"node.name" : "AS400 UNITED",
"index.number_of_replicas" : "0",
"cluster.name" : "cluster_prod",
"name" : "AS400 PROD",
"path.logs" : "/opt/elasticsearch/logs"
}
}
}
}

The particularity is that I'm running ES on an iSeries IBM server. I
don't know if it could have an influence but this is the only none
conventional thing in the setup.

Detail:

IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 OS/400 ppc-32
jvmap3260sr10-20111208 (JIT enabled, AOT enabled)

On Thu, May 23, 2013 at 6:54 AM, David Pilato david@pilato.fr wrote:

Out of topic: Cool! Was that this year or last year?
About refresh issue, I don't understand how this can happen. Can you print
your elasticsearch settings?

curl -XGET 'http://localhost:9200/_nodes/settings?pretty'

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 23 mai 2013 à 12:40, Franck Garcia garcia.franck@gmail.com a écrit :

No I left the default value (1s). BTW I saw your DEVOXX presentation on ES
that inspired me a lot! Thanks for that.

On Wed, May 22, 2013 at 4:54 PM, David Pilato david@pilato.fr wrote:

Did you somehow disable refresh (-1 value)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 mai 2013 à 21:43, Franck garcia.franck@gmail.com a écrit :

If anybody can shed a light on the following strange behavior:
My index is updated every minute using a bulk processor with tens of
documents per cycle. After 40 cycles or so, I noticed that my query doesn't
count the new added documents (I'm using a simple HTTP GET req with the
_search endpoint). It is stuck with the same count of documents until I run
an optimize on the index. The optimization seems to "refresh" the index and
then my query count is accurate again.... until the next cycle kicks in and
then I have to optimize the index again to see the latest docs counted in.
I also noticed that if I query a specific document using its _id in the
HTTP GET request, then ES returns me the document details without having to
use optimize.
I currently using v 0.20.6 with a single shard no replica.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/wTbgqF-traM/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Franck

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Franck

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

After a deep debugging session, I found out that the ES scheduler thread
(for some reason) refuses to schedule any jobs after a while. The Index
Refresher job was among them, hence my problem where I had to manually
optimize the index which itself triggers a manual refresh in order to see
my new documents addition. Dumping the JVM shows that the scheduler thread
was alive but no more jobs were launched.(? and there were no hidden
exception etc.)..
I've discussed with my admin who setup a service package (PTF) on the
machine that seems to have fixed the problem (jobs are scheduled, refresh
is working, cross my fingers).... This is the detail of the working JVM
(For the record):
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 OS/400 ppc-32
jvmap3260sr12-20121025_01 (JIT enabled, AOT enabled)

J9VM - 20121024_126067

JIT - r9_20120914_26057

GC - 20120928_AA)

JCL - 20121014_01

$

On Thu, May 23, 2013 at 7:24 AM, Franck Garcia garcia.franck@gmail.comwrote:

Out of Topic: it was the one from last year, saw it on Parleys, excellent.

Ok this is my settings:
{

"ok" : true,
"cluster_name" : "cluster_prod",
"nodes" : {
"oM9TizIFT3CFN59cfoyJDA" : {
"name" : "AS400 PROD",
"transport_address" : "inet[/172.17.54.81:9300]",
"hostname" : "PROD.mycomp.com",
"version" : "0.20.6",
"http_address" : "inet[/172.17.54.81:9200]",
"settings" : {
"foreground" : "yes",
"logger.prefix" : "",
"path.home" : "/opt/elasticsearch",
"pidfile" : "/home/jboss/es.pid",
"compress.lzf.decoder" : "safe",
"index.number_of_shards" : "1",
"node.name" : "AS400 UNITED",
"index.number_of_replicas" : "0",
"cluster.name" : "cluster_prod",
"name" : "AS400 PROD",
"path.logs" : "/opt/elasticsearch/logs"
}
}
}
}

The particularity is that I'm running ES on an iSeries IBM server. I don't know if it could have an influence but this is the only none conventional thing in the setup.

Detail:

IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 OS/400 ppc-32 jvmap3260sr10-20111208 (JIT enabled, AOT enabled)

On Thu, May 23, 2013 at 6:54 AM, David Pilato david@pilato.fr wrote:

Out of topic: Cool! Was that this year or last year?
About refresh issue, I don't understand how this can happen. Can you
print your elasticsearch settings?

curl -XGET 'http://localhost:9200/_nodes/settings?pretty'

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 23 mai 2013 à 12:40, Franck Garcia garcia.franck@gmail.com a écrit :

No I left the default value (1s). BTW I saw your DEVOXX presentation on
ES that inspired me a lot! Thanks for that.

On Wed, May 22, 2013 at 4:54 PM, David Pilato david@pilato.fr wrote:

Did you somehow disable refresh (-1 value)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 mai 2013 à 21:43, Franck garcia.franck@gmail.com a écrit :

If anybody can shed a light on the following strange behavior:
My index is updated every minute using a bulk processor with tens of
documents per cycle. After 40 cycles or so, I noticed that my query doesn't
count the new added documents (I'm using a simple HTTP GET req with the
_search endpoint). It is stuck with the same count of documents until I run
an optimize on the index. The optimization seems to "refresh" the index and
then my query count is accurate again.... until the next cycle kicks in and
then I have to optimize the index again to see the latest docs counted in.
I also noticed that if I query a specific document using its _id in the
HTTP GET request, then ES returns me the document details without having to
use optimize.
I currently using v 0.20.6 with a single shard no replica.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/wTbgqF-traM/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Franck

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Franck

--
Franck

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I am having what seems like an identical issue: despite running a bulk
insert job with 48 bulk threads active (queuing hovers ~100), I see no docs
on the index. Yet if I do a simple curl call to insert a test document, it
gets reflected immediately. Despite the apparent bulk thread activity,
resource usage on the box is appallingly low, and I see no growth in the
index files. This is a bare metal box with 24 cores and 64GB RAM. My
settings are as follows (note refresh to 30s -- manual refresh/optimize
seems to have no effect)

"indices" : {
"store" : {
"throttle" : {
"type" : "merge",
"max_bytes_per_sec" : "500mb"
}
},
"memory" : {
"index_buffer_size" : "70%"
}
},
"cluster" : {
"name" : "mycluster"
},
"config" : "/config/elasticsearch.yml",
"threadpool" : {
"search" : {
"type" : "fixed",
"size" : "20",
"queue_size" : "1000"
},
"bulk" : {
"type" : "fixed",
"size" : "48",
"queue_size" : "1000"
},
"index" : {
"type" : "fixed",
"queue_size" : "1000",
"size" : "20"
}
},
"index" : {
"store" : {
"compress" : {
"stored" : "true"
},
"type" : "mmapfs"
},
"translog" : {
"flush_threshold_ops" : "25000"
},
"refresh_interval" : "30s",
"merge" : {
"policy" : {
"max_merge_size" : "500mb"
}
}
},
"name" : "Vindicator"
}
}
}
}

On Thursday, May 23, 2013 6:54:19 AM UTC-4, David Pilato wrote:

Out of topic: Cool! Was that this year or last year?
About refresh issue, I don't understand how this can happen. Can you print
your elasticsearch settings?

curl -XGET 'http://localhost:9200/_nodes/settings?pretty'

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 23 mai 2013 à 12:40, Franck Garcia <garcia...@gmail.com <javascript:>>
a écrit :

No I left the default value (1s). BTW I saw your DEVOXX presentation on ES
that inspired me a lot! Thanks for that.

On Wed, May 22, 2013 at 4:54 PM, David Pilato <da...@pilato.fr
<javascript:>> wrote:

Did you somehow disable refresh (-1 value)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 mai 2013 à 21:43, Franck <garcia...@gmail.com <javascript:>> a
écrit :

If anybody can shed a light on the following strange behavior:
My index is updated every minute using a bulk processor with tens of
documents per cycle. After 40 cycles or so, I noticed that my query doesn't
count the new added documents (I'm using a simple HTTP GET req with the
_search endpoint). It is stuck with the same count of documents until I run
an optimize on the index. The optimization seems to "refresh" the index and
then my query count is accurate again.... until the next cycle kicks in and
then I have to optimize the index again to see the latest docs counted in.
I also noticed that if I query a specific document using its _id in the
HTTP GET request, then ES returns me the document details without having to
use optimize.
I currently using v 0.20.6 with a single shard no replica.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/wTbgqF-traM/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Franck

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0776710a-a432-4953-a76f-a7ab7f5c4c48%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.