Index size impact on search performance?

Hi folks, I have an index storing lots of time serial data. The data are put
into index by :

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '

{

"val": 23,

"timestamp": 1406822400

}'

And the only thing I search in this index is histogram facet in a very short
time range, like "recent 5 min". I found that the performance was pretty
good at first. But when the index get bigger, the performance dropped to
unacceptable. I found the IO maybe the bottleneck by checking the result of
iostat.

My question is, even I only facet in a very short time range, why the size
of index has so big impact on the performance of such query? Do I have to
use daily index, just like logstash?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Well. I guess it depends on your query. What does it look like?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 04:14, "Wang Yong" cnwangyong@gmail.com a écrit :

Hi folks, I have an index storing lots of time serial data. The data are put into index by :

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '
{
"val": 23,
"timestamp": 1406822400
}'

And the only thing I search in this index is histogram facet in a very short time range, like “recent 5 min”. I found that the performance was pretty good at first. But when the index get bigger, the performance dropped to unacceptable. I found the IO maybe the bottleneck by checking the result of iostat.

My question is, even I only facet in a very short time range, why the size of index has so big impact on the performance of such query? Do I have to use daily index, just like logstash?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

If you're using time series data then it makes sense to use time based
indexes.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 1 August 2014 12:43, David Pilato david@pilato.fr wrote:

Well. I guess it depends on your query. What does it look like?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 04:14, "Wang Yong" cnwangyong@gmail.com a écrit :

Hi folks, I have an index storing lots of time serial data. The data are
put into index by :

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '

{

"val": 23,

"timestamp": 1406822400

}'

And the only thing I search in this index is histogram facet in a very
short time range, like “recent 5 min”. I found that the performance was
pretty good at first. But when the index get bigger, the performance
dropped to unacceptable. I found the IO maybe the bottleneck by checking
the result of iostat.

My question is, even I only facet in a very short time range, why the size
of index has so big impact on the performance of such query? Do I have to
use daily index, just like logstash?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com
https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YEg4XVNGAN%2BL8P6QHKjFn_9JTqkJE7TtMHh8fPqZhFxg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thank you david :blush:

most of my query looks like:

{

"filtered": {

  "query": {

    "match_all": {}

  },

  "filter": {

               "range": {

            "timestamp": {

              "from": 1403567280,

              "to": 1403567340,

              "include_lower": true,

              "include_upper": false

            }

          }

    }

},

"facets" : {

"val": {

  "statistical": {


    "field": "val"



  }


}

}

}

Sent from Surface

From: David Pilato
Sent: ‎Friday‎, ‎August‎ ‎1‎, ‎2014 ‎10‎:‎43‎ ‎AM
To: elasticsearch@googlegroups.com

Well. I guess it depends on your query. What does it look like?

--

David :wink:

Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 04:14, "Wang Yong" cnwangyong@gmail.com a écrit :

Hi folks, I have an index storing lots of time serial data. The data are put into index by :

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '

{

"val": 23,

"timestamp": 1406822400

}'

And the only thing I search in this index is histogram facet in a very short time range, like “recent 5 min”. I found that the performance was pretty good at first. But when the index get bigger, the performance dropped to unacceptable. I found the IO maybe the bottleneck by checking the result of iostat.

My question is, even I only facet in a very short time range, why the size of index has so big impact on the performance of such query? Do I have to use daily index, just like logstash?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53db1703.a567440a.7a19.36b5%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

Thank you Mark,

in your word “time based indexes”, is that means create one index every day? if I index my data in this way, i have to specify which index to search when create query in my java client, based on the “from” and “to”.

Sent from Surface

From: Mark Walkom
Sent: ‎Friday‎, ‎August‎ ‎1‎, ‎2014 ‎11‎:‎02‎ ‎AM
To: elasticsearch@googlegroups.com

If you're using time series data then it makes sense to use time based indexes.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 1 August 2014 12:43, David Pilato david@pilato.fr wrote:

Well. I guess it depends on your query. What does it look like?

--

David :wink:

Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 04:14, "Wang Yong" cnwangyong@gmail.com a écrit :

Hi folks, I have an index storing lots of time serial data. The data are put into index by :

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '

{

"val": 23,

"timestamp": 1406822400

}'

And the only thing I search in this index is histogram facet in a very short time range, like “recent 5 min”. I found that the performance was pretty good at first. But when the index get bigger, the performance dropped to unacceptable. I found the IO maybe the bottleneck by checking the result of iostat.

My question is, even I only facet in a very short time range, why the size of index has so big impact on the performance of such query? Do I have to use daily index, just like logstash?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YEg4XVNGAN%2BL8P6QHKjFn_9JTqkJE7TtMHh8fPqZhFxg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53db1a2f.8d5b460a.30e5.41d5%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

There is an issue when using long instead of strings.
Cache is not used.

Try with something like "from":"2014-06-01","to":"2015-01-01" for example.
Or explicitly set cache to true.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 06:25, Wang Yong cnwangyong@gmail.com a écrit :

Thank you david :blush:
most of my query looks like:

{
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"range": {
"timestamp": {
"from": 1403567280,
"to": 1403567340,
"include_lower": true,
"include_upper": false
}
}
}
},

"facets" : {

"val": {
"statistical": {
"field": "val"
}
}
}
}

Sent from Surface

From: David Pilato
Sent: ‎Friday‎, ‎August‎ ‎1‎, ‎2014 ‎10‎:‎43‎ ‎AM
To: elasticsearch@googlegroups.com

Well. I guess it depends on your query. What does it look like?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 04:14, "Wang Yong" cnwangyong@gmail.com a écrit :

Hi folks, I have an index storing lots of time serial data. The data are put into index by :

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '
{
"val": 23,
"timestamp": 1406822400
}'

And the only thing I search in this index is histogram facet in a very short time range, like “recent 5 min”. I found that the performance was pretty good at first. But when the index get bigger, the performance dropped to unacceptable. I found the IO maybe the bottleneck by checking the result of iostat.

My question is, even I only facet in a very short time range, why the size of index has so big impact on the performance of such query? Do I have to use daily index, just like logstash?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53db1703.a567440a.7a19.36b5%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/93F415A4-AF53-4EAF-999E-07C5CB1BE6C5%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

You can create one every hour if you want, daily is good.
You may have to deal with that in the code however it makes management a
lot easier.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 1 August 2014 14:35, Wang Yong cnwangyong@gmail.com wrote:

Thank you Mark,
in your word “time based indexes”, is that means create one index every
day? if I index my data in this way, i have to specify which index to
search when create query in my java client, based on the “from” and “to”.

Sent from Surface

From: Mark Walkom markw@campaignmonitor.com
Sent: ‎Friday‎, ‎August‎ ‎1‎, ‎2014 ‎11‎:‎02‎ ‎AM
To: elasticsearch@googlegroups.com

If you're using time series data then it makes sense to use time based
indexes.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 1 August 2014 12:43, David Pilato david@pilato.fr wrote:

Well. I guess it depends on your query. What does it look like?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 04:14, "Wang Yong" cnwangyong@gmail.com a écrit :

Hi folks, I have an index storing lots of time serial data. The data are
put into index by :

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '

{

"val": 23,

"timestamp": 1406822400

}'

And the only thing I search in this index is histogram facet in a very
short time range, like “recent 5 min”. I found that the performance was
pretty good at first. But when the index get bigger, the performance
dropped to unacceptable. I found the IO maybe the bottleneck by checking
the result of iostat.

My question is, even I only facet in a very short time range, why the
size of index has so big impact on the performance of such query? Do I have
to use daily index, just like logstash?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com
https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e$57938420$06ba8c60$%40gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YEg4XVNGAN%2BL8P6QHKjFn_9JTqkJE7TtMHh8fPqZhFxg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEM624YEg4XVNGAN%2BL8P6QHKjFn_9JTqkJE7TtMHh8fPqZhFxg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/53db1a2f.8d5b460a.30e5.41d5%40mx.google.com
https://groups.google.com/d/msgid/elasticsearch/53db1a2f.8d5b460a.30e5.41d5%40mx.google.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZoiAOdaFoRaCXjUyKPe%3D1ah%2B_q79wA%2BDy8bkEDVL%2ByRw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I just found this great blog about range filter:

from this blog, there are some words about how elasticsearch handle range filter:

“For Elasticsearch to answer this query it had to load all the datetime fields from disk to memory for the query, compute the range, and then throw away the data it didn’t need.”

I think that's why there is so many disk IO when I facet on a very small time range. And also, someone suggest to use normal filter range with execute option set to datafield. will elasticsearch act exactly the same way as the words above? if that's true, maybe daily index will be my only choice.

Alan

Sent from Surface

From: David Pilato
Sent: ‎Friday‎, ‎August‎ ‎1‎, ‎2014 ‎2‎:‎16‎ ‎PM
To: elasticsearch@googlegroups.com

There is an issue when using long instead of strings.

Cache is not used.

Try with something like "from":"2014-06-01","to":"2015-01-01" for example.

Or explicitly set cache to true.

--

David :wink:

Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 06:25, Wang Yong cnwangyong@gmail.com a écrit :

Thank you david :blush:

most of my query looks like:

{

"filtered": {

  "query": {

    "match_all": {}

  },

  "filter": {

               "range": {

            "timestamp": {

              "from": 1403567280,

              "to": 1403567340,

              "include_lower": true,

              "include_upper": false

            }

          }

    }

},

"facets" : {

"val": {

  "statistical": {


    "field": "val"



  }


}

}

}

Sent from Surface

From: David Pilato
Sent: ‎Friday‎, ‎August‎ ‎1‎, ‎2014 ‎10‎:‎43‎ ‎AM
To: elasticsearch@googlegroups.com

Well. I guess it depends on your query. What does it look like?

--

David :wink:

Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 04:14, "Wang Yong" cnwangyong@gmail.com a écrit :

Hi folks, I have an index storing lots of time serial data. The data are put into index by :

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '

{

"val": 23,

"timestamp": 1406822400

}'

And the only thing I search in this index is histogram facet in a very short time range, like “recent 5 min”. I found that the performance was pretty good at first. But when the index get bigger, the performance dropped to unacceptable. I found the IO maybe the bottleneck by checking the result of iostat.

My question is, even I only facet in a very short time range, why the size of index has so big impact on the performance of such query? Do I have to use daily index, just like logstash?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53db1703.a567440a.7a19.36b5%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/93F415A4-AF53-4EAF-999E-07C5CB1BE6C5%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53db6dbc.05f4440a.1bfb.622a%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

But did you try what I wrote?

You still see a lot of IOs on the second call?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 12:30, Wang Yong cnwangyong@gmail.com a écrit :

I just found this great blog about range filter:
https://blog.hipchat.com/2014/06/24/elasticsearch-at-hipchat-10x-faster-queries/

from this blog, there are some words about how elasticsearch handle range filter:

“For Elasticsearch to answer this query it had to load all the datetime fields from disk to memory for the query, compute the range, and then throw away the data it didn’t need.”

I think that's why there is so many disk IO when I facet on a very small time range. And also, someone suggest to use normal filter range with execute option set to datafield. will elasticsearch act exactly the same way as the words above? if that's true, maybe daily index will be my only choice.

Alan
Sent from Surface

From: David Pilato
Sent: ‎Friday‎, ‎August‎ ‎1‎, ‎2014 ‎2‎:‎16‎ ‎PM
To: elasticsearch@googlegroups.com

There is an issue when using long instead of strings.
Cache is not used.

Try with something like "from":"2014-06-01","to":"2015-01-01" for example.
Or explicitly set cache to true.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 06:25, Wang Yong cnwangyong@gmail.com a écrit :

Thank you david :blush:
most of my query looks like:

{
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"range": {
"timestamp": {
"from": 1403567280,
"to": 1403567340,
"include_lower": true,
"include_upper": false
}
}
}
},

"facets" : {

"val": {
"statistical": {
"field": "val"
}
}
}
}

Sent from Surface

From: David Pilato
Sent: ‎Friday‎, ‎August‎ ‎1‎, ‎2014 ‎10‎:‎43‎ ‎AM
To: elasticsearch@googlegroups.com

Well. I guess it depends on your query. What does it look like?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 août 2014 à 04:14, "Wang Yong" cnwangyong@gmail.com a écrit :

Hi folks, I have an index storing lots of time serial data. The data are put into index by :

curl -XPUT 'localhost:9200/testindex/action1/1?pretty' -d '
{
"val": 23,
"timestamp": 1406822400
}'

And the only thing I search in this index is histogram facet in a very short time range, like “recent 5 min”. I found that the performance was pretty good at first. But when the index get bigger, the performance dropped to unacceptable. I found the IO maybe the bottleneck by checking the result of iostat.

My question is, even I only facet in a very short time range, why the size of index has so big impact on the performance of such query? Do I have to use daily index, just like logstash?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00fd01cfad2e%2457938420%2406ba8c60%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0D3FCF70-13F5-4E92-8407-47E907736F79%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53db1703.a567440a.7a19.36b5%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/93F415A4-AF53-4EAF-999E-07C5CB1BE6C5%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53db6dbc.05f4440a.1bfb.622a%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/210F2325-C896-40AC-B919-39B655600C28%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.