# Very slow ElasticSearch Index
**URL:** https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906
**Category:** Elasticsearch
**Created:** [December 17, 2013, 7:55am UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906 "2013-12-17T07:55:59Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@lekkie](https://discuss.elastic.co/u/lekkie)
#### Post date: [December 17, 2013, 7:55am UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906/1 "2013-12-17T07:55:59Z")
Hi guys,
We index a document at \*\*\* milli on ElasticSearch, which I think is too
slow especially for the amount of resources we have setup. We will like to
index at the rate of 500tps. Each document weighs between 20K and 30K.
How many indexes are advisable to be done at once (assuming we can afford
to send multiple http index request to the server at once)? I understand
bulk indexing is a preferred approach, for a 30K document how much can be
bulked at once? How many http bulk request (supposing I am using a
multi-threaded http client ot make requests) is advisable to make?
I will appreciate suggestions and how to index this document as fast as
possible. We have two nodes set up, the config below is for one out of the
two:
Shards: 5
Replica: 1
"nodes" : {
"T5l5mvIdQsW3je7WmSPOcg" : {
"name" : "SEARCH-01",
"version" : "0.90.7",
"attributes" : {
"rack\_id" : "prod",
"max\_local\_storage\_nodes" : "1"
},
"settings" : {
"node.rack\_id" : "prod",
"action.disable\_delete\_all\_indices" : "true",
"cloud.node.auto\_attributes" : "true",
"indices.ttl.interval" : "90d",
"node.max\_local\_storage\_nodes" : "1",
"bootstrap.mlockall" : "true",
"index.mapper.dynamic" : "true",
"cluster.routing.allocation.awareness.attributes" : "rack\_id",
"discovery.zen.minimum\_master\_nodes" : "3",
"gateway.expected\_nodes" : "1",
"discovery.zen.ping.unicast.hosts" :
"172.25.15.170,172.25.15.172,172.46.1.170,172.46.1.172",
"discovery.zen.ping.multicast.enabled" : "false",
"action.auto\_create\_index" : "true"
},
"os" : {
"refresh\_interval" : 1000,
"available\_processors" : 8,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 2600,
"total\_cores" : 8,
"total\_sockets" : 2,
"cores\_per\_socket" : 4,
"cache\_size" : "20kb",
"cache\_size\_in\_bytes" : 20480
},
"mem" : {
"total" : "17.5gb",
"total\_in\_bytes" : 18836545536
},
"swap" : {
"total" : "5.8gb",
"total\_in\_bytes" : 6274670592
}
},
"process" : {
"refresh\_interval" : 1000,
"id" : 3459,
"max\_file\_descriptors" : 64000
},
"jvm" : {
"pid" : 3459,
"version" : "1.7.0\_45",
"vm\_name" : "OpenJDK 64-Bit Server VM",
"vm\_version" : "24.45-b08",
"vm\_vendor" : "Oracle Corporation",
"start\_time" : 1386953353018,
"mem" : {
"heap\_init" : "10.5gb",
"heap\_init\_in\_bytes" : 11301552128,
"heap\_max" : "10.4gb",
"heap\_max\_in\_bytes" : 11231821824,
"non\_heap\_init" : "23.1mb",
"non\_heap\_init\_in\_bytes" : 24313856,
"non\_heap\_max" : "214mb",
"non\_heap\_max\_in\_bytes" : 224395264,
"direct\_max" : "10.4gb",
"direct\_max\_in\_bytes" : 11231821824
}
},
"thread\_pool" : {
"generic" : {
"type" : "cached",
"keep\_alive" : "30s"
},
"index" : {
"type" : "fixed",
"min" : 8,
"max" : 8,
"queue\_size" : "200"
},
"get" : {
"type" : "fixed",
"min" : 8,
"max" : 8,
"queue\_size" : "1k"
},
"snapshot" : {
"type" : "scaling",
"min" : 1,
"max" : 4,
"keep\_alive" : "5m"
},
"merge" : {
"type" : "scaling",
"min" : 1,
"max" : 4,
"keep\_alive" : "5m"
},
"suggest" : {
"type" : "fixed",
"min" : 8,
"max" : 8,
"queue\_size" : "1k"
},
"bulk" : {
"type" : "fixed",
"min" : 8,
"max" : 8,
"queue\_size" : "50"
},
"optimize" : {
"type" : "fixed",
"min" : 1,
"max" : 1
},
"warmer" : {
"type" : "scaling",
"min" : 1,
"max" : 4,
"keep\_alive" : "5m"
},
"flush" : {
"type" : "scaling",
"min" : 1,
"max" : 4,
"keep\_alive" : "5m"
},
"search" : {
"type" : "fixed",
"min" : 24,
"max" : 24,
"queue\_size" : "1k"
},
"percolate" : {
"type" : "fixed",
"min" : 8,
"max" : 8,
"queue\_size" : "1k"
},
"management" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep\_alive" : "5m"
},
"refresh" : {
"type" : "scaling",
"min" : 1,
"max" : 4,
"keep\_alive" : "5m"
}
},
"network" : {
"refresh\_interval" : 5000
},
"http" : {
"max\_content\_length" : "100mb",
"max\_content\_length\_in\_bytes" : 104857600
},
"plugins" : []
}
}
The Mapping is dynamically created because we create types daily and it
looks like:
{
"consumers-20131216": {
"properties": {
"requestData": {
"type": "string"
},
"requestTimestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"responseData": {
"type": "string"
},
"responseTimestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"sequenceId": {
"type": "long"
},
"service": {
"type": "string"
},
"systemResponseCode": {
"type": "string"
},
"systemResponseMessage": {
"type": "string"
},
"transactionComponentTypeId": {
"type": "long"
},
"transactionLogId": {
"type": "long"
},
"user": {
"type": "string"
}
}
}
}
Regards.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 17, 2013, 8:12am UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906/2 "2013-12-17T08:12:52Z")
First tip would be to drop OpenJDK and move to Oracle, you'll get a lot
better performance.
Bulk depends a lot on your setup and document size etc, but upwards of 5K
is generally towards the upper limit.
It might also be worth removing indices.ttl.interval and just using a
script to delete old indices as TTL searches can use a fair bit of
resources.
You also mentioned you have 2 nodes, but there are a lot more IPs listed in
the discovery hosts, is that intentional? Same for minimum\_master\_nodes
being 3.
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 17 December 2013 18:55, lekkie omotayo [lekkie.aydot@gmail.com](mailto:lekkie.aydot@gmail.com) wrote:
> Hi guys,
>
> We index a document at \*\*\* milli on Elasticsearch, which I think is too
> slow especially for the amount of resources we have setup. We will like to
> index at the rate of 500tps. Each document weighs between 20K and 30K.
> How many indexes are advisable to be done at once (assuming we can afford
> to send multiple http index request to the server at once)? I understand
> bulk indexing is a preferred approach, for a 30K document how much can be
> bulked at once? How many http bulk request (supposing I am using a
> multi-threaded http client ot make requests) is advisable to make?
>
> I will appreciate suggestions and how to index this document as fast as
> possible. We have two nodes set up, the config below is for one out of the
> two:
>
> Shards: 5
> Replica: 1
>
> "nodes" : {
> "T5l5mvIdQsW3je7WmSPOcg" : {
> "name" : "SEARCH-01",
> "version" : "0.90.7",
> "attributes" : {
> "rack\_id" : "prod",
> "max\_local\_storage\_nodes" : "1"
> },
> "settings" : {
> "node.rack\_id" : "prod",
> "action.disable\_delete\_all\_indices" : "true",
> "cloud.node.auto\_attributes" : "true",
> "indices.ttl.interval" : "90d",
> "node.max\_local\_storage\_nodes" : "1",
> "bootstrap.mlockall" : "true",
> "index.mapper.dynamic" : "true",
> "cluster.routing.allocation.awareness.attributes" : "rack\_id",
> "discovery.zen.minimum\_master\_nodes" : "3",
> "gateway.expected\_nodes" : "1",
> "discovery.zen.ping.unicast.hosts" :
> "172.25.15.170,172.25.15.172,172.46.1.170,172.46.1.172",
> "discovery.zen.ping.multicast.enabled" : "false",
> "action.auto\_create\_index" : "true"
> },
> "os" : {
> "refresh\_interval" : 1000,
> "available\_processors" : 8,
> "cpu" : {
> "vendor" : "Intel",
> "model" : "Xeon",
> "mhz" : 2600,
> "total\_cores" : 8,
> "total\_sockets" : 2,
> "cores\_per\_socket" : 4,
> "cache\_size" : "20kb",
> "cache\_size\_in\_bytes" : 20480
> },
> "mem" : {
> "total" : "17.5gb",
> "total\_in\_bytes" : 18836545536
> },
> "swap" : {
> "total" : "5.8gb",
> "total\_in\_bytes" : 6274670592
> }
> },
> "process" : {
> "refresh\_interval" : 1000,
> "id" : 3459,
> "max\_file\_descriptors" : 64000
> },
> "jvm" : {
> "pid" : 3459,
> "version" : "1.7.0\_45",
> "vm\_name" : "OpenJDK 64-Bit Server VM",
> "vm\_version" : "24.45-b08",
> "vm\_vendor" : "Oracle Corporation",
> "start\_time" : 1386953353018,
> "mem" : {
> "heap\_init" : "10.5gb",
> "heap\_init\_in\_bytes" : 11301552128,
> "heap\_max" : "10.4gb",
> "heap\_max\_in\_bytes" : 11231821824,
> "non\_heap\_init" : "23.1mb",
> "non\_heap\_init\_in\_bytes" : 24313856,
> "non\_heap\_max" : "214mb",
> "non\_heap\_max\_in\_bytes" : 224395264,
> "direct\_max" : "10.4gb",
> "direct\_max\_in\_bytes" : 11231821824
> }
> },
> "thread\_pool" : {
> "generic" : {
> "type" : "cached",
> "keep\_alive" : "30s"
> },
> "index" : {
> "type" : "fixed",
> "min" : 8,
> "max" : 8,
> "queue\_size" : "200"
> },
> "get" : {
> "type" : "fixed",
> "min" : 8,
> "max" : 8,
> "queue\_size" : "1k"
> },
> "snapshot" : {
> "type" : "scaling",
> "min" : 1,
> "max" : 4,
> "keep\_alive" : "5m"
> },
> "merge" : {
> "type" : "scaling",
> "min" : 1,
> "max" : 4,
> "keep\_alive" : "5m"
> },
> "suggest" : {
> "type" : "fixed",
> "min" : 8,
> "max" : 8,
> "queue\_size" : "1k"
> },
> "bulk" : {
> "type" : "fixed",
> "min" : 8,
> "max" : 8,
> "queue\_size" : "50"
> },
> "optimize" : {
> "type" : "fixed",
> "min" : 1,
> "max" : 1
> },
> "warmer" : {
> "type" : "scaling",
> "min" : 1,
> "max" : 4,
> "keep\_alive" : "5m"
> },
> "flush" : {
> "type" : "scaling",
> "min" : 1,
> "max" : 4,
> "keep\_alive" : "5m"
> },
> "search" : {
> "type" : "fixed",
> "min" : 24,
> "max" : 24,
> "queue\_size" : "1k"
> },
> "percolate" : {
> "type" : "fixed",
> "min" : 8,
> "max" : 8,
> "queue\_size" : "1k"
> },
> "management" : {
> "type" : "scaling",
> "min" : 1,
> "max" : 5,
> "keep\_alive" : "5m"
> },
> "refresh" : {
> "type" : "scaling",
> "min" : 1,
> "max" : 4,
> "keep\_alive" : "5m"
> }
> },
> "network" : {
> "refresh\_interval" : 5000
> },
> "http" : {
> "max\_content\_length" : "100mb",
> "max\_content\_length\_in\_bytes" : 104857600
> },
> "plugins" :
> }
> }
>
> The Mapping is dynamically created because we create types daily and it
> looks like:
> {
> "consumers-20131216": {
> "properties": {
> "requestData": {
> "type": "string"
> },
> "requestTimestamp": {
> "type": "date",
> "format": "dateOptionalTime"
> },
> "responseData": {
> "type": "string"
> },
> "responseTimestamp": {
> "type": "date",
> "format": "dateOptionalTime"
> },
> "sequenceId": {
> "type": "long"
> },
> "service": {
> "type": "string"
> },
> "systemResponseCode": {
> "type": "string"
> },
> "systemResponseMessage": {
> "type": "string"
> },
> "transactionComponentTypeId": {
> "type": "long"
> },
> "transactionLogId": {
> "type": "long"
> },
> "user": {
> "type": "string"
> }
> }
> }
> }
>
> Regards.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%40googlegroups.com)
> .
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YfDT4WToSXE8S9Vm1Mkp0hrn45bZuH3cZWrmzKVQssCg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YfDT4WToSXE8S9Vm1Mkp0hrn45bZuH3cZWrmzKVQssCg%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@lekkie](https://discuss.elastic.co/u/lekkie)
#### Post date: [December 17, 2013, 1:44pm UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906/3 "2013-12-17T13:44:17Z")
Thanks for the insight.
> > > > First tip would be to drop OpenJDK and move to Oracle, you'll get a
> > > > lot better performance.
So I changed to Oracle JDK and latency dropped from 4000millisecond to
around 2500millisecond.
.....................
> > > > It might also be worth removing indices.ttl.interval and just using a
> > > > script to delete old indices as TTL searches can use a fair bit of
> > > > resources.
We also dropped indices.ttl.interval and it further dropped to
1500milliseconds.
> > > > You also mentioned you have 2 nodes, but there are a lot more IPs
> > > > listed in the discovery hosts, is that intentional? Same for minimum\_master\_nodes
> > > > being 3.
> > > > Yes, the other 2 nodes are DR nodes. So we basically have 4 nodes but 2 are
> > > > for disaster recovery. And the discovery.zen.minimum\_master\_nodes was
> > > > calculated based on the n/2 + 1, where n was 4.
One other thing to note, every request is an upsert.
What we have now is 1500milliseconds per upsert. This is still very high.
We are looking at doing sub-zero millisecond or 10s of millisecond for bulk
upload. Can this be achieved or it is a pipe dream?
On Tuesday, 17 December 2013 09:12:52 UTC+1, Mark Walkom wrote:
> First tip would be to drop OpenJDK and move to Oracle, you'll get a lot
> better performance.
> Bulk depends a lot on your setup and document size etc, but upwards of 5K
> is generally towards the upper limit.
> It might also be worth removing indices.ttl.interval and just using a
> script to delete old indices as TTL searches can use a fair bit of
> resources.
>
> You also mentioned you have 2 nodes, but there are a lot more IPs listed
> in the discovery hosts, is that intentional? Same for minimum\_master\_nodes
> being 3.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
>
> On 17 December 2013 18:55, lekkie omotayo \<[lekkie...@gmail.com](mailto:lekkie...@gmail.com)\
>
> > wrote:
>
> > Hi guys,
> >
> > We index a document at \*\*\* milli on Elasticsearch, which I think is too
> > slow especially for the amount of resources we have setup. We will like to
> > index at the rate of 500tps. Each document weighs between 20K and 30K.
> > How many indexes are advisable to be done at once (assuming we can afford
> > to send multiple http index request to the server at once)? I understand
> > bulk indexing is a preferred approach, for a 30K document how much can be
> > bulked at once? How many http bulk request (supposing I am using a
> > multi-threaded http client ot make requests) is advisable to make?
> >
> > I will appreciate suggestions and how to index this document as fast as
> > possible. We have two nodes set up, the config below is for one out of the
> > two:
> >
> > Shards: 5
> > Replica: 1
> >
> > "nodes" : {
> > "T5l5mvIdQsW3je7WmSPOcg" : {
> > "name" : "SEARCH-01",
> > "version" : "0.90.7",
> > "attributes" : {
> > "rack\_id" : "prod",
> > "max\_local\_storage\_nodes" : "1"
> > },
> > "settings" : {
> > "node.rack\_id" : "prod",
> > "action.disable\_delete\_all\_indices" : "true",
> > "cloud.node.auto\_attributes" : "true",
> > "indices.ttl.interval" : "90d",
> > "node.max\_local\_storage\_nodes" : "1",
> > "bootstrap.mlockall" : "true",
> > "index.mapper.dynamic" : "true",
> > "cluster.routing.allocation.awareness.attributes" : "rack\_id",
> > "discovery.zen.minimum\_master\_nodes" : "3",
> > "gateway.expected\_nodes" : "1",
> > "discovery.zen.ping.unicast.hosts" :
> > "172.25.15.170,172.25.15.172,172.46.1.170,172.46.1.172",
> > "discovery.zen.ping.multicast.enabled" : "false",
> > "action.auto\_create\_index" : "true"
> > },
> > "os" : {
> > "refresh\_interval" : 1000,
> > "available\_processors" : 8,
> > "cpu" : {
> > "vendor" : "Intel",
> > "model" : "Xeon",
> > "mhz" : 2600,
> > "total\_cores" : 8,
> > "total\_sockets" : 2,
> > "cores\_per\_socket" : 4,
> > "cache\_size" : "20kb",
> > "cache\_size\_in\_bytes" : 20480
> > },
> > "mem" : {
> > "total" : "17.5gb",
> > "total\_in\_bytes" : 18836545536
> > },
> > "swap" : {
> > "total" : "5.8gb",
> > "total\_in\_bytes" : 6274670592
> > }
> > },
> > "process" : {
> > "refresh\_interval" : 1000,
> > "id" : 3459,
> > "max\_file\_descriptors" : 64000
> > },
> > "jvm" : {
> > "pid" : 3459,
> > "version" : "1.7.0\_45",
> > "vm\_name" : "OpenJDK 64-Bit Server VM",
> > "vm\_version" : "24.45-b08",
> > "vm\_vendor" : "Oracle Corporation",
> > "start\_time" : 1386953353018,
> > "mem" : {
> > "heap\_init" : "10.5gb",
> > "heap\_init\_in\_bytes" : 11301552128,
> > "heap\_max" : "10.4gb",
> > "heap\_max\_in\_bytes" : 11231821824,
> > "non\_heap\_init" : "23.1mb",
> > "non\_heap\_init\_in\_bytes" : 24313856,
> > "non\_heap\_max" : "214mb",
> > "non\_heap\_max\_in\_bytes" : 224395264,
> > "direct\_max" : "10.4gb",
> > "direct\_max\_in\_bytes" : 11231821824
> > }
> > },
> > "thread\_pool" : {
> > "generic" : {
> > "type" : "cached",
> > "keep\_alive" : "30s"
> > },
> > "index" : {
> > "type" : "fixed",
> > "min" : 8,
> > "max" : 8,
> > "queue\_size" : "200"
> > },
> > "get" : {
> > "type" : "fixed",
> > "min" : 8,
> > "max" : 8,
> > "queue\_size" : "1k"
> > },
> > "snapshot" : {
> > "type" : "scaling",
> > "min" : 1,
> > "max" : 4,
> > "keep\_alive" : "5m"
> > },
> > "merge" : {
> > "type" : "scaling",
> > "min" : 1,
> > "max" : 4,
> > "keep\_alive" : "5m"
> > },
> > "suggest" : {
> > "type" : "fixed",
> > "min" : 8,
> > "max" : 8,
> > "queue\_size" : "1k"
> > },
> > "bulk" : {
> > "type" : "fixed",
> > "min" : 8,
> > "max" : 8,
> > "queue\_size" : "50"
> > },
> > "optimize" : {
> > "type" : "fixed",
> > "min" : 1,
> > "max" : 1
> > },
> > "warmer" : {
> > "type" : "scaling",
> > "min" : 1,
> > "max" : 4,
> > "keep\_alive" : "5m"
> > },
> > "flush" : {
> > "type" : "scaling",
> > "min" : 1,
> > "max" : 4,
> > "keep\_alive" : "5m"
> > },
> > "search" : {
> > "type" : "fixed",
> > "min" : 24,
> > "max" : 24,
> > "queue\_size" : "1k"
> > },
> > "percolate" : {
> > "type" : "fixed",
> > "min" : 8,
> > "max" : 8,
> > "queue\_size" : "1k"
> > },
> > "management" : {
> > "type" : "scaling",
> > "min" : 1,
> > "max" : 5,
> > "keep\_alive" : "5m"
> > },
> > "refresh" : {
> > "type" : "scaling",
> > "min" : 1,
> > "max" : 4,
> > "keep\_alive" : "5m"
> > }
> > },
> > "network" : {
> > "refresh\_interval" : 5000
> > },
> > "http" : {
> > "max\_content\_length" : "100mb",
> > "max\_content\_length\_in\_bytes" : 104857600
> > },
> > "plugins" :
> > }
> > }
> >
> > The Mapping is dynamically created because we create types daily and it
> > looks like:
> > {
> > "consumers-20131216": {
> > "properties": {
> > "requestData": {
> > "type": "string"
> > },
> > "requestTimestamp": {
> > "type": "date",
> > "format": "dateOptionalTime"
> > },
> > "responseData": {
> > "type": "string"
> > },
> > "responseTimestamp": {
> > "type": "date",
> > "format": "dateOptionalTime"
> > },
> > "sequenceId": {
> > "type": "long"
> > },
> > "service": {
> > "type": "string"
> > },
> > "systemResponseCode": {
> > "type": "string"
> > },
> > "systemResponseMessage": {
> > "type": "string"
> > },
> > "transactionComponentTypeId": {
> > "type": "long"
> > },
> > "transactionLogId": {
> > "type": "long"
> > },
> > "user": {
> > "type": "string"
> > }
> > }
> > }
> > }
> >
> > Regards.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%40googlegroups.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f0be2cb9-9832-4de3-a3ab-ab0843662ae4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0be2cb9-9832-4de3-a3ab-ab0843662ae4%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 17, 2013, 9:33pm UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906/4 "2013-12-17T21:33:07Z")
ES will only go as fast as the slowest node. With that in mind, are your
"DR" nodes the same capacity?
I also notice they are in different subnets, does that imply they are in
different datacenters?
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 18 December 2013 00:44, lekkie omotayo [lekkie.aydot@gmail.com](mailto:lekkie.aydot@gmail.com) wrote:
> Thanks for the insight.
>
> > > > > First tip would be to drop OpenJDK and move to Oracle, you'll get a
> > > > > lot better performance.
>
> So I changed to Oracle JDK and latency dropped from 4000millisecond to
> around 2500millisecond.
>
> .....................
>
> > > > > It might also be worth removing indices.ttl.interval and just using a
> > > > > script to delete old indices as TTL searches can use a fair bit of
> > > > > resources.
>
> We also dropped indices.ttl.interval and it further dropped to
> 1500milliseconds.
>
> > > > > You also mentioned you have 2 nodes, but there are a lot more IPs
> > > > > listed in the discovery hosts, is that intentional? Same for
> > > > > minimum\_master\_nodes being 3.
> > > > > Yes, the other 2 nodes are DR nodes. So we basically have 4 nodes but 2
> > > > > are for disaster recovery. And the discovery.zen.minimum\_master\_nodes was
> > > > > calculated based on the n/2 + 1, where n was 4.
>
> One other thing to note, every request is an upsert.
>
> What we have now is 1500milliseconds per upsert. This is still very high.
> We are looking at doing sub-zero millisecond or 10s of millisecond for bulk
> upload. Can this be achieved or it is a pipe dream?
>
> On Tuesday, 17 December 2013 09:12:52 UTC+1, Mark Walkom wrote:
>
> > First tip would be to drop OpenJDK and move to Oracle, you'll get a lot
> > better performance.
> > Bulk depends a lot on your setup and document size etc, but upwards of 5K
> > is generally towards the upper limit.
> > It might also be worth removing indices.ttl.interval and just using a
> > script to delete old indices as TTL searches can use a fair bit of
> > resources.
> >
> > You also mentioned you have 2 nodes, but there are a lot more IPs listed
> > in the discovery hosts, is that intentional? Same for minimum\_master\_nodes
> > being 3.
> >
> > Regards,
> > Mark Walkom
> >
> > Infrastructure Engineer
> > Campaign Monitor
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> >
> > On 17 December 2013 18:55, lekkie omotayo [lekkie...@gmail.com](mailto:lekkie...@gmail.com) wrote:
> >
> > > Hi guys,
> > >
> > > We index a document at \*\*\* milli on Elasticsearch, which I think is too
> > > slow especially for the amount of resources we have setup. We will like to
> > > index at the rate of 500tps. Each document weighs between 20K and 30K.
> > > How many indexes are advisable to be done at once (assuming we can
> > > afford to send multiple http index request to the server at once)? I
> > > understand bulk indexing is a preferred approach, for a 30K document how
> > > much can be bulked at once? How many http bulk request (supposing I am
> > > using a multi-threaded http client ot make requests) is advisable to make?
> > >
> > > I will appreciate suggestions and how to index this document as fast as
> > > possible. We have two nodes set up, the config below is for one out of the
> > > two:
> > >
> > > Shards: 5
> > > Replica: 1
> > >
> > > "nodes" : {
> > > "T5l5mvIdQsW3je7WmSPOcg" : {
> > > "name" : "SEARCH-01",
> > > "version" : "0.90.7",
> > > "attributes" : {
> > > "rack\_id" : "prod",
> > > "max\_local\_storage\_nodes" : "1"
> > > },
> > > "settings" : {
> > > "node.rack\_id" : "prod",
> > > "action.disable\_delete\_all\_indices" : "true",
> > > "cloud.node.auto\_attributes" : "true",
> > > "indices.ttl.interval" : "90d",
> > > "node.max\_local\_storage\_nodes" : "1",
> > > "bootstrap.mlockall" : "true",
> > > "index.mapper.dynamic" : "true",
> > > "cluster.routing.allocation.awareness.attributes" : "rack\_id",
> > > "discovery.zen.minimum\_master\_nodes" : "3",
> > > "gateway.expected\_nodes" : "1",
> > > "discovery.zen.ping.unicast.hosts" :
> > > "172.25.15.170,172.25.15.172,172.46.1.170,172.46.1.172",
> > > "discovery.zen.ping.multicast.enabled" : "false",
> > > "action.auto\_create\_index" : "true"
> > > },
> > > "os" : {
> > > "refresh\_interval" : 1000,
> > > "available\_processors" : 8,
> > > "cpu" : {
> > > "vendor" : "Intel",
> > > "model" : "Xeon",
> > > "mhz" : 2600,
> > > "total\_cores" : 8,
> > > "total\_sockets" : 2,
> > > "cores\_per\_socket" : 4,
> > > "cache\_size" : "20kb",
> > > "cache\_size\_in\_bytes" : 20480
> > > },
> > > "mem" : {
> > > "total" : "17.5gb",
> > > "total\_in\_bytes" : 18836545536
> > > },
> > > "swap" : {
> > > "total" : "5.8gb",
> > > "total\_in\_bytes" : 6274670592
> > > }
> > > },
> > > "process" : {
> > > "refresh\_interval" : 1000,
> > > "id" : 3459,
> > > "max\_file\_descriptors" : 64000
> > > },
> > > "jvm" : {
> > > "pid" : 3459,
> > > "version" : "1.7.0\_45",
> > > "vm\_name" : "OpenJDK 64-Bit Server VM",
> > > "vm\_version" : "24.45-b08",
> > > "vm\_vendor" : "Oracle Corporation",
> > > "start\_time" : 1386953353018,
> > > "mem" : {
> > > "heap\_init" : "10.5gb",
> > > "heap\_init\_in\_bytes" : 11301552128,
> > > "heap\_max" : "10.4gb",
> > > "heap\_max\_in\_bytes" : 11231821824,
> > > "non\_heap\_init" : "23.1mb",
> > > "non\_heap\_init\_in\_bytes" : 24313856,
> > > "non\_heap\_max" : "214mb",
> > > "non\_heap\_max\_in\_bytes" : 224395264,
> > > "direct\_max" : "10.4gb",
> > > "direct\_max\_in\_bytes" : 11231821824
> > > }
> > > },
> > > "thread\_pool" : {
> > > "generic" : {
> > > "type" : "cached",
> > > "keep\_alive" : "30s"
> > > },
> > > "index" : {
> > > "type" : "fixed",
> > > "min" : 8,
> > > "max" : 8,
> > > "queue\_size" : "200"
> > > },
> > > "get" : {
> > > "type" : "fixed",
> > > "min" : 8,
> > > "max" : 8,
> > > "queue\_size" : "1k"
> > > },
> > > "snapshot" : {
> > > "type" : "scaling",
> > > "min" : 1,
> > > "max" : 4,
> > > "keep\_alive" : "5m"
> > > },
> > > "merge" : {
> > > "type" : "scaling",
> > > "min" : 1,
> > > "max" : 4,
> > > "keep\_alive" : "5m"
> > > },
> > > "suggest" : {
> > > "type" : "fixed",
> > > "min" : 8,
> > > "max" : 8,
> > > "queue\_size" : "1k"
> > > },
> > > "bulk" : {
> > > "type" : "fixed",
> > > "min" : 8,
> > > "max" : 8,
> > > "queue\_size" : "50"
> > > },
> > > "optimize" : {
> > > "type" : "fixed",
> > > "min" : 1,
> > > "max" : 1
> > > },
> > > "warmer" : {
> > > "type" : "scaling",
> > > "min" : 1,
> > > "max" : 4,
> > > "keep\_alive" : "5m"
> > > },
> > > "flush" : {
> > > "type" : "scaling",
> > > "min" : 1,
> > > "max" : 4,
> > > "keep\_alive" : "5m"
> > > },
> > > "search" : {
> > > "type" : "fixed",
> > > "min" : 24,
> > > "max" : 24,
> > > "queue\_size" : "1k"
> > > },
> > > "percolate" : {
> > > "type" : "fixed",
> > > "min" : 8,
> > > "max" : 8,
> > > "queue\_size" : "1k"
> > > },
> > > "management" : {
> > > "type" : "scaling",
> > > "min" : 1,
> > > "max" : 5,
> > > "keep\_alive" : "5m"
> > > },
> > > "refresh" : {
> > > "type" : "scaling",
> > > "min" : 1,
> > > "max" : 4,
> > > "keep\_alive" : "5m"
> > > }
> > > },
> > > "network" : {
> > > "refresh\_interval" : 5000
> > > },
> > > "http" : {
> > > "max\_content\_length" : "100mb",
> > > "max\_content\_length\_in\_bytes" : 104857600
> > > },
> > > "plugins" :
> > > }
> > > }
> > >
> > > The Mapping is dynamically created because we create types daily and it
> > > looks like:
> > > {
> > > "consumers-20131216": {
> > > "properties": {
> > > "requestData": {
> > > "type": "string"
> > > },
> > > "requestTimestamp": {
> > > "type": "date",
> > > "format": "dateOptionalTime"
> > > },
> > > "responseData": {
> > > "type": "string"
> > > },
> > > "responseTimestamp": {
> > > "type": "date",
> > > "format": "dateOptionalTime"
> > > },
> > > "sequenceId": {
> > > "type": "long"
> > > },
> > > "service": {
> > > "type": "string"
> > > },
> > > "systemResponseCode": {
> > > "type": "string"
> > > },
> > > "systemResponseMessage": {
> > > "type": "string"
> > > },
> > > "transactionComponentTypeId": {
> > > "type": "long"
> > > },
> > > "transactionLogId": {
> > > "type": "long"
> > > },
> > > "user": {
> > > "type": "string"
> > > }
> > > }
> > > }
> > > }
> > >
> > > Regards.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%
> > > [40googlegroups.com](http://40googlegroups.com).
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/f0be2cb9-9832-4de3-a3ab-ab0843662ae4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0be2cb9-9832-4de3-a3ab-ab0843662ae4%40googlegroups.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ZShZP7WLgCAFHuhFdSGgJF41P3qWVwhB0LCo8hoGNGHA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ZShZP7WLgCAFHuhFdSGgJF41P3qWVwhB0LCo8hoGNGHA%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@lekkie](https://discuss.elastic.co/u/lekkie)
#### Post date: [December 18, 2013, 7:12am UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906/5 "2013-12-18T07:12:30Z")
Yes they all have the same capacity.
Yes, they are in different data centers (off-site).
On Tuesday, 17 December 2013 22:33:07 UTC+1, Mark Walkom wrote:
> ES will only go as fast as the slowest node. With that in mind, are your
> "DR" nodes the same capacity?
>
> I also notice they are in different subnets, does that imply they are in
> different datacenters?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
>
> On 18 December 2013 00:44, lekkie omotayo \<[lekkie...@gmail.com](mailto:lekkie...@gmail.com)\
>
> > wrote:
>
> > Thanks for the insight.
> >
> > > > > > First tip would be to drop OpenJDK and move to Oracle, you'll get a
> > > > > > lot better performance.
> >
> > So I changed to Oracle JDK and latency dropped from 4000millisecond to
> > around 2500millisecond.
> >
> > .....................
> >
> > > > > > It might also be worth removing indices.ttl.interval and just using
> > > > > > a script to delete old indices as TTL searches can use a fair bit of
> > > > > > resources.
> >
> > We also dropped indices.ttl.interval and it further dropped to
> > 1500milliseconds.
> >
> > > > > > You also mentioned you have 2 nodes, but there are a lot more IPs
> > > > > > listed in the discovery hosts, is that intentional? Same for
> > > > > > minimum\_master\_nodes being 3.
> > > > > > Yes, the other 2 nodes are DR nodes. So we basically have 4 nodes but 2
> > > > > > are for disaster recovery. And the discovery.zen.minimum\_master\_nodes
> > > > > > was calculated based on the n/2 + 1, where n was 4.
> >
> > One other thing to note, every request is an upsert.
> >
> > What we have now is 1500milliseconds per upsert. This is still very high.
> > We are looking at doing sub-zero millisecond or 10s of millisecond for bulk
> > upload. Can this be achieved or it is a pipe dream?
> >
> > On Tuesday, 17 December 2013 09:12:52 UTC+1, Mark Walkom wrote:
> >
> > > First tip would be to drop OpenJDK and move to Oracle, you'll get a lot
> > > better performance.
> > > Bulk depends a lot on your setup and document size etc, but upwards of
> > > 5K is generally towards the upper limit.
> > > It might also be worth removing indices.ttl.interval and just using a
> > > script to delete old indices as TTL searches can use a fair bit of
> > > resources.
> > >
> > > You also mentioned you have 2 nodes, but there are a lot more IPs listed
> > > in the discovery hosts, is that intentional? Same for minimum\_master\_nodes
> > > being 3.
> > >
> > > Regards,
> > > Mark Walkom
> > >
> > > Infrastructure Engineer
> > > Campaign Monitor
> > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > >
> > > On 17 December 2013 18:55, lekkie omotayo [lekkie...@gmail.com](mailto:lekkie...@gmail.com) wrote:
> > >
> > > > Hi guys,
> > > >
> > > > We index a document at \*\*\* milli on Elasticsearch, which I think is too
> > > > slow especially for the amount of resources we have setup. We will like to
> > > > index at the rate of 500tps. Each document weighs between 20K and 30K.
> > > > How many indexes are advisable to be done at once (assuming we can
> > > > afford to send multiple http index request to the server at once)? I
> > > > understand bulk indexing is a preferred approach, for a 30K document how
> > > > much can be bulked at once? How many http bulk request (supposing I am
> > > > using a multi-threaded http client ot make requests) is advisable to make?
> > > >
> > > > I will appreciate suggestions and how to index this document as fast as
> > > > possible. We have two nodes set up, the config below is for one out of the
> > > > two:
> > > >
> > > > Shards: 5
> > > > Replica: 1
> > > >
> > > > "nodes" : {
> > > > "T5l5mvIdQsW3je7WmSPOcg" : {
> > > > "name" : "SEARCH-01",
> > > > "version" : "0.90.7",
> > > > "attributes" : {
> > > > "rack\_id" : "prod",
> > > > "max\_local\_storage\_nodes" : "1"
> > > > },
> > > > "settings" : {
> > > > "node.rack\_id" : "prod",
> > > > "action.disable\_delete\_all\_indices" : "true",
> > > > "cloud.node.auto\_attributes" : "true",
> > > > "indices.ttl.interval" : "90d",
> > > > "node.max\_local\_storage\_nodes" : "1",
> > > > "bootstrap.mlockall" : "true",
> > > > "index.mapper.dynamic" : "true",
> > > > "cluster.routing.allocation.awareness.attributes" : "rack\_id",
> > > > "discovery.zen.minimum\_master\_nodes" : "3",
> > > > "gateway.expected\_nodes" : "1",
> > > > "discovery.zen.ping.unicast.hosts" :
> > > > "172.25.15.170,172.25.15.172,172.46.1.170,172.46.1.172",
> > > > "discovery.zen.ping.multicast.enabled" : "false",
> > > > "action.auto\_create\_index" : "true"
> > > > },
> > > > "os" : {
> > > > "refresh\_interval" : 1000,
> > > > "available\_processors" : 8,
> > > > "cpu" : {
> > > > "vendor" : "Intel",
> > > > "model" : "Xeon",
> > > > "mhz" : 2600,
> > > > "total\_cores" : 8,
> > > > "total\_sockets" : 2,
> > > > "cores\_per\_socket" : 4,
> > > > "cache\_size" : "20kb",
> > > > "cache\_size\_in\_bytes" : 20480
> > > > },
> > > > "mem" : {
> > > > "total" : "17.5gb",
> > > > "total\_in\_bytes" : 18836545536
> > > > },
> > > > "swap" : {
> > > > "total" : "5.8gb",
> > > > "total\_in\_bytes" : 6274670592
> > > > }
> > > > },
> > > > "process" : {
> > > > "refresh\_interval" : 1000,
> > > > "id" : 3459,
> > > > "max\_file\_descriptors" : 64000
> > > > },
> > > > "jvm" : {
> > > > "pid" : 3459,
> > > > "version" : "1.7.0\_45",
> > > > "vm\_name" : "OpenJDK 64-Bit Server VM",
> > > > "vm\_version" : "24.45-b08",
> > > > "vm\_vendor" : "Oracle Corporation",
> > > > "start\_time" : 1386953353018,
> > > > "mem" : {
> > > > "heap\_init" : "10.5gb",
> > > > "heap\_init\_in\_bytes" : 11301552128,
> > > > "heap\_max" : "10.4gb",
> > > > "heap\_max\_in\_bytes" : 11231821824,
> > > > "non\_heap\_init" : "23.1mb",
> > > > "non\_heap\_init\_in\_bytes" : 24313856,
> > > > "non\_heap\_max" : "214mb",
> > > > "non\_heap\_max\_in\_bytes" : 224395264,
> > > > "direct\_max" : "10.4gb",
> > > > "direct\_max\_in\_bytes" : 11231821824
> > > > }
> > > > },
> > > > "thread\_pool" : {
> > > > "generic" : {
> > > > "type" : "cached",
> > > > "keep\_alive" : "30s"
> > > > },
> > > > "index" : {
> > > > "type" : "fixed",
> > > > "min" : 8,
> > > > "max" : 8,
> > > > "queue\_size" : "200"
> > > > },
> > > > "get" : {
> > > > "type" : "fixed",
> > > > "min" : 8,
> > > > "max" : 8,
> > > > "queue\_size" : "1k"
> > > > },
> > > > "snapshot" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 4,
> > > > "keep\_alive" : "5m"
> > > > },
> > > > "merge" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 4,
> > > > "keep\_alive" : "5m"
> > > > },
> > > > "suggest" : {
> > > > "type" : "fixed",
> > > > "min" : 8,
> > > > "max" : 8,
> > > > "queue\_size" : "1k"
> > > > },
> > > > "bulk" : {
> > > > "type" : "fixed",
> > > > "min" : 8,
> > > > "max" : 8,
> > > > "queue\_size" : "50"
> > > > },
> > > > "optimize" : {
> > > > "type" : "fixed",
> > > > "min" : 1,
> > > > "max" : 1
> > > > },
> > > > "warmer" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 4,
> > > > "keep\_alive" : "5m"
> > > > },
> > > > "flush" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 4,
> > > > "keep\_alive" : "5m"
> > > > },
> > > > "search" : {
> > > > "type" : "fixed",
> > > > "min" : 24,
> > > > "max" : 24,
> > > > "queue\_size" : "1k"
> > > > },
> > > > "percolate" : {
> > > > "type" : "fixed",
> > > > "min" : 8,
> > > > "max" : 8,
> > > > "queue\_size" : "1k"
> > > > },
> > > > "management" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 5,
> > > > "keep\_alive" : "5m"
> > > > },
> > > > "refresh" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 4,
> > > > "keep\_alive" : "5m"
> > > > }
> > > > },
> > > > "network" : {
> > > > "refresh\_interval" : 5000
> > > > },
> > > > "http" : {
> > > > "max\_content\_length" : "100mb",
> > > > "max\_content\_length\_in\_bytes" : 104857600
> > > > },
> > > > "plugins" :
> > > > }
> > > > }
> > > >
> > > > The Mapping is dynamically created because we create types daily and it
> > > > looks like:
> > > > {
> > > > "consumers-20131216": {
> > > > "properties": {
> > > > "requestData": {
> > > > "type": "string"
> > > > },
> > > > "requestTimestamp": {
> > > > "type": "date",
> > > > "format": "dateOptionalTime"
> > > > },
> > > > "responseData": {
> > > > "type": "string"
> > > > },
> > > > "responseTimestamp": {
> > > > "type": "date",
> > > > "format": "dateOptionalTime"
> > > > },
> > > > "sequenceId": {
> > > > "type": "long"
> > > > },
> > > > "service": {
> > > > "type": "string"
> > > > },
> > > > "systemResponseCode": {
> > > > "type": "string"
> > > > },
> > > > "systemResponseMessage": {
> > > > "type": "string"
> > > > },
> > > > "transactionComponentTypeId": {
> > > > "type": "long"
> > > > },
> > > > "transactionLogId": {
> > > > "type": "long"
> > > > },
> > > > "user": {
> > > > "type": "string"
> > > > }
> > > > }
> > > > }
> > > > }
> > > >
> > > > Regards.
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%
> > > > [40googlegroups.com](http://40googlegroups.com).
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/f0be2cb9-9832-4de3-a3ab-ab0843662ae4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0be2cb9-9832-4de3-a3ab-ab0843662ae4%40googlegroups.com)
> > > .
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a1961bc0-f9f9-46fa-a8e5-59b37fadfb28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a1961bc0-f9f9-46fa-a8e5-59b37fadfb28%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 18, 2013, 10:49am UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906/6 "2013-12-18T10:49:07Z")
The lag over that inter-DC link is probably causing your issues.
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 18 December 2013 18:12, lekkie omotayo [lekkie.aydot@gmail.com](mailto:lekkie.aydot@gmail.com) wrote:
> Yes they all have the same capacity.
>
> Yes, they are in different data centers (off-site).
>
> On Tuesday, 17 December 2013 22:33:07 UTC+1, Mark Walkom wrote:
>
> > ES will only go as fast as the slowest node. With that in mind, are your
> > "DR" nodes the same capacity?
> >
> > I also notice they are in different subnets, does that imply they are in
> > different datacenters?
> >
> > Regards,
> > Mark Walkom
> >
> > Infrastructure Engineer
> > Campaign Monitor
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> >
> > On 18 December 2013 00:44, lekkie omotayo [lekkie...@gmail.com](mailto:lekkie...@gmail.com) wrote:
> >
> > > Thanks for the insight.
> > >
> > > > > > > First tip would be to drop OpenJDK and move to Oracle, you'll get a
> > > > > > > lot better performance.
> > >
> > > So I changed to Oracle JDK and latency dropped from 4000millisecond to
> > > around 2500millisecond.
> > >
> > > .....................
> > >
> > > > > > > It might also be worth removing indices.ttl.interval and just using
> > > > > > > a script to delete old indices as TTL searches can use a fair bit of
> > > > > > > resources.
> > >
> > > We also dropped indices.ttl.interval and it further dropped to
> > > 1500milliseconds.
> > >
> > > > > > > You also mentioned you have 2 nodes, but there are a lot more IPs
> > > > > > > listed in the discovery hosts, is that intentional? Same for
> > > > > > > minimum\_master\_nodes being 3.
> > > > > > > Yes, the other 2 nodes are DR nodes. So we basically have 4 nodes but 2
> > > > > > > are for disaster recovery. And the discovery.zen.minimum\_master\_nodes
> > > > > > > was calculated based on the n/2 + 1, where n was 4.
> > >
> > > One other thing to note, every request is an upsert.
> > >
> > > What we have now is 1500milliseconds per upsert. This is still very
> > > high. We are looking at doing sub-zero millisecond or 10s of millisecond
> > > for bulk upload. Can this be achieved or it is a pipe dream?
> > >
> > > On Tuesday, 17 December 2013 09:12:52 UTC+1, Mark Walkom wrote:
> > >
> > > > First tip would be to drop OpenJDK and move to Oracle, you'll get a lot
> > > > better performance.
> > > > Bulk depends a lot on your setup and document size etc, but upwards of
> > > > 5K is generally towards the upper limit.
> > > > It might also be worth removing indices.ttl.interval and just using a
> > > > script to delete old indices as TTL searches can use a fair bit of
> > > > resources.
> > > >
> > > > You also mentioned you have 2 nodes, but there are a lot more IPs
> > > > listed in the discovery hosts, is that intentional? Same for
> > > > minimum\_master\_nodes being 3.
> > > >
> > > > Regards,
> > > > Mark Walkom
> > > >
> > > > Infrastructure Engineer
> > > > Campaign Monitor
> > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > > >
> > > > On 17 December 2013 18:55, lekkie omotayo [lekkie...@gmail.com](mailto:lekkie...@gmail.com) wrote:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > We index a document at \*\*\* milli on Elasticsearch, which I think is
> > > > > too slow especially for the amount of resources we have setup. We will like
> > > > > to index at the rate of 500tps. Each document weighs between 20K and 30K.
> > > > > How many indexes are advisable to be done at once (assuming we can
> > > > > afford to send multiple http index request to the server at once)? I
> > > > > understand bulk indexing is a preferred approach, for a 30K document how
> > > > > much can be bulked at once? How many http bulk request (supposing I am
> > > > > using a multi-threaded http client ot make requests) is advisable to make?
> > > > >
> > > > > I will appreciate suggestions and how to index this document as fast
> > > > > as possible. We have two nodes set up, the config below is for one out of
> > > > > the two:
> > > > >
> > > > > Shards: 5
> > > > > Replica: 1
> > > > >
> > > > > "nodes" : {
> > > > > "T5l5mvIdQsW3je7WmSPOcg" : {
> > > > > "name" : "SEARCH-01",
> > > > > "version" : "0.90.7",
> > > > > "attributes" : {
> > > > > "rack\_id" : "prod",
> > > > > "max\_local\_storage\_nodes" : "1"
> > > > > },
> > > > > "settings" : {
> > > > > "node.rack\_id" : "prod",
> > > > > "action.disable\_delete\_all\_indices" : "true",
> > > > > "cloud.node.auto\_attributes" : "true",
> > > > > "indices.ttl.interval" : "90d",
> > > > > "node.max\_local\_storage\_nodes" : "1",
> > > > > "bootstrap.mlockall" : "true",
> > > > > "index.mapper.dynamic" : "true",
> > > > > "cluster.routing.allocation.awareness.attributes" : "rack\_id",
> > > > > "discovery.zen.minimum\_master\_nodes" : "3",
> > > > > "gateway.expected\_nodes" : "1",
> > > > > "discovery.zen.ping.unicast.hosts" :
> > > > > "172.25.15.170,172.25.15.172,172.46.1.170,172.46.1.172",
> > > > > "discovery.zen.ping.multicast.enabled" : "false",
> > > > > "action.auto\_create\_index" : "true"
> > > > > },
> > > > > "os" : {
> > > > > "refresh\_interval" : 1000,
> > > > > "available\_processors" : 8,
> > > > > "cpu" : {
> > > > > "vendor" : "Intel",
> > > > > "model" : "Xeon",
> > > > > "mhz" : 2600,
> > > > > "total\_cores" : 8,
> > > > > "total\_sockets" : 2,
> > > > > "cores\_per\_socket" : 4,
> > > > > "cache\_size" : "20kb",
> > > > > "cache\_size\_in\_bytes" : 20480
> > > > > },
> > > > > "mem" : {
> > > > > "total" : "17.5gb",
> > > > > "total\_in\_bytes" : 18836545536
> > > > > },
> > > > > "swap" : {
> > > > > "total" : "5.8gb",
> > > > > "total\_in\_bytes" : 6274670592
> > > > > }
> > > > > },
> > > > > "process" : {
> > > > > "refresh\_interval" : 1000,
> > > > > "id" : 3459,
> > > > > "max\_file\_descriptors" : 64000
> > > > > },
> > > > > "jvm" : {
> > > > > "pid" : 3459,
> > > > > "version" : "1.7.0\_45",
> > > > > "vm\_name" : "OpenJDK 64-Bit Server VM",
> > > > > "vm\_version" : "24.45-b08",
> > > > > "vm\_vendor" : "Oracle Corporation",
> > > > > "start\_time" : 1386953353018,
> > > > > "mem" : {
> > > > > "heap\_init" : "10.5gb",
> > > > > "heap\_init\_in\_bytes" : 11301552128,
> > > > > "heap\_max" : "10.4gb",
> > > > > "heap\_max\_in\_bytes" : 11231821824,
> > > > > "non\_heap\_init" : "23.1mb",
> > > > > "non\_heap\_init\_in\_bytes" : 24313856,
> > > > > "non\_heap\_max" : "214mb",
> > > > > "non\_heap\_max\_in\_bytes" : 224395264,
> > > > > "direct\_max" : "10.4gb",
> > > > > "direct\_max\_in\_bytes" : 11231821824
> > > > > }
> > > > > },
> > > > > "thread\_pool" : {
> > > > > "generic" : {
> > > > > "type" : "cached",
> > > > > "keep\_alive" : "30s"
> > > > > },
> > > > > "index" : {
> > > > > "type" : "fixed",
> > > > > "min" : 8,
> > > > > "max" : 8,
> > > > > "queue\_size" : "200"
> > > > > },
> > > > > "get" : {
> > > > > "type" : "fixed",
> > > > > "min" : 8,
> > > > > "max" : 8,
> > > > > "queue\_size" : "1k"
> > > > > },
> > > > > "snapshot" : {
> > > > > "type" : "scaling",
> > > > > "min" : 1,
> > > > > "max" : 4,
> > > > > "keep\_alive" : "5m"
> > > > > },
> > > > > "merge" : {
> > > > > "type" : "scaling",
> > > > > "min" : 1,
> > > > > "max" : 4,
> > > > > "keep\_alive" : "5m"
> > > > > },
> > > > > "suggest" : {
> > > > > "type" : "fixed",
> > > > > "min" : 8,
> > > > > "max" : 8,
> > > > > "queue\_size" : "1k"
> > > > > },
> > > > > "bulk" : {
> > > > > "type" : "fixed",
> > > > > "min" : 8,
> > > > > "max" : 8,
> > > > > "queue\_size" : "50"
> > > > > },
> > > > > "optimize" : {
> > > > > "type" : "fixed",
> > > > > "min" : 1,
> > > > > "max" : 1
> > > > > },
> > > > > "warmer" : {
> > > > > "type" : "scaling",
> > > > > "min" : 1,
> > > > > "max" : 4,
> > > > > "keep\_alive" : "5m"
> > > > > },
> > > > > "flush" : {
> > > > > "type" : "scaling",
> > > > > "min" : 1,
> > > > > "max" : 4,
> > > > > "keep\_alive" : "5m"
> > > > > },
> > > > > "search" : {
> > > > > "type" : "fixed",
> > > > > "min" : 24,
> > > > > "max" : 24,
> > > > > "queue\_size" : "1k"
> > > > > },
> > > > > "percolate" : {
> > > > > "type" : "fixed",
> > > > > "min" : 8,
> > > > > "max" : 8,
> > > > > "queue\_size" : "1k"
> > > > > },
> > > > > "management" : {
> > > > > "type" : "scaling",
> > > > > "min" : 1,
> > > > > "max" : 5,
> > > > > "keep\_alive" : "5m"
> > > > > },
> > > > > "refresh" : {
> > > > > "type" : "scaling",
> > > > > "min" : 1,
> > > > > "max" : 4,
> > > > > "keep\_alive" : "5m"
> > > > > }
> > > > > },
> > > > > "network" : {
> > > > > "refresh\_interval" : 5000
> > > > > },
> > > > > "http" : {
> > > > > "max\_content\_length" : "100mb",
> > > > > "max\_content\_length\_in\_bytes" : 104857600
> > > > > },
> > > > > "plugins" :
> > > > > }
> > > > > }
> > > > >
> > > > > The Mapping is dynamically created because we create types daily and
> > > > > it looks like:
> > > > > {
> > > > > "consumers-20131216": {
> > > > > "properties": {
> > > > > "requestData": {
> > > > > "type": "string"
> > > > > },
> > > > > "requestTimestamp": {
> > > > > "type": "date",
> > > > > "format": "dateOptionalTime"
> > > > > },
> > > > > "responseData": {
> > > > > "type": "string"
> > > > > },
> > > > > "responseTimestamp": {
> > > > > "type": "date",
> > > > > "format": "dateOptionalTime"
> > > > > },
> > > > > "sequenceId": {
> > > > > "type": "long"
> > > > > },
> > > > > "service": {
> > > > > "type": "string"
> > > > > },
> > > > > "systemResponseCode": {
> > > > > "type": "string"
> > > > > },
> > > > > "systemResponseMessage": {
> > > > > "type": "string"
> > > > > },
> > > > > "transactionComponentTypeId": {
> > > > > "type": "long"
> > > > > },
> > > > > "transactionLogId": {
> > > > > "type": "long"
> > > > > },
> > > > > "user": {
> > > > > "type": "string"
> > > > > }
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > > Regards.
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%40goo
> > > > > [glegroups.com](http://glegroups.com).
> > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > msgid/elasticsearch/f0be2cb9-9832-4de3-a3ab-ab0843662ae4%
> > > > [40googlegroups.com](http://40googlegroups.com).
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/a1961bc0-f9f9-46fa-a8e5-59b37fadfb28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a1961bc0-f9f9-46fa-a8e5-59b37fadfb28%40googlegroups.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624a5Rz9dEfvohDWO7NwzaJ%3Dw96Dv8ysah7XhUZ6m\_XssJQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624a5Rz9dEfvohDWO7NwzaJ%3Dw96Dv8ysah7XhUZ6m_XssJQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@lekkie](https://discuss.elastic.co/u/lekkie)
#### Post date: [December 18, 2013, 4:10pm UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906/7 "2013-12-18T16:10:42Z")
Are there any other protocols other than HTTP I can send request over?
Something faster than HTTP? Or do you mean the physical NIC? We run on a LAN
On Wednesday, 18 December 2013 11:49:07 UTC+1, Mark Walkom wrote:
> The lag over that inter-DC link is probably causing your issues.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
>
> On 18 December 2013 18:12, lekkie omotayo \<[lekkie...@gmail.com](mailto:lekkie...@gmail.com)\
>
> > wrote:
>
> > Yes they all have the same capacity.
> >
> > Yes, they are in different data centers (off-site).
> >
> > On Tuesday, 17 December 2013 22:33:07 UTC+1, Mark Walkom wrote:
> >
> > > ES will only go as fast as the slowest node. With that in mind, are your
> > > "DR" nodes the same capacity?
> > >
> > > I also notice they are in different subnets, does that imply they are in
> > > different datacenters?
> > >
> > > Regards,
> > > Mark Walkom
> > >
> > > Infrastructure Engineer
> > > Campaign Monitor
> > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > >
> > > On 18 December 2013 00:44, lekkie omotayo [lekkie...@gmail.com](mailto:lekkie...@gmail.com) wrote:
> > >
> > > > Thanks for the insight.
> > > >
> > > > > > > > First tip would be to drop OpenJDK and move to Oracle, you'll get
> > > > > > > > a lot better performance.
> > > >
> > > > So I changed to Oracle JDK and latency dropped from 4000millisecond to
> > > > around 2500millisecond.
> > > >
> > > > .....................
> > > >
> > > > > > > > It might also be worth removing indices.ttl.interval and just
> > > > > > > > using a script to delete old indices as TTL searches can use a fair bit of
> > > > > > > > resources.
> > > >
> > > > We also dropped indices.ttl.interval and it further dropped to
> > > > 1500milliseconds.
> > > >
> > > > > > > > You also mentioned you have 2 nodes, but there are a lot more IPs
> > > > > > > > listed in the discovery hosts, is that intentional? Same for
> > > > > > > > minimum\_master\_nodes being 3.
> > > > > > > > Yes, the other 2 nodes are DR nodes. So we basically have 4 nodes but 2
> > > > > > > > are for disaster recovery. And the discovery.zen.minimum\_master\_nodes
> > > > > > > > was calculated based on the n/2 + 1, where n was 4.
> > > >
> > > > One other thing to note, every request is an upsert.
> > > >
> > > > What we have now is 1500milliseconds per upsert. This is still very
> > > > high. We are looking at doing sub-zero millisecond or 10s of millisecond
> > > > for bulk upload. Can this be achieved or it is a pipe dream?
> > > >
> > > > On Tuesday, 17 December 2013 09:12:52 UTC+1, Mark Walkom wrote:
> > > >
> > > > > First tip would be to drop OpenJDK and move to Oracle, you'll get a
> > > > > lot better performance.
> > > > > Bulk depends a lot on your setup and document size etc, but upwards of
> > > > > 5K is generally towards the upper limit.
> > > > > It might also be worth removing indices.ttl.interval and just using a
> > > > > script to delete old indices as TTL searches can use a fair bit of
> > > > > resources.
> > > > >
> > > > > You also mentioned you have 2 nodes, but there are a lot more IPs
> > > > > listed in the discovery hosts, is that intentional? Same for
> > > > > minimum\_master\_nodes being 3.
> > > > >
> > > > > Regards,
> > > > > Mark Walkom
> > > > >
> > > > > Infrastructure Engineer
> > > > > Campaign Monitor
> > > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > > > >
> > > > > On 17 December 2013 18:55, lekkie omotayo [lekkie...@gmail.com](mailto:lekkie...@gmail.com) wrote:
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > We index a document at \*\*\* milli on Elasticsearch, which I think is
> > > > > > too slow especially for the amount of resources we have setup. We will like
> > > > > > to index at the rate of 500tps. Each document weighs between 20K and 30K.
> > > > > > How many indexes are advisable to be done at once (assuming we can
> > > > > > afford to send multiple http index request to the server at once)? I
> > > > > > understand bulk indexing is a preferred approach, for a 30K document how
> > > > > > much can be bulked at once? How many http bulk request (supposing I am
> > > > > > using a multi-threaded http client ot make requests) is advisable to make?
> > > > > >
> > > > > > I will appreciate suggestions and how to index this document as fast
> > > > > > as possible. We have two nodes set up, the config below is for one out of
> > > > > > the two:
> > > > > >
> > > > > > Shards: 5
> > > > > > Replica: 1
> > > > > >
> > > > > > "nodes" : {
> > > > > > "T5l5mvIdQsW3je7WmSPOcg" : {
> > > > > > "name" : "SEARCH-01",
> > > > > > "version" : "0.90.7",
> > > > > > "attributes" : {
> > > > > > "rack\_id" : "prod",
> > > > > > "max\_local\_storage\_nodes" : "1"
> > > > > > },
> > > > > > "settings" : {
> > > > > > "node.rack\_id" : "prod",
> > > > > > "action.disable\_delete\_all\_indices" : "true",
> > > > > > "cloud.node.auto\_attributes" : "true",
> > > > > > "indices.ttl.interval" : "90d",
> > > > > > "node.max\_local\_storage\_nodes" : "1",
> > > > > > "bootstrap.mlockall" : "true",
> > > > > > "index.mapper.dynamic" : "true",
> > > > > > "cluster.routing.allocation.awareness.attributes" :
> > > > > > "rack\_id",
> > > > > > "discovery.zen.minimum\_master\_nodes" : "3",
> > > > > > "gateway.expected\_nodes" : "1",
> > > > > > "discovery.zen.ping.unicast.hosts" :
> > > > > > "172.25.15.170,172.25.15.172,172.46.1.170,172.46.1.172",
> > > > > > "discovery.zen.ping.multicast.enabled" : "false",
> > > > > > "action.auto\_create\_index" : "true"
> > > > > > },
> > > > > > "os" : {
> > > > > > "refresh\_interval" : 1000,
> > > > > > "available\_processors" : 8,
> > > > > > "cpu" : {
> > > > > > "vendor" : "Intel",
> > > > > > "model" : "Xeon",
> > > > > > "mhz" : 2600,
> > > > > > "total\_cores" : 8,
> > > > > > "total\_sockets" : 2,
> > > > > > "cores\_per\_socket" : 4,
> > > > > > "cache\_size" : "20kb",
> > > > > > "cache\_size\_in\_bytes" : 20480
> > > > > > },
> > > > > > "mem" : {
> > > > > > "total" : "17.5gb",
> > > > > > "total\_in\_bytes" : 18836545536
> > > > > > },
> > > > > > "swap" : {
> > > > > > "total" : "5.8gb",
> > > > > > "total\_in\_bytes" : 6274670592
> > > > > > }
> > > > > > },
> > > > > > "process" : {
> > > > > > "refresh\_interval" : 1000,
> > > > > > "id" : 3459,
> > > > > > "max\_file\_descriptors" : 64000
> > > > > > },
> > > > > > "jvm" : {
> > > > > > "pid" : 3459,
> > > > > > "version" : "1.7.0\_45",
> > > > > > "vm\_name" : "OpenJDK 64-Bit Server VM",
> > > > > > "vm\_version" : "24.45-b08",
> > > > > > "vm\_vendor" : "Oracle Corporation",
> > > > > > "start\_time" : 1386953353018,
> > > > > > "mem" : {
> > > > > > "heap\_init" : "10.5gb",
> > > > > > "heap\_init\_in\_bytes" : 11301552128,
> > > > > > "heap\_max" : "10.4gb",
> > > > > > "heap\_max\_in\_bytes" : 11231821824,
> > > > > > "non\_heap\_init" : "23.1mb",
> > > > > > "non\_heap\_init\_in\_bytes" : 24313856,
> > > > > > "non\_heap\_max" : "214mb",
> > > > > > "non\_heap\_max\_in\_bytes" : 224395264,
> > > > > > "direct\_max" : "10.4gb",
> > > > > > "direct\_max\_in\_bytes" : 11231821824
> > > > > > }
> > > > > > },
> > > > > > "thread\_pool" : {
> > > > > > "generic" : {
> > > > > > "type" : "cached",
> > > > > > "keep\_alive" : "30s"
> > > > > > },
> > > > > > "index" : {
> > > > > > "type" : "fixed",
> > > > > > "min" : 8,
> > > > > > "max" : 8,
> > > > > > "queue\_size" : "200"
> > > > > > },
> > > > > > "get" : {
> > > > > > "type" : "fixed",
> > > > > > "min" : 8,
> > > > > > "max" : 8,
> > > > > > "queue\_size" : "1k"
> > > > > > },
> > > > > > "snapshot" : {
> > > > > > "type" : "scaling",
> > > > > > "min" : 1,
> > > > > > "max" : 4,
> > > > > > "keep\_alive" : "5m"
> > > > > > },
> > > > > > "merge" : {
> > > > > > "type" : "scaling",
> > > > > > "min" : 1,
> > > > > > "max" : 4,
> > > > > > "keep\_alive" : "5m"
> > > > > > },
> > > > > > "suggest" : {
> > > > > > "type" : "fixed",
> > > > > > "min" : 8,
> > > > > > "max" : 8,
> > > > > > "queue\_size" : "1k"
> > > > > > },
> > > > > > "bulk" : {
> > > > > > "type" : "fixed",
> > > > > > "min" : 8,
> > > > > > "max" : 8,
> > > > > > "queue\_size" : "50"
> > > > > > },
> > > > > > "optimize" : {
> > > > > > "type" : "fixed",
> > > > > > "min" : 1,
> > > > > > "max" : 1
> > > > > > },
> > > > > > "warmer" : {
> > > > > > "type" : "scaling",
> > > > > > "min" : 1,
> > > > > > "max" : 4,
> > > > > > "keep\_alive" : "5m"
> > > > > > },
> > > > > > "flush" : {
> > > > > > "type" : "scaling",
> > > > > > "min" : 1,
> > > > > > "max" : 4,
> > > > > > "keep\_alive" : "5m"
> > > > > > },
> > > > > > "search" : {
> > > > > > "type" : "fixed",
> > > > > > "min" : 24,
> > > > > > "max" : 24,
> > > > > > "queue\_size" : "1k"
> > > > > > },
> > > > > > "percolate" : {
> > > > > > "type" : "fixed",
> > > > > > "min" : 8,
> > > > > > "max" : 8,
> > > > > > "queue\_size" : "1k"
> > > > > > },
> > > > > > "management" : {
> > > > > > "type" : "scaling",
> > > > > > "min" : 1,
> > > > > > "max" : 5,
> > > > > > "keep\_alive" : "5m"
> > > > > > },
> > > > > > "refresh" : {
> > > > > > "type" : "scaling",
> > > > > > "min" : 1,
> > > > > > "max" : 4,
> > > > > > "keep\_alive" : "5m"
> > > > > > }
> > > > > > },
> > > > > > "network" : {
> > > > > > "refresh\_interval" : 5000
> > > > > > },
> > > > > > "http" : {
> > > > > > "max\_content\_length" : "100mb",
> > > > > > "max\_content\_length\_in\_bytes" : 104857600
> > > > > > },
> > > > > > "plugins" :
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > The Mapping is dynamically created because we create types daily and
> > > > > > it looks like:
> > > > > > {
> > > > > > "consumers-20131216": {
> > > > > > "properties": {
> > > > > > "requestData": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "requestTimestamp": {
> > > > > > "type": "date",
> > > > > > "format": "dateOptionalTime"
> > > > > > },
> > > > > > "responseData": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "responseTimestamp": {
> > > > > > "type": "date",
> > > > > > "format": "dateOptionalTime"
> > > > > > },
> > > > > > "sequenceId": {
> > > > > > "type": "long"
> > > > > > },
> > > > > > "service": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "systemResponseCode": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "systemResponseMessage": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "transactionComponentTypeId": {
> > > > > > "type": "long"
> > > > > > },
> > > > > > "transactionLogId": {
> > > > > > "type": "long"
> > > > > > },
> > > > > > "user": {
> > > > > > "type": "string"
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > Regards.
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > > msgid/elasticsearch/94df7d01-94d5-49f2-b817-f821e4910219%40goo
> > > > > > [glegroups.com](http://glegroups.com).
> > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > msgid/elasticsearch/f0be2cb9-9832-4de3-a3ab-ab0843662ae4%
> > > > > [40googlegroups.com](http://40googlegroups.com).
> > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/a1961bc0-f9f9-46fa-a8e5-59b37fadfb28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a1961bc0-f9f9-46fa-a8e5-59b37fadfb28%40googlegroups.com)
> > > .
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a0bb33c4-e6e9-46c7-aa3c-d261fa5090ae%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a0bb33c4-e6e9-46c7-aa3c-d261fa5090ae%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 19, 2013, 12:34am UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906/8 "2013-12-19T00:34:44Z")
Intra cluster comms are all handled over HTTP. What is the link between
your DCs like; 100M, 1G, 10G?
You could try using something like logstash to replicate the indexes, that
way you can have two clusters and it should reduce any latency.
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 19 December 2013 03:10, lekkie omotayo [lekkie.aydot@gmail.com](mailto:lekkie.aydot@gmail.com) wrote:
> Are there any other protocols other than HTTP I can send request over?
> Something faster than HTTP? Or do you mean the physical NIC? We run on a LAN
>
> On Wednesday, 18 December 2013 11:49:07 UTC+1, Mark Walkom wrote:
>
> > The lag over that inter-DC link is probably causing your issues.
> >
> > Regards,
> > Mark Walkom
> >
> > Infrastructure Engineer
> > Campaign Monitor
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> >
> > On 18 December 2013 18:12, lekkie omotayo [lekkie...@gmail.com](mailto:lekkie...@gmail.com) wrote:
> >
> > > Yes they all have the same capacity.
> > >
> > > Yes, they are in different data centers (off-site).
> > >
> > > On Tuesday, 17 December 2013 22:33:07 UTC+1, Mark Walkom wrote:
> > >
> > > > ES will only go as fast as the slowest node. With that in mind, are
> > > > your "DR" nodes the same capacity?
> > > >
> > > > I also notice they are in different subnets, does that imply they are
> > > > in different datacenters?
> > > >
> > > > Regards,
> > > > Mark Walkom
> > > >
> > > > Infrastructure Engineer
> > > > Campaign Monitor
> > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > > >
> > > > On 18 December 2013 00:44, lekkie omotayo [lekkie...@gmail.com](mailto:lekkie...@gmail.com) wrote:
> > > >
> > > > > Thanks for the insight.
> > > > >
> > > > > > > > > First tip would be to drop OpenJDK and move to Oracle, you'll get
> > > > > > > > > a lot better performance.
> > > > >
> > > > > So I changed to Oracle JDK and latency dropped from 4000millisecond to
> > > > > around 2500millisecond.
> > > > >
> > > > > .....................
> > > > >
> > > > > > > > > It might also be worth removing indices.ttl.interval and just
> > > > > > > > > using a script to delete old indices as TTL searches can use a fair bit of
> > > > > > > > > resources.
> > > > >
> > > > > We also dropped indices.ttl.interval and it further dropped to
> > > > > 1500milliseconds.
> > > > >
> > > > > > > > > You also mentioned you have 2 nodes, but there are a lot more IPs
> > > > > > > > > listed in the discovery hosts, is that intentional? Same for
> > > > > > > > > minimum\_master\_nodes being 3.
> > > > > > > > > Yes, the other 2 nodes are DR nodes. So we basically have 4 nodes but
> > > > > > > > > 2 are for disaster recovery. And the discovery.zen.minimum\_master\_nodes
> > > > > > > > > was calculated based on the n/2 + 1, where n was 4.
> > > > >
> > > > > One other thing to note, every request is an upsert.
> > > > >
> > > > > What we have now is 1500milliseconds per upsert. This is still very
> > > > > high. We are looking at doing sub-zero millisecond or 10s of millisecond
> > > > > for bulk upload. Can this be achieved or it is a pipe dream?
> > > > >
> > > > > On Tuesday, 17 December 2013 09:12:52 UTC+1, Mark Walkom wrote:
> > > > >
> > > > > > First tip would be to drop OpenJDK and move to Oracle, you'll get a
> > > > > > lot better performance.
> > > > > > Bulk depends a lot on your setup and document size etc, but upwards
> > > > > > of 5K is generally towards the upper limit.
> > > > > > It might also be worth removing indices.ttl.interval and just using a
> > > > > > script to delete old indices as TTL searches can use a fair bit of
> > > > > > resources.
> > > > > >
> > > > > > You also mentioned you have 2 nodes, but there are a lot more IPs
> > > > > > listed in the discovery hosts, is that intentional? Same for
> > > > > > minimum\_master\_nodes being 3.
> > > > > >
> > > > > > Regards,
> > > > > > Mark Walkom
> > > > > >
> > > > > > Infrastructure Engineer
> > > > > > Campaign Monitor
> > > > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > > > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > > > > >
> > > > > > On 17 December 2013 18:55, lekkie omotayo [lekkie...@gmail.com](mailto:lekkie...@gmail.com)wrote:
> > > > > >
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > We index a document at \*\*\* milli on Elasticsearch, which I think is
> > > > > > > too slow especially for the amount of resources we have setup. We will like
> > > > > > > to index at the rate of 500tps. Each document weighs between 20K and 30K.
> > > > > > > How many indexes are advisable to be done at once (assuming we can
> > > > > > > afford to send multiple http index request to the server at once)? I
> > > > > > > understand bulk indexing is a preferred approach, for a 30K document how
> > > > > > > much can be bulked at once? How many http bulk request (supposing I am
> > > > > > > using a multi-threaded http client ot make requests) is advisable to make?
> > > > > > >
> > > > > > > I will appreciate suggestions and how to index this document as fast
> > > > > > > as possible. We have two nodes set up, the config below is for one out of
> > > > > > > the two:
> > > > > > >
> > > > > > > Shards: 5
> > > > > > > Replica: 1
> > > > > > >
> > > > > > > "nodes" : {
> > > > > > > "T5l5mvIdQsW3je7WmSPOcg" : {
> > > > > > > "name" : "SEARCH-01",
> > > > > > > "version" : "0.90.7",
> > > > > > > "attributes" : {
> > > > > > > "rack\_id" : "prod",
> > > > > > > "max\_local\_storage\_nodes" : "1"
> > > > > > > },
> > > > > > > "settings" : {
> > > > > > > "node.rack\_id" : "prod",
> > > > > > > "action.disable\_delete\_all\_indices" : "true",
> > > > > > > "cloud.node.auto\_attributes" : "true",
> > > > > > > "indices.ttl.interval" : "90d",
> > > > > > > "node.max\_local\_storage\_nodes" : "1",
> > > > > > > "bootstrap.mlockall" : "true",
> > > > > > > "index.mapper.dynamic" : "true",
> > > > > > > "cluster.routing.allocation.awareness.attributes" :
> > > > > > > "rack\_id",
> > > > > > > "discovery.zen.minimum\_master\_nodes" : "3",
> > > > > > > "gateway.expected\_nodes" : "1",
> > > > > > > "discovery.zen.ping.unicast.hosts" :
> > > > > > > "172.25.15.170,172.25.15.172,172.46.1.170,172.46.1.172",
> > > > > > > "discovery.zen.ping.multicast.enabled" : "false",
> > > > > > > "action.auto\_create\_index" : "true"
> > > > > > > },
> > > > > > > "os" : {
> > > > > > > "refresh\_interval" : 1000,
> > > > > > > "available\_processors" : 8,
> > > > > > > "cpu" : {
> > > > > > > "vendor" : "Intel",
> > > > > > > "model" : "Xeon",
> > > > > > > "mhz" : 2600,
> > > > > > > "total\_cores" : 8,
> > > > > > > "total\_sockets" : 2,
> > > > > > > "cores\_per\_socket" : 4,
> > > > > > > "cache\_size" : "20kb",
> > > > > > > "cache\_size\_in\_bytes" : 20480
> > > > > > > },
> > > > > > > "mem" : {
> > > > > > > "total" : "17.5gb",
> > > > > > > "total\_in\_bytes" : 18836545536
> > > > > > > },
> > > > > > > "swap" : {
> > > > > > > "total" : "5.8gb",
> > > > > > > "total\_in\_bytes" : 6274670592
> > > > > > > }
> > > > > > > },
> > > > > > > "process" : {
> > > > > > > "refresh\_interval" : 1000,
> > > > > > > "id" : 3459,
> > > > > > > "max\_file\_descriptors" : 64000
> > > > > > > },
> > > > > > > "jvm" : {
> > > > > > > "pid" : 3459,
> > > > > > > "version" : "1.7.0\_45",
> > > > > > > "vm\_name" : "OpenJDK 64-Bit Server VM",
> > > > > > > "vm\_version" : "24.45-b08",
> > > > > > > "vm\_vendor" : "Oracle Corporation",
> > > > > > > "start\_time" : 1386953353018,
> > > > > > > "mem" : {
> > > > > > > "heap\_init" : "10.5gb",
> > > > > > > "heap\_init\_in\_bytes" : 11301552128,
> > > > > > > "heap\_max" : "10.4gb",
> > > > > > > "heap\_max\_in\_bytes" : 11231821824,
> > > > > > > "non\_heap\_init" : "23.1mb",
> > > > > > > "non\_heap\_init\_in\_bytes" : 24313856,
> > > > > > > "non\_heap\_max" : "214mb",
> > > > > > > "non\_heap\_max\_in\_bytes" : 224395264,
> > > > > > > "direct\_max" : "10.4gb",
> > > > > > > "direct\_max\_in\_bytes" : 11231821824
> > > > > > > }
> > > > > > > },
> > > > > > > "thread\_pool" : {
> > > > > > > "generic" : {
> > > > > > > "type" : "cached",
> > > > > > > "keep\_alive" : "30s"
> > > > > > > },
> > > > > > > "index" : {
> > > > > > > "type" : "fixed",
> > > > > > > "min" : 8,
> > > > > > > "max" : 8,
> > > > > > > "queue\_size" : "200"
> > > > > > > },
> > > > > > > "get" : {
> > > > > > > "type" : "fixed",
> > > > > > > "min" : 8,
> > > > > > > "max" : 8,
> > > > > > > "queue\_size" : "1k"
> > > > > > > },
> > > > > > > "snapshot" : {
> > > > > > > "type" : "scaling",
> > > > > > > "min" : 1,
> > > > > > > "max" : 4,
> > > > > > > "keep\_alive" : "5m"
> > > > > > > },
> > > > > > > "merge" : {
> > > > > > > "type" : "scaling",
> > > > > > > "min" : 1,
> > > > > > > "max" : 4,
> > > > > > > "keep\_alive" : "5m"
> > > > > > > },
> > > > > > > "suggest" : {
> > > > > > > "type" : "fixed",
> > > > > > > "min" : 8,
> > > > > > > "max" : 8,
> > > > > > > "queue\_size" : "1k"
> > > > > > > },
> > > > > > > "bulk" : {
> > > > > > > "type" : "fixed",
> > > > > > > "min" : 8,
> > > > > > > "max" : 8,
> > > > > > > "queue\_size" : "50"
> > > > > > > },
> > > > > > > "optimize" : {
> > > > > > > "type" : "fixed",
> > > > > > > "min" : 1,
> > > > > > > "max" : 1
> > > > > > > },
> > > > > > > "warmer" : {
> > > > > > > "type" : "scaling",
> > > > > > > "min" : 1,
> > > > > > > "max" : 4,
> > > > > > > "keep\_alive" : "5m"
> > > > > > > },
> > > > > > > "flush" : {
> > > > > > > "type" : "scaling",
> > > > > > > "min" : 1,
> > > > > > > "max" : 4,
> > > > > > > "keep\_alive" : "5m"
> > > > > > > },
> > > > > > > "search" : {
> > > > > > > "type" : "fixed",
> > > > > > > "min" : 24,
> > > > > > > "max" : 24,
> > > > > > > "queue\_size" : "1k"
> > > > > > > },
> > > > > > > "percolate" : {
> > > > > > > "type" : "fixed",
> > > > > > > "min" : 8,
> > > > > > > "max" : 8,
> > > > > > > "queue\_size" : "1k"
> > > > > > > },
> > > > > > > "management" : {
> > > > > > > "type" : "scaling",
> > > > > > > "min" : 1,
> > > > > > > "max" : 5,
> > > > > > > "keep\_alive" : "5m"
> > > > > > > },
> > > > > > > "refresh" : {
> > > > > > > "type" : "scaling",
> > > > > > > "min" : 1,
> > > > > > > "max" : 4,
> > > > > > > "keep\_alive" : "5m"
> > > > > > > }
> > > > > > > },
> > > > > > > "network" : {
> > > > > > > "refresh\_interval" : 5000
> > > > > > > },
> > > > > > > "http" : {
> > > > > > > "max\_content\_length" : "100mb",
> > > > > > > "max\_content\_length\_in\_bytes" : 104857600
> > > > > > > },
> > > > > > > "plugins" :
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > > The Mapping is dynamically created because we create types daily and
> > > > > > > it looks like:
> > > > > > > {
> > > > > > > "consumers-20131216": {
> > > > > > > "properties": {
> > > > > > > "requestData": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "requestTimestamp": {
> > > > > > > "type": "date",
> > > > > > > "format": "dateOptionalTime"
> > > > > > > },
> > > > > > > "responseData": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "responseTimestamp": {
> > > > > > > "type": "date",
> > > > > > > "format": "dateOptionalTime"
> > > > > > > },
> > > > > > > "sequenceId": {
> > > > > > > "type": "long"
> > > > > > > },
> > > > > > > "service": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "systemResponseCode": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "systemResponseMessage": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "transactionComponentTypeId": {
> > > > > > > "type": "long"
> > > > > > > },
> > > > > > > "transactionLogId": {
> > > > > > > "type": "long"
> > > > > > > },
> > > > > > > "user": {
> > > > > > > "type": "string"
> > > > > > > }
> > > > > > > }
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > > Regards.
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > > > > Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > > To view this discussion on the web visit
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/94df7d01-94d](https://groups.google.com/d/msgid/elasticsearch/94df7d01-94d)
> > > > > > > 5-49f2-b817-f821e4910219%[40googlegroups.com](http://40googlegroups.com).
> > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > > msgid/elasticsearch/f0be2cb9-9832-4de3-a3ab-ab0843662ae4%40goo
> > > > > > [glegroups.com](http://glegroups.com).
> > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > msgid/elasticsearch/a1961bc0-f9f9-46fa-a8e5-59b37fadfb28%
> > > > [40googlegroups.com](http://40googlegroups.com).
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/a0bb33c4-e6e9-46c7-aa3c-d261fa5090ae%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a0bb33c4-e6e9-46c7-aa3c-d261fa5090ae%40googlegroups.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bx-Fs8ivTatR70yv\_FurwsY9VHSo02F4e6LjaELyX1og%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bx-Fs8ivTatR70yv_FurwsY9VHSo02F4e6LjaELyX1og%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:00am UTC](https://discuss.elastic.co/t/very-slow-elasticsearch-index/14906/9 "2017-07-06T02:00:19Z")