# What can I do to make the "readings" do not disturb "writings"? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 4, 2013, 2:03pm UTC](https://discuss.elastic.co/t/what-can-i-do-to-make-the-readings-do-not-disturb-writings/14243 "2013-11-04T14:03:00Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Regis\_Depraute](https://avatars.discourse-cdn.com/v4/letter/r/258eb7/32.png) [@Regis\_Depraute](https://discuss.elastic.co/u/Regis_Depraute)\ **Post date:** [November 4, 2013, 2:03pm UTC](https://discuss.elastic.co/t/what-can-i-do-to-make-the-readings-do-not-disturb-writings/14243/1 "2013-11-04T14:03:00Z")
Hello, I use Elasticsearch to store logs into a database, I receive approximately 50G of logs per day, After logstash process, and elasticsearch storing, I have +- 22Go of logs per index. I have a problem with indexing and reading at the same time with elasticsearch. Everything is working fine until kibana is querying into the elasticsearch database, (some queries last 15 minutes) and no more data is written (or nearly, sometimes i got 25% of the logs) _What can I do to make the "readings" do not disturb "writings" ?_ The scheme is : logs goes to 512(UDP or TCP) =\> logstash =\> ES\_cluster \<= kibana(reader) - _I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) :_ he is receiving logs on port 512 and output is elasticsearch\_cluster (node03) config logstash : input { syslog { type =\> "syslog" format =\> "json" port =\> 514 } } filter { grok { ... too long ... } } output { elasticsearch { host =\> "aaa.bbb.ccc.ddd" ---\>here is the ip address of the node03 port =\> "9302" node\_name =\> "logstash-node" } } - _I have an Elasticsearch cluster : (elasticsearch-0.90.5) :_ node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9300 node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9301 node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data:\* false\*port 9302 - _and a dedicated server for kibana:_ he is reading logs into elasticsearch node03 _every VM are CentOS(6.4)_ \* \* Thanks RD. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 4, 2013, 5:32pm UTC](https://discuss.elastic.co/t/what-can-i-do-to-make-the-readings-do-not-disturb-writings/14243/2 "2013-11-04T17:32:34Z")
Basically elasticsearch uses shards for read and write operations. You don't need to have some "passive" nodes. If you feel that you can't search fast enough then you should consider for example increase the number of replicas (and the number of nodes). SSD drives, more memory should help as well. May be having 1 shard per index or using routing could help. HTH -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 4 nov. 2013 à 15:03, Regis Depraute [regis.depraute@gmail.com](mailto:regis.depraute@gmail.com) a écrit : > Hello, > > I use Elasticsearch to store logs into a database, I receive approximately 50G of logs per day, > After logstash process, and elasticsearch storing, I have +- 22Go of logs per index. > > I have a problem with indexing and reading at the same time with elasticsearch. > Everything is working fine until kibana is querying into the elasticsearch database, (some queries last 15 minutes) and no more data is written (or nearly, sometimes i got 25% of the logs) > > What can I do to make the "readings" do not disturb "writings" ? > > The scheme is : > > logs goes to 512(UDP or TCP) =\> logstash =\> ES\_cluster \<= kibana(reader) > > I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) : > he is receiving logs on port 512 > and output is elasticsearch\_cluster (node03) > > config logstash : > input { > syslog { > type =\> "syslog" > format =\> "json" > port =\> 514 > } > } > filter { > grok { > ... too long ... > } > } > output { > elasticsearch { > host =\> "aaa.bbb.ccc.ddd" ---\>here is the ip address of the node03 > port =\> "9302" > node\_name =\> "logstash-node" > } > } > > I have an Elasticsearch cluster : (elasticsearch-0.90.5) : > node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9300 > node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9301 > node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: false port 9302 > > and a dedicated server for kibana: > he is reading logs into elasticsearch node03 > > every VM are CentOS(6.4) > > Thanks > RD. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Regis\_Depraute](https://avatars.discourse-cdn.com/v4/letter/r/258eb7/32.png) [@Regis\_Depraute](https://discuss.elastic.co/u/Regis_Depraute)\ **Post date:** [November 5, 2013, 8:06am UTC](https://discuss.elastic.co/t/what-can-i-do-to-make-the-readings-do-not-disturb-writings/14243/3 "2013-11-05T08:06:31Z")
Hello, Is there a way to limit the cpu usage when i'm querying the elasticsearch database ? I'm loosing data (new logs - incoming data) when the cluster is too busy in querying. RD. Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit : > Basically elasticsearch uses shards for read and write operations. > You don't need to have some "passive" nodes. > > If you feel that you can't search fast enough then you should consider for > example increase the number of replicas (and the number of nodes). > > SSD drives, more memory should help as well. > May be having 1 shard per index or using routing could help. > > HTH > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 4 nov. 2013 à 15:03, Regis Depraute \<[regis.d...@gmail.com](mailto:regis.d...@gmail.com) \\> > a écrit : > > Hello, > > I use Elasticsearch to store logs into a database, I receive approximately > 50G of logs per day, > After logstash process, and elasticsearch storing, I have +- 22Go of logs > per index. > > I have a problem with indexing and reading at the same time with > elasticsearch. > Everything is working fine until kibana is querying into the elasticsearch > database, (some queries last 15 minutes) and no more data is written (or > nearly, sometimes i got 25% of the logs) > > _What can I do to make the "readings" do not disturb "writings" ?_ > > The scheme is : > > logs goes to 512(UDP or TCP) =\> logstash =\> ES\_cluster \<= kibana(reader) > > - \*I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) : > > - > > he is receiving logs on port 512 > and output is elasticsearch\_cluster (node03) > > config logstash : > input { > syslog { > type =\> "syslog" > format =\> "json" > port =\> 514 > } > } > filter { > grok { > ... too long ... > } > } > output { > elasticsearch { > host =\> "aaa.bbb.ccc.ddd" ---\>here is the ip address of the node03 > port =\> "9302" > node\_name =\> "logstash-node" > } > } > > - _I have an Elasticsearch cluster : (elasticsearch-0.90.5) :_ > > node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port > 9300 > node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port > 9301 > node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data:\* false\*port 9302 > > - _and a dedicated server for kibana:_ > > he is reading logs into elasticsearch node03 > > _every VM are CentOS(6.4)_ > \* > \* > Thanks > RD. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 5, 2013, 8:47am UTC](https://discuss.elastic.co/t/what-can-i-do-to-make-the-readings-do-not-disturb-writings/14243/4 "2013-11-05T08:47:59Z")
You should never loose any data. At least, your injection process should be aware that an index operation failed. And you can manage errors on a client level. I guess it's because of UDP or something? Unsure about your exact use case or architecture but you could consider to add another layer such as rabbitMQ to store first your documents in a queue and then consume them from rabbitMQ to elasticsearch although it's absolutely not something mandatory. Elasticsearch could perfectly manage your injection load if your architecture is well designed. How many shard do you have per index? How many index? Do you have enough file descriptors? Anything in logs? Also, I think you don't have enough memory to consume your data with Kibana as Kibana is using facets. 6 Gb RAM means 3Gb for elasticsearch VM? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr 5 novembre 2013 at 09:06:34, Regis Depraute ([regis.depraute@gmail.com](mailto:regis.depraute@gmail.com)) a écrit: Hello, Is there a way to limit the cpu usage when i'm querying the elasticsearch database ? I'm loosing data (new logs - incoming data) when the cluster is too busy in querying. RD. Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit : Basically elasticsearch uses shards for read and write operations. You don't need to have some "passive" nodes. If you feel that you can't search fast enough then you should consider for example increase the number of replicas (and the number of nodes). SSD drives, more memory should help as well. May be having 1 shard per index or using routing could help. HTH -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 4 nov. 2013 à 15:03, Regis Depraute [regis.d...@gmail.com](mailto:regis.d...@gmail.com) a écrit : Hello, I use Elasticsearch to store logs into a database, I receive approximately 50G of logs per day, After logstash process, and elasticsearch storing, I have +- 22Go of logs per index. I have a problem with indexing and reading at the same time with elasticsearch. Everything is working fine until kibana is querying into the elasticsearch database, (some queries last 15 minutes) and no more data is written (or nearly, sometimes i got 25% of the logs) What can I do to make the "readings" do not disturb "writings" ? The scheme is : logs goes to 512(UDP or TCP) =\> logstash =\> ES\_cluster \<= kibana(reader) I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) : he is receiving logs on port 512 and output is elasticsearch\_cluster (node03) config logstash : input { syslog { type =\> "syslog" format =\> "json" port =\> 514 } } filter { grok { ... too long ... } } output { elasticsearch { host =\> "aaa.bbb.ccc.ddd" ---\>here is the ip address of the node03 port =\> "9302" node\_name =\> "logstash-node" } } I have an Elasticsearch cluster : (elasticsearch-0.90.5) : node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9300 node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9301 node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: false port 9302 and a dedicated server for kibana: he is reading logs into elasticsearch node03 every VM are CentOS(6.4) Thanks RD. ## -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Regis\_Depraute](https://avatars.discourse-cdn.com/v4/letter/r/258eb7/32.png) [@Regis\_Depraute](https://discuss.elastic.co/u/Regis_Depraute)\ **Post date:** [November 5, 2013, 9:44am UTC](https://discuss.elastic.co/t/what-can-i-do-to-make-the-readings-do-not-disturb-writings/14243/5 "2013-11-05T09:44:56Z")
_I guess it's because of UDP or something?_ Yes, it's UDP I'm going to add "redis" to make a buffer, do you think it's a good idea ? Or it's better to use rabbitMQ ? _How many shard do you have per index? How many index?_ I have 5 shards 7 index each index weight +- 22 gb _Do you have enough file descriptors?_ What is file descriptors? _Anything in logs?_ I think I should increase RAM Memory : [2013-11-04 09:21:53,606][WARN][index.engine.robin] [node01] [ logstash-2013.11.04][0] failed engine java.lang.OutOfMemoryError: Java heap space [2013-11-04 09:22:28,988][WARN][cluster.action.shard] [node01]sending failed shard for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED],reason [engine failure, message [OutOfMemoryError[Java heap space]]] [2013-11-04 09:22:28,988][WARN][cluster.action.shard] [node01]received shard failed for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED],reason [engine failure, message [OutOfMemoryError[Java heap space]]] _6 Gb RAM means 3Gb for elasticsearch VM?_ yes Le mardi 5 novembre 2013 09:47:59 UTC+1, David Pilato a écrit : > You should never loose any data. At least, your injection process should > be aware that an index operation failed. > And you can manage errors on a client level. > > I guess it's because of UDP or something? > > Unsure about your exact use case or architecture but you could consider to > add another layer such as rabbitMQ to store first your documents in a queue > and then consume them from rabbitMQ to elasticsearch although it's > absolutely not something mandatory. Elasticsearch could perfectly manage > your injection load if your architecture is well designed. > > How many shard do you have per index? How many index? > Do you have enough file descriptors? > > Anything in logs? > > Also, I think you don't have enough memory to consume your data with > Kibana as Kibana is using facets. 6 Gb RAM means 3Gb for elasticsearch VM? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > 5 novembre 2013 at 09:06:34, Regis Depraute ([regis.d...@gmail.com](mailto:regis.d...@gmail.com)\) > a écrit: > > Hello, > > Is there a way to limit the cpu usage when i'm querying the elasticsearch > database ? > I'm loosing data (new logs - incoming data) when the cluster is too busy > in querying. > > RD. > > Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit : > > > Basically elasticsearch uses shards for read and write operations. > > You don't need to have some "passive" nodes. > > > > If you feel that you can't search fast enough then you should consider > > for example increase the number of replicas (and the number of nodes). > > > > SSD drives, more memory should help as well. > > May be having 1 shard per index or using routing could help. > > > > HTH > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 4 nov. 2013 à 15:03, Regis Depraute [regis.d...@gmail.com](mailto:regis.d...@gmail.com) a écrit : > > > > Hello, > > > > I use Elasticsearch to store logs into a database, I receive > > approximately 50G of logs per day, > > After logstash process, and elasticsearch storing, I have +- 22Go of logs > > per index. > > > > I have a problem with indexing and reading at the same time with > > elasticsearch. > > Everything is working fine until kibana is querying into the > > elasticsearch database, (some queries last 15 minutes) and no more data is > > written (or nearly, sometimes i got 25% of the logs) > > > > _What can I do to make the "readings" do not disturb "writings" ?_ > > > > The scheme is : > > > > logs goes to 512(UDP or TCP) =\> logstash =\> ES\_cluster \<= kibana(reader) > > > > - _I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) > > :_ > > > > he is receiving logs on port 512 > > and output is elasticsearch\_cluster (node03) > > > > config logstash : > > input { > > syslog { > > type =\> "syslog" > > format =\> "json" > > port =\> 514 > > } > > } > > filter { > > grok { > > ... too long ... > > } > > } > > output { > > elasticsearch { > > host =\> "aaa.bbb.ccc.ddd" ---\>here is the ip address of the node03 > > port =\> "9302" > > node\_name =\> "logstash-node" > > } > > } > > > > - _I have an Elasticsearch cluster : (elasticsearch-0.90.5) :_ > > > > node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true > > port 9300 > > node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true > > port 9301 > > node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: _false_port 9302 > > > > - _and a dedicated server for kibana:_ > > > > he is reading logs into elasticsearch node03 > > > > _every VM are CentOS(6.4)_ > > \* > > \* > > Thanks > > RD. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 5, 2013, 12:35pm UTC](https://discuss.elastic.co/t/what-can-i-do-to-make-the-readings-do-not-disturb-writings/14243/6 "2013-11-05T12:35:28Z")
So add more memory or more nodes. I think you don't need to "complicate" things with another layer. Check this webinar: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/webinars/elasticsearch-pre-flight-checklist/) It will help you a lot! -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 5 nov. 2013 à 10:44, Regis Depraute [regis.depraute@gmail.com](mailto:regis.depraute@gmail.com) a écrit : > I guess it's because of UDP or something? > Yes, it's UDP > > I'm going to add "redis" to make a buffer, do you think it's a good idea ? > Or it's better to use rabbitMQ ? > > How many shard do you have per index? How many index? > I have 5 shards > 7 index > each index weight +- 22 gb > > Do you have enough file descriptors? > What is file descriptors? > > Anything in logs? > I think I should increase RAM Memory : > > [2013-11-04 09:21:53,606][WARN][index.engine.robin] [node01] [logstash-2013.11.04][0] failed engine > java.lang.OutOfMemoryError: Java heap space > [2013-11-04 09:22:28,988][WARN][cluster.action.shard] [node01] sending failed shard for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]] > [2013-11-04 09:22:28,988][WARN][cluster.action.shard] [node01] received shard failed for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]] > > 6 Gb RAM means 3Gb for elasticsearch VM? > yes > > Le mardi 5 novembre 2013 09:47:59 UTC+1, David Pilato a écrit : > > > You should never loose any data. At least, your injection process should be aware that an index operation failed. > > And you can manage errors on a client level. > > > > I guess it's because of UDP or something? > > > > Unsure about your exact use case or architecture but you could consider to add another layer such as rabbitMQ to store first your documents in a queue and then consume them from rabbitMQ to elasticsearch although it's absolutely not something mandatory. Elasticsearch could perfectly manage your injection load if your architecture is well designed. > > > > How many shard do you have per index? How many index? > > Do you have enough file descriptors? > > > > Anything in logs? > > > > Also, I think you don't have enough memory to consume your data with Kibana as Kibana is using facets. 6 Gb RAM means 3Gb for elasticsearch VM? > > > > -- > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > > @dadoonet | @elasticsearchfr > > > > 5 novembre 2013 at 09:06:34, Regis Depraute ([regis.d...@gmail.com](mailto:regis.d...@gmail.com)) a écrit: > > > > > Hello, > > > > > > Is there a way to limit the cpu usage when i'm querying the elasticsearch database ? > > > I'm loosing data (new logs - incoming data) when the cluster is too busy in querying. > > > > > > RD. > > > > > > Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit : > > > > > > > Basically elasticsearch uses shards for read and write operations. > > > > You don't need to have some "passive" nodes. > > > > > > > > If you feel that you can't search fast enough then you should consider for example increase the number of replicas (and the number of nodes). > > > > > > > > SSD drives, more memory should help as well. > > > > May be having 1 shard per index or using routing could help. > > > > > > > > HTH > > > > > > > > -- > > > > David 😉 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 4 nov. 2013 à 15:03, Regis Depraute [regis.d...@gmail.com](mailto:regis.d...@gmail.com) a écrit : > > > > > > > > > Hello, > > > > > > > > > > I use Elasticsearch to store logs into a database, I receive approximately 50G of logs per day, > > > > > After logstash process, and elasticsearch storing, I have +- 22Go of logs per index. > > > > > > > > > > I have a problem with indexing and reading at the same time with elasticsearch. > > > > > Everything is working fine until kibana is querying into the elasticsearch database, (some queries last 15 minutes) and no more data is written (or nearly, sometimes i got 25% of the logs) > > > > > > > > > > What can I do to make the "readings" do not disturb "writings" ? > > > > > > > > > > The scheme is : > > > > > > > > > > logs goes to 512(UDP or TCP) =\> logstash =\> ES\_cluster \<= kibana(reader) > > > > > > > > > > I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) : > > > > > he is receiving logs on port 512 > > > > > and output is elasticsearch\_cluster (node03) > > > > > > > > > > config logstash : > > > > > input { > > > > > syslog { > > > > > type =\> "syslog" > > > > > format =\> "json" > > > > > port =\> 514 > > > > > } > > > > > } > > > > > filter { > > > > > grok { > > > > > ... too long ... > > > > > } > > > > > } > > > > > output { > > > > > elasticsearch { > > > > > host =\> "aaa.bbb.ccc.ddd" ---\>here is the ip address of the node03 > > > > > port =\> "9302" > > > > > node\_name =\> "logstash-node" > > > > > } > > > > > } > > > > > > > > > > I have an Elasticsearch cluster : (elasticsearch-0.90.5) : > > > > > node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9300 > > > > > node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9301 > > > > > node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: false port 9302 > > > > > > > > > > and a dedicated server for kibana: > > > > > he is reading logs into elasticsearch node03 > > > > > > > > > > every VM are CentOS(6.4) > > > > > > > > > > Thanks > > > > > RD. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > -- > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Regis\_Depraute](https://avatars.discourse-cdn.com/v4/letter/r/258eb7/32.png) [@Regis\_Depraute](https://discuss.elastic.co/u/Regis_Depraute)\ **Post date:** [November 5, 2013, 3:11pm UTC](https://discuss.elastic.co/t/what-can-i-do-to-make-the-readings-do-not-disturb-writings/14243/7 "2013-11-05T15:11:11Z")
Thanks 🙂 Le mardi 5 novembre 2013 13:35:28 UTC+1, David Pilato a écrit : > So add more memory or more nodes. > I think you don't need to "complicate" things with another layer. > > Check this webinar: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/webinars/elasticsearch-pre-flight-checklist/) > > It will help you a lot! > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 5 nov. 2013 à 10:44, Regis Depraute \<[regis.d...@gmail.com](mailto:regis.d...@gmail.com) \\> > a écrit : > > _I guess it's because of UDP or something?_ > Yes, it's UDP > > I'm going to add "redis" to make a buffer, do you think it's a good idea ? > Or it's better to use rabbitMQ ? > > _How many shard do you have per index? How many index?_ > I have 5 shards > 7 index > each index weight +- 22 gb > > _Do you have enough file descriptors?_ > What is file descriptors? > > _Anything in logs?_ > I think I should increase RAM Memory : > > [2013-11-04 09:21:53,606][WARN][index.engine.robin] [node01] [ > logstash-2013.11.04][0] failed engine > java.lang.OutOfMemoryError: Java heap space > [2013-11-04 09:22:28,988][WARN][cluster.action.shard] [node01]sending failed shard > for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED > ], reason [engine failure, message [OutOfMemoryError[Java heap space]]] > [2013-11-04 09:22:28,988][WARN][cluster.action.shard] [node01]received shard failed > for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED > ], reason [engine failure, message [OutOfMemoryError[Java heap space]]] > > _6 Gb RAM means 3Gb for elasticsearch VM?_ > yes > > Le mardi 5 novembre 2013 09:47:59 UTC+1, David Pilato a écrit : > > > You should never loose any data. At least, your injection process should > > be aware that an index operation failed. > > And you can manage errors on a client level. > > > > I guess it's because of UDP or something? > > > > Unsure about your exact use case or architecture but you could consider > > to add another layer such as rabbitMQ to store first your documents in a > > queue and then consume them from rabbitMQ to elasticsearch although it's > > absolutely not something mandatory. Elasticsearch could perfectly manage > > your injection load if your architecture is well designed. > > > > How many shard do you have per index? How many index? > > Do you have enough file descriptors? > > > > Anything in logs? > > > > Also, I think you don't have enough memory to consume your data with > > Kibana as Kibana is using facets. 6 Gb RAM means 3Gb for elasticsearch VM? > > > > -- > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > > > 5 novembre 2013 at 09:06:34, Regis Depraute ([regis.d...@gmail.com](mailto:regis.d...@gmail.com)) a > > écrit: > > > > Hello, > > > > Is there a way to limit the cpu usage when i'm querying the elasticsearch > > database ? > > I'm loosing data (new logs - incoming data) when the cluster is too busy > > in querying. > > > > RD. > > > > Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit : > > > > > Basically elasticsearch uses shards for read and write operations. > > > You don't need to have some "passive" nodes. > > > > > > If you feel that you can't search fast enough then you should consider > > > for example increase the number of replicas (and the number of nodes). > > > > > > SSD drives, more memory should help as well. > > > May be having 1 shard per index or using routing could help. > > > > > > HTH > > > > > > -- > > > David 😉 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 4 nov. 2013 à 15:03, Regis Depraute [regis.d...@gmail.com](mailto:regis.d...@gmail.com) a écrit : > > > > > > Hello, > > > > > > I use Elasticsearch to store logs into a database, I receive > > > approximately 50G of logs per day, > > > After logstash process, and elasticsearch storing, I have +- 22Go of > > > logs per index. > > > > > > I have a problem with indexing and reading at the same time with > > > elasticsearch. > > > Everything is working fine until kibana is querying into the > > > elasticsearch database, (some queries last 15 minutes) and no more data is > > > written (or nearly, sometimes i got 25% of the logs) > > > > > > _What can I do to make the "readings" do not disturb "writings" ?_ > > > > > > The scheme is : > > > > > > logs goes to 512(UDP or TCP) =\> logstash =\> ES\_cluster \<= kibana(reader) > > > > > > - _I have a server dedicated to logstash > > > (logstash-1.2.2-flatjar.jar) :_ > > > > > > he is receiving logs on port 512 > > > and output is elasticsearch\_cluster (node03) > > > > > > config logstash : > > > input { > > > syslog { > > > type =\> "syslog" > > > format =\> "json" > > > port =\> 514 > > > } > > > } > > > filter { > > > grok { > > > ... too long ... > > > } > > > } > > > output { > > > elasticsearch { > > > host =\> "aaa.bbb.ccc.ddd" ---\>here is the ip address of the node03 > > > port =\> "9302" > > > node\_name =\> "logstash-node" > > > } > > > } > > > > > > - _I have an Elasticsearch cluster : (elasticsearch-0.90.5) :_ > > > > > > node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true > > > port 9300 > > > node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true > > > port 9301 > > > node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: _false_port 9302 > > > > > > - _and a dedicated server for kibana:_ > > > > > > he is reading logs into elasticsearch node03 > > > > > > _every VM are CentOS(6.4)_ > > > \* > > > \* > > > Thanks > > > RD. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:08am UTC](https://discuss.elastic.co/t/what-can-i-do-to-make-the-readings-do-not-disturb-writings/14243/8 "2017-07-06T02:08:50Z")