# Elasticsearch inserting issue **URL:** \ **Category:** Elasticsearch\ **Created:** [June 16, 2014, 10:53am UTC](https://discuss.elastic.co/t/elasticsearch-inserting-issue/18128 "2014-06-16T10:53:10Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Ciprian\_2](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@Ciprian\_2](https://discuss.elastic.co/u/Ciprian_2)\ **Post date:** [June 16, 2014, 10:53am UTC](https://discuss.elastic.co/t/elasticsearch-inserting-issue/18128/1 "2014-06-16T10:53:10Z")
Hi, My application is reading data from a sqs and writtes the data in elasticsearch(alias ES) and mysql. We are writting in both for back-up(for the moment). This is done with 2 consumers with 5 workers. One worker is writing in mysql and ES. The app is written in java and supports inserting, deleting and updating documents(one by one and not bulk). The ES insertings and updatings are done with upsert(@update : a document may be updated if exists or inserted if it doesn't; @insert : insert the document or if the update is first done, we shouldn't do anything). The ES is installed on a different server(ubuntu; 8gb Ram; 4 cpu) than mysql. ES configuration : /etc/elasticsearch/elasticsearch.yml bootstrap.mlockall: true ES\_MIN\_MEM: 4g ES\_MAX\_MEM: 4g http.max\_initial\_line\_length : 48k /etc/default/elasticsearch ES\_HEAP\_SIZE=4g MAX\_LOCKED\_MEMORY=unlimited When I run in console - ulimit the answer is unlimited - ulimit -n the answer is 1024 The code for upsert : ``` client.prepareUpdate(indexListener, tableListener, suId) .setScript("ctx._source." + columnName + "=\"" + ``` columnValue + """) .setConsistencyLevel(WriteConsistencyLevel.DEFAULT).setRetryOnConflict(10) .setUpsert(map) .execute().actionGet(); Node configuration : _localhost:9200/\_nodes?pretty_ { "cluster\_name" : "test", "nodes" : { "Aa-bbb" : { "settings" : { "node" : { "name" : "testing" }, "bootstrap" : { "mlockall" : "true" }, "http" : { "max\_initial\_line\_length" : "48k" }, "ES\_MIN\_MEM" : "4g", "name" : "testing", "pidfile" : "/var/run/elasticsearch.pid", "path" : { "data" : "/var/lib/elasticsearch", "work" : "/tmp/elasticsearch", "home" : "/usr/share/elasticsearch", "conf" : "/etc/elasticsearch", "logs" : "/var/log/elasticsearch" }, "cluster" : { "name" : "test" }, "ES\_MAX\_MEM" : "4g", "config" : "/etc/elasticsearch/elasticsearch.yml" }, "os" : { "refresh\_interval" : 1000, "available\_processors" : 4, "cpu" : { "vendor" : "Intel", "model" : "Xeon", "mhz" : 2800, "total\_cores" : 4, "total\_sockets" : 4, "cores\_per\_socket" : 32, "cache\_size\_in\_bytes" : 25600 }, "mem" : { "total\_in\_bytes" : 7812386816 }, "swap" : { "total\_in\_bytes" : 0 } }, "process" : { "refresh\_interval" : 1000, "id" : 1111, "max\_file\_descriptors" : 65535, "mlockall" : true }, "jvm" : { "pid" : 1111, "version" : "1.7.0\_55", "vm\_name" : "OpenJDK 64-Bit Server VM", "vm\_version" : "vers", "vm\_vendor" : "Oracle Corporation", "start\_time" : 1402687303832, "mem" : { "heap\_init\_in\_bytes" : 4294967296, "heap\_max\_in\_bytes" : 4260102144, "non\_heap\_init\_in\_bytes" : 24313856, "non\_heap\_max\_in\_bytes" : 224395264, "direct\_max\_in\_bytes" : 4260102144 }, "gc\_collectors" : ["ParNew", "ConcurrentMarkSweep"], "memory\_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ] }, "thread\_pool" : { "generic" : { "type" : "cached", "keep\_alive" : "30s" }, "index" : { "type" : "fixed", "min" : 4, "max" : 4, "queue\_size" : "200" }, "get" : { "type" : "fixed", "min" : 4, "max" : 4, "queue\_size" : "1k" }, "snapshot" : { "type" : "scaling", "min" : 1, "max" : 2, "keep\_alive" : "5m" }, "merge" : { "type" : "scaling", "min" : 1, "max" : 2, "keep\_alive" : "5m" }, "suggest" : { "type" : "fixed", "min" : 4, "max" : 4, "queue\_size" : "1k" }, "bulk" : { "type" : "fixed", "min" : 4, "max" : 4, "queue\_size" : "50" }, "optimize" : { "type" : "fixed", "min" : 1, "max" : 1 }, "warmer" : { "type" : "scaling", "min" : 1, "max" : 2, "keep\_alive" : "5m" }, "flush" : { "type" : "scaling", "min" : 1, "max" : 2, "keep\_alive" : "5m" }, "search" : { "type" : "fixed", "min" : 12, "max" : 12, "queue\_size" : "1k" }, "percolate" : { "type" : "fixed", "min" : 4, "max" : 4, "queue\_size" : "1k" }, "management" : { "type" : "scaling", "min" : 1, "max" : 5, "keep\_alive" : "5m" }, "refresh" : { "type" : "scaling", "min" : 1, "max" : 2, "keep\_alive" : "5m" } }, "network" : { "refresh\_interval" : 5000, ``` } ``` } } ``` The problem is that we have a lot of messages in sqs that are not ``` consumed(~20000). If I'm going to stop the Es, the documents are read ok. Is there any configuration that I've missed it for ES? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/875319d3-861a-4799-8527-fee2ca5d9b6c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/875319d3-861a-4799-8527-fee2ca5d9b6c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:22am UTC](https://discuss.elastic.co/t/elasticsearch-inserting-issue/18128/2 "2017-07-06T01:22:17Z")
Closed on July 6, 2017, 1:22am UTC