# Problem Performance Elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [March 22, 2017, 5:53pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632 "2017-03-22T17:53:27Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [March 22, 2017, 5:53pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/1 "2017-03-22T17:53:27Z")
Hello, I have a problem with elasticsearch... My logs are upload very very slow. Here the context : OS : RH 7.1 12 CPU 32 Go RAM. Advanced config : Memlock true, swap desactivation. 1600 files for a total of 55 000 000 of lines. I have tested a scenario and, It's upload 20 000 lines per minutes ! It's very poor, i have 55 000 000 lines, so It would take about 45 hours to upload it ! What do you think about that ? @ruflin @Christian_Dahlqvist ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 22, 2017, 6:00pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/2 "2017-03-22T18:00:40Z")
How are you uploading? What does your configuration look like? Which version are you on? ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [March 23, 2017, 9:17am UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/3 "2017-03-23T09:17:57Z")
So, Here an example of line : 1490090124,ASR\_01,0003diamproxy;14318488;22688;58d0f,REQUEST,47845979,47845979,47845979,20801,0709945330,CCBOS,,0,,,,,,,,0,POS,,,,,,,,,,0,950976,,MAXAGE,4836480,0 Here the pattern i use for it : ``` if [type] == "log" { if [message] =~ "\bCTEName\b" { drop { } } else { csv { columns => [" ...~ about 45 fields "] separator => ";" } } mutate { convert => { "log_UsedOctets" => "integer" "log_TotalOctets" => "integer" } remove_field => ["message", "log_ID", "log_Flag", [...]" ] } ``` I use ELK stack on version 5.1 ! Thank you for your help Annexe (graph of upload of log) : ![](https://us1.discourse-cdn.com/elastic/original/3X/8/4/84438d27aefed9cd83e89879315eb7848a7e3c59.JPG) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2017, 9:24am UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/4 "2017-03-23T09:24:53Z")
> [@Beuhlet\_Reseau](#): > > separator =\> ";" It looks like this primarily is a comma separate list. Why use ';' as a separator? Are the events looking as you expect? What does the rest of your logstash config look like? ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [March 23, 2017, 10:42am UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/5 "2017-03-23T10:42:33Z")
I don't understand, each field are separate by ',' so, i indicate just the separator ? So, for the input : ``` input { beats { port => 5044 ssl => true ssl_certificate => "/etc/ssl/logstash-forwarder.crt" ssl_key => "/etc/ssl/logstash-forwarder.key" } } ``` and Output : ``` output { stdout { codec => rubydebug } if [type] == "log" { elasticsearch { hosts => ["localhost:9200"] index => "log--%{+YYYY.MM.dd}" } } } ``` Edit : Is the ssl which slow parse ? Maybe, it's not necessary to index each field (i believe it's analyzed) ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2017, 10:54am UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/6 "2017-03-23T10:54:00Z")
> [@Beuhlet\_Reseau](#): > > stdout { codec =\> rubydebug } The stdout block can slow down processing throughput. You may also want to increase the [internal batch size](https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html) to see if that has any impact. > [@Beuhlet\_Reseau](#): > > I don't understand, each field are separate by ',' so, i indicate just the separator ? If you inspect the events in Discover node, do they look like you expect? ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [March 23, 2017, 2:32pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/7 "2017-03-23T14:32:33Z")
> [@Christian\_Dahlqvist](#): > > The stdout block can slow down processing throughput. You may also want to increase the internal batch size to see if that has any impact. Ok i will try to delete stdout block. Hummm increase the internal batch size ? How 😄 Can it be written to a configuration file for eternity ? > [@Christian\_Dahlqvist](#): > > If you inspect the events in Discover node, do they look like you expect? I will see but, to make graphic i must specifiy few fields in integer to do a sum ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2017, 2:40pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/8 "2017-03-23T14:40:44Z")
> [@Beuhlet\_Reseau](#): > > I will see but, to make graphic i must specifiy few fields in integer to do a sum You can view the raw documents in 'Discover' mode. ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [March 23, 2017, 4:10pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/9 "2017-03-23T16:10:38Z")
Oh thank you a lots @Christian_Dahlqvist I went from 20 000 to 180 000 lines per minutes. (without modify batch size or other conf) ! ☀ I have a question about the configuration : -b : pipeline.batch.size SIZE : default is 125. On the doc, he talks about passing it around 3,000, what dou think ? Other conf is : -w : --pipeline.workers COUNT : default is 5, but i read that the value must be equal to number of core (so 12 in my case) I see too that the values can be specified in logstash.yml ?: ``` # pipeline.workers: 12 # pipeline.batch.size: 3000 ``` I will increase size batch 125 to 3 000 and workers to 12 (125 ans 2 previously) Can you explain me what is options (In a fairly simple way?) As a reminder, I have only one file in /etc/logstash/conf.d/logstash.conf, and inside i have, for the time being, 2 type of treatment but This will increase. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2017, 4:20pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/10 "2017-03-23T16:20:32Z")
Have you gone through the following [guide](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/tune-for-indexing-speed.html#_disable_refresh_and_replicas_for_initial_loads) for Elasticsearch tuning? ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [March 23, 2017, 10:45pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/11 "2017-03-23T22:45:08Z")
Just a word, perfect. @Christian_Dahlqvist ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [March 24, 2017, 1:04pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/12 "2017-03-24T13:04:36Z")
A last things, the auto generate ID ? It's by default in elasticsearch or i must activate it somewhere ? EDIT @Christian_Dahlqvist : [https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html#\_use\_the\_smallest\_numeric\_type\_that\_is\_sufficient](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html#_use_the_smallest_numeric_type_that_is_sufficient) I ask myself a question about the use of integer type. In the logstash's filter, If I do not define field type, elastic store it as a string. If i use convert =\> "integer", it's store as a long. (If i put =\> "short", logstash return me an error... grrrr) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 24, 2017, 1:11pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/13 "2017-03-24T13:11:39Z")
If you do not explicitly provide an id, one will be auto-generated for you. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 24, 2017, 1:29pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/14 "2017-03-24T13:29:39Z")
> [@Beuhlet\_Reseau](#): > > I ask myself a question about the use of integer type. In the logstash's filter, If I do not define field type, elastic store it as a string. If i use convert =\> "integer", it's store as a long. This is all handled through [index templates in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/indices-templates.html). ---
**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Post date:** [March 24, 2017, 3:54pm UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/16 "2017-03-24T15:54:18Z")
@Christian_Dahlqvist I tried to update setting by the elasticsearch.yml file but at start ES return this error : ``` Found index level settings on node level configuration. Since elasticsearch 5.x index level settings can NOT be set on the nodes configuration like the elasticsearch.yaml ``` So, i understand that settings must be udapte by PUT commands. _(I have not yet created an index, i purged the BDD to start from 0)_ I have succeeded in modify number\_of\_replicas and refresh\_interval by this command line in linux shell : ``` curl -XPUT 'localhost:9200/_settings' -d ' { "index" : { "refresh_interval" : -1 } }' ``` It's ok, it return me acknoledge true. But I tested to surrender refresh interval to 30 but i have this message : ("refresh\_interval" : 30) ``` {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"failed to parse setting [index.refresh_interval] ``` Why ? Also, i tested to set memory buffer size to 1,2GB with : "memory.index\_buffer\_size" : 1,2G It's return me : ``` {"error":{"root_cause":[{"type":"settings_exception","reason":"Failed to load settings ``` It's complicated .... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 25, 2017, 8:50am UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/17 "2017-03-25T08:50:56Z")
What you tried in your latest post do not seem to match anything described in the link I provided. Did you read it? This is however a completely different question, unrelated to the initial performance issue, so I would recommend creating a new issue instead of continuing discussing various topics in this thread. That will make it a lot easier for others to find appropriate topics when searching the forum. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 22, 2017, 8:51am UTC](https://discuss.elastic.co/t/problem-performance-elasticsearch/79632/18 "2017-04-22T08:51:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.