# Improve performance of Logstash data loading into ES **URL:** \ **Category:** Logstash\ **Created:** [June 9, 2017, 10:33am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828 "2017-06-09T10:33:08Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![udaykirankona](https://avatars.discourse-cdn.com/v4/letter/u/ac91a4/32.png) [@udaykirankona](https://discuss.elastic.co/u/udaykirankona)\ **Post date:** [June 9, 2017, 10:33am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/1 "2017-06-09T10:33:08Z")
HI, I am trying to load data into elasticsearch using logstash, but its loading very slow. 80000/minute into elasticsearch. So, its taking 10 mins for 11GB data. I have requirement to reduce this to 10 secs. Please help me what all changes I need to do in ELK and in system Hardware . The following is my Logstash config input { file { type =\> "engine" path =\> "Path/\*" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" ignore\_older =\> 0 codec =\> multiline { pattern =\> "^qns|^lb0" negate =\> true what =\> "previous" } } } filter { if [type] == "engine" { mutate { gsub =\> ["message", "\n", ""] } mutate { gsub =\> ["message", "\r", ""] } mutate { gsub =\> ["message", "\t", ""] } mutate { gsub =\> ["message", "=", ""] } grok { break\_on\_match =\> false match =\> ["message", "%{WORD:VM} [%{TIMESTAMP\_ISO8601:Timestamp}] %{GREEDYDATA:Record}", "Record", "POLICY RESULT ERROR: %{DATA:Policy\_Result\_Error}session action %{GREEDYDATA:Remaining}"] } mutate { add\_field =\> { "Policy\_Result\_Error\_Analyzed" =\> "%{Policy\_Result\_Error}" } remove\_field =\> ["message","Remaining"] } date { match =\> ["Timestamp" , "yyyy-MM-dd HH:mm:ss,SSS"] } } } output { if [type] == "engine" { # stdout { codec =\> rubydebug } ``` elasticsearch { hosts => "IP:9200" index => "hathway_test_engine" workers => 40 flush_size => 1000 } ``` } } **I have 3-node cluster in AWS** **My system hardware config of each node as follows** **224 gb RAM, 30 GB Hrd disk with 32-core** I did following changes in /etc/default/logstash **Arguments to pass to logstash agent** **LS\_OPTS="-b 1000 -w 40"** **Arguments to pass to java** **LS\_HEAP\_SIZE="128g"** **LS\_JAVA\_OPTS="-Djava.io.tmpdir=$HOME"** **LS\_JAVA\_OPTS="-Xmx100g -Xms4G"** also in bin/logstash **LS\_HEAP\_SIZE="xxx" size for the -Xmx${LS\_HEAP\_SIZE} maximum Java heap size option, default is "1g"** **LS\_HEAP\_SIZE="128g"** **LS\_JAVA\_OPTS="xxx" to append extra options to the defaults JAVA\_OPTS provided by logstash** **LS\_JAVA\_OPTS="-Xmx100g -Xms4g"** but still i don't see any improvement in logstash performance. The following are screen shots of CPU/Memory/IO when I was running Logstash, Instance-1 ![](https://us1.discourse-cdn.com/elastic/original/3X/3/b/3bbe54044258c421787d43f7d83829875829afe7.png) Instance-2 ![](https://us1.discourse-cdn.com/elastic/original/3X/0/8/0843df3d7424b62f68c751281baf57699380d4af.png) Instance-3 ![](https://us1.discourse-cdn.com/elastic/original/3X/3/0/304da0128895be986e5b48c1fc989f0f6567219e.png) Instance-1/CPU ![](https://us1.discourse-cdn.com/elastic/original/3X/0/b/0bd6e0aba02d7184c4e7f913b2b5ca60f65093f1.png) Instance-3/CPU ![](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d955e177c4a9ea6a64ff231e57d8ffa6353ba38d.png) Instance-2/CPU ![](https://us1.discourse-cdn.com/elastic/original/3X/6/5/654b8927f341f75dc9589d5eec55e493e3b6fb68.png) Instance-1/Memory ![](https://us1.discourse-cdn.com/elastic/original/3X/3/4/349f984c2db1563cd455a943745fc81c5569853e.png) Instance-3/memory ![](https://us1.discourse-cdn.com/elastic/original/3X/1/9/19ec7f65703893feead5f3f9e7d6e5a245318c32.png) Instance-2/Memory ![](https://us1.discourse-cdn.com/elastic/original/3X/d/a/da86968045e60d1a7178ae82dbc4918aa4a44c57.png) Instance-1/IOSTAT ![](https://us1.discourse-cdn.com/elastic/original/3X/f/8/f808141acc88cf0012c023c79f27b5aa329b8a9a.png) Instance-2/IOSTAT ![](https://us1.discourse-cdn.com/elastic/original/3X/4/6/46c462892cd754d934c83388f4745f6dc1a3a966.png) Instance-3/IOSTAT ![](https://us1.discourse-cdn.com/elastic/original/3X/e/7/e7cc8309ce0ac8a7a7cb0916823b1238d247f51d.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 9, 2017, 11:28am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/2 "2017-06-09T11:28:57Z")
How many files is your data spread across? What does you data look like? Which version of Logstash and Elasticsearch are you using? It also looks to me like you are deleting fields that are never created. Does this indicate that what you have posted is only a partial config? ---
**Author:** ![udaykirankona](https://avatars.discourse-cdn.com/v4/letter/u/ac91a4/32.png) [@udaykirankona](https://discuss.elastic.co/u/udaykirankona)\ **Post date:** [June 9, 2017, 12:02pm UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/3 "2017-06-09T12:02:19Z")
Hi Christian, Thanks for your quick reply, I am using 2.4 version of both Logstash and ES. I have total 11 file each 1GB and remove field thing was by mistake I modified that. My log is multiline log, it look like as follows, qns03 [2017-03-17 16:25:25,315] =============================================== POLICY RESULT ERROR: :Login-User\_SUSPENDED session action = Create credential = sk.157 domainId = Default subscriberId = 00235000e4b0dc4458bf6c6b SERVICES: FUP-D5M-U5M-T-D1M-U1M TRIGGER: com.broadhop.radius.messages.impl.RadiusAccessRequestMessage request: NAS-PORT = 0 CISCO-AVPAIR = client-mac-address=e46f.13c1.f346 NAS-PORT-ID = 0/3/0/505 FRAMED-PROTOCOL = 1 NAS-PORT-TYPE = 5 CALLING-STATION-ID = e4-6f-13-c1-f3-46 NAS-IP-ADDRESS = 202.88.216.192 USER-NAME = sk.157 SERVICE-TYPE = 2 USER-PASSWORD = xxxxxxxxx ACCT-SESSION-ID = 02413493 NAS-IDENTIFIER = Noida\_ISG-2.hathway DEBUG MSGS: INFO : (core) Tagging message with ID: com.broadhop.radius.impl.devicemanager.IsgNetworkDeviceManager INFO : (radius) RADIUS device group assigned to: Noida ISG-2 INFO : (radius) Checking for duplicates by MAC Address E46F.13C1.F346 INFO : (core) Start session triggered INFO : (core) Stop session triggered INFO : (FTTX Policy) FTTX NASPORTID 0/3/0/505 INFO : (auth) TAL not attempted no TAL credential found for user INFO : (auth) Success USUM\_AUTHORIZATION INFO : (core) Switching credential id to sk.157 for session INFO : (service) Balance processing enabled for subscriber INFO : (action) Updated credential: E46F.13C1.F346, with last used date avp ERROR : (balance) OneTimeUsageCharge has a source field of: RadiusAccountingTotalBytesRetriever which returned a non-numeric value: null INFO : (balance) Error found, rolling back transaction ERROR : (core) Error processing policy request: :Login-User\_SUSPENDED ERROR : (core) Error processing policy request: :Login-User\_SUSPENDED SENT MESSAGES (asynchronous): Thanks, Uday.K ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 9, 2017, 12:11pm UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/4 "2017-06-09T12:11:05Z")
Given that you have a few potentially CPU intensive gsub and grok filters in your config, I am surprised that Logstash is not using more CPU. Might it be that you are limited by the performance of the file input plugin? You might be able to check this by setting up a separate file input for each file and see if that makes any difference. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 9, 2017, 12:30pm UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/5 "2017-06-09T12:30:50Z")
> So, its taking 10 mins for 11GB data. I have requirement to reduce this to 10 secs. That's 1 GB per second. That's... a lot for a three-node ES cluster. ---
**Author:** ![udaykirankona](https://avatars.discourse-cdn.com/v4/letter/u/ac91a4/32.png) [@udaykirankona](https://discuss.elastic.co/u/udaykirankona)\ **Post date:** [June 9, 2017, 3:43pm UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/6 "2017-06-09T15:43:31Z")
Hi Magnus, Thanks for you quick response. can we reduce loading time to less than 10 mins ? If yes, what changes (Either in my hardware/config changes)I need to do for that? ---
**Author:** ![udaykirankona](https://avatars.discourse-cdn.com/v4/letter/u/ac91a4/32.png) [@udaykirankona](https://discuss.elastic.co/u/udaykirankona)\ **Post date:** [June 12, 2017, 5:47am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/7 "2017-06-12T05:47:11Z")
HI Christian, Thanks for your response, You mean to create 10 inputs for 10 file, in Logsatsh configuration and test CPU usage. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 12, 2017, 5:48am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/8 "2017-06-12T05:48:14Z")
> can we reduce loading time to less than 10 mins ? Maybe. What's your current bottleneck? For example, if you have nothing but your file input in your Logstash configuration, what event rate are you getting? How many input files do you have? If the number of files is too small it's hard to parallelize the reading of them and then it'll be hard to improve the performance. Ultimately you might be in the hands of the multiline codec and its configuration. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 12, 2017, 6:24am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/9 "2017-06-12T06:24:12Z")
Yes, I would try creating a separate file input block per file to see if that makes a difference. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 12, 2017, 8:02am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/10 "2017-06-12T08:02:28Z")
Ah yes—you definitely want to have one file input per file so that Logstash at least parallelizes the reading of each file instead of reading from them serially. ---
**Author:** ![udaykirankona](https://avatars.discourse-cdn.com/v4/letter/u/ac91a4/32.png) [@udaykirankona](https://discuss.elastic.co/u/udaykirankona)\ **Post date:** [June 12, 2017, 9:40am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/11 "2017-06-12T09:40:47Z")
Hi Christian/Magnus, Thanks for your response. If we give separate input for every file on same path, How this gonna work in real time. everytime the path is updated with new files. How can we read automatically with out manual intervention. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 12, 2017, 9:42am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/12 "2017-06-12T09:42:31Z")
> If we give separate input for every file on same path, How this gonna work in real time. everytime the path is updated with new files. Well, in that case you have to generate a new configuration file and reload Logstash. If you want the fastest possible ingestion of files you shouldn't use wildcards in the file input's `path` option. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 12, 2017, 9:43am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/13 "2017-06-12T09:43:33Z")
I did not necessarily suggest this as a final solution, but rather as a test to see if the level of parallelism for inputs is an issue. ---
**Author:** ![Jay\_Dihenkar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jay_dihenkar/32/26375_2.png) [@Jay\_Dihenkar](https://discuss.elastic.co/u/Jay_Dihenkar)\ **Post date:** [June 16, 2017, 7:36am UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/14 "2017-06-16T07:36:53Z")
A little more optimization if you're looking for... - let filebeat handle multiline - let logstash not do multiline parsing at all and use more workers on LS - increase the batch size/ worker on es output [till es can take :o - dangerous] - let filebeat handle regex for filename [like toload.\*.log for toload.1.log etc....] that will be one time setup and you keep adding new files and removing the older ones. ---
**Author:** ![Ranjith\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjith_m/32/19272_2.png) [@Ranjith\_M](https://discuss.elastic.co/u/Ranjith_M)\ **Post date:** [June 19, 2017, 3:31pm UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/15 "2017-06-19T15:31:56Z")
Apart from what Jay said, one additional setting is to make sure shards are distributed across different nodes. Which will utilize 3 nodes which will process in parallel, instead you have all the primary shards on the same node. ---
**Author:** ![udaykirankona](https://avatars.discourse-cdn.com/v4/letter/u/ac91a4/32.png) [@udaykirankona](https://discuss.elastic.co/u/udaykirankona)\ **Post date:** [June 20, 2017, 1:51pm UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/16 "2017-06-20T13:51:03Z")
Hi Jay/Ranjith, Thanks for your replay. I will try with that. Thanks, Uday.K ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 18, 2017, 1:51pm UTC](https://discuss.elastic.co/t/improve-performance-of-logstash-data-loading-into-es/88828/17 "2017-07-18T13:51:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.