Inserts in Ver 2.0.0 extremly slow

Elastic Stack Elasticsearch
1

I ran a Jmeter test to insert 2k size documents in version 1.6.0 and 2.0.0 and difference in insert rates is quite significant. Cluster had 5 data nodes with 8 cores 16GB, 3 master nodes with same cpu/memory/storage and ES config across tests. Max rate achieved in 1.6.0 is much better. and total time taken to complete full inserts is also more than 3 time of version 1.6.0. See the max rated achieved from test logs.

Here are the rates reported from test:
in ES version 2.0.0
summary + 3804 in 12.3s = 310.5/s Avg: 111 Min: 3 Max: 1238 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 1829 in 10.1s = 180.3/s Avg: 182 Min: 3 Max: 1302 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 1926 in 10.4s = 184.6/s Avg: 178 Min: 3 Max: 1305 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 2972 in 12s = 247.5/s Avg: 138 Min: 3 Max: 1325 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 3528 in 13s = 280.3/s Avg: 124 Min: 3 Max: 1270 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 2025 in 11s = 186.4/s Avg: 179 Min: 3 Max: 1329 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 1454 in 9.4s = 154.2/s Avg: 206 Min: 3 Max: 1791 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 2163 in 11.4s = 189.3/s Avg: 177 Min: 3 Max: 1258 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 2332 in 11.3s = 206.2/s Avg: 163 Min: 3 Max: 1281 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 2249 in 12.2s = 184.9/s Avg: 188 Min: 3 Max: 1310 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary + 5676 in 31.1s = 182.4/s Avg: 238 Min: 3 Max: 1316 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary = 9480 in 42.2s = 224.4/s Avg: 187 Min: 3 Max: 1316 Err: 0 (0.00%)
summary + 5989 in 31.1s = 192.8/s Avg: 224 Min: 2 Max: 1330 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary = 7818 in 40.1s = 194.7/s Avg: 214 Min: 2 Max: 1330 Err: 0 (0.00%)
summary + 5591 in 31.2s = 179.2/s Avg: 241 Min: 2 Max: 1317 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0

in ES version 1.6.0

summary = 508586 in 676s = 752.5/s Avg: 59 Min: 3 Max: 7091 Err: 0 (0.00%)
summary + 20964 in 31s = 676.1/s Avg: 64 Min: 3 Max: 3050 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary = 519142 in 678s = 766.1/s Avg: 58 Min: 2 Max: 4069 Err: 0 (0.00%)
summary + 22072 in 31s = 713.4/s Avg: 60 Min: 3 Max: 3071 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary = 516133 in 678s = 761.3/s Avg: 58 Min: 2 Max: 7041 Err: 0 (0.00%)
summary + 22858 in 31s = 745.8/s Avg: 59 Min: 3 Max: 3055 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary = 525520 in 677s = 776.4/s Avg: 57 Min: 2 Max: 7134 Err: 0 (0.00%)
summary + 21756 in 31s = 702.9/s Avg: 61 Min: 3 Max: 3267 Err: 0 (0.00%) Active: 45 Started: 45 Finished: 0
summary = 500759 in 677s = 740.2/s Avg: 60 Min: 3 Max: 7132 Err: 0 (0.00%)

2

Could you provide an example document?

Could you use ``` to fence things like the jmeter output and the document so the forum renders them in a fixed width font? That is super helpful!

3

Document layout in Jmeter

: "${DEV_NAME}" , "location": "${LOC}", "app_id":"${APP}","vin":&
quot;${VIN}", "org_id":"${ORG}", "category":"${CAT}","standard":"
${STD}" ,"date_created":"${DATE}", "fuel_level": "${FUEL}","current_locatio
n&quot;:&quot;${CLOC}&quot;  ,&quot;Junk_1k&quot; : &quot;${JUNK_1K}&quot;   }</stringProp>```

Sample data record:
```A19080|in molestie|tellus sem|-51.01699, -40.79354|UCW53ALG0SSeKc1|LWFWB99806561IdJFCNPPFU12816|HEYZTBKAV-99244-UoEAiE-613648-ouUIeOaIo415924911|adipiscing. Mauris molestie pharetra nibh.|Accumsan Sed Industries|1420798802|89|-42.48112, -92.03784

one field in the document are 2048 bytes (JUNK_1K) and Jmeter test generates random string. So overall document size about 2k+bytes.

4

Is there any chance you could turn that into an example document? I can't really tell what is up for all the escaping.

What kind of bulk size are you using?

5

Uploaded as image from HQ display (contains all fields except for the large string of 2K in the document.
It is not attempting bulk insert but multi threaded clients inserting one document at a time.(total of 450 threads in parallel, 10 clients 45 threads each). In version 1.6.0 , 45 milliion documents were inserted in about 100 minutes.

es2_insert.jpg960×720 71.3 KB

6

Here is json formatted one record.

          "standard": [
            "Lacinia Ltd"
          ],
          "device_name": [
            "pede. Cras"
          ],
          "fuel_level": [
            "54"
          ],
          "Junk_1k": [
            "FE4BbCow9edRQ6Gt-F1Qg1Cc2VOmbEYVG5jwQ=EuA3Vsg12Udd5d6N+EPN=ZT1wjOORP+GdiJiS9keZ6iZ++b27dCvK4gkJpuohlAlTWBdzaxxEnQpr3KqgBGt+64miWx=F4O42y8HioGjnuZpn8qXO7O=WC4N82fo7Dh=T05a3vEXqJGwbvpV0X-fsYALCFePWU-vmCmhAOoE1-z7ISoX7ix8sheuLEb-=JndsCRMP4hQgfhzFGCjhqtRVNQlqJ+znQ7LHg6axUNAJ+5hM=NL8Sb9OPSH7cbE8RHs73BaOJ9ewlqijjTL5patTDrw7TfbaBlRNEZLJpfoXM3doIpQeTH1ThwqSP2k0TS2lkCY-5r3Pw0RbYI0Co4n=Zp7Bql4vdcx20ewmuPgLxs02vKYYexdb086dSiLWSbPYIF2wf=+9ne6F1N8VCfy8=iRWXZpFRWeR6=tdkBcUXykRIa0tZMu9NyGSSHkHNL+osTSRP3ny5ON05jSps539qBthWms66CcP4R8zr55Z7QXNWMqGQonqV1PiHNX33ltl9uiooi4HQjnXaJAXxA4S7V37tTm5C-xLGCsLk+4MsMCqT3MNmibN1EPudkowMfEDbu1k06XTvz8wzYAu2npfGfAIDoR-kKPa6O7PniQ8rB2U3enalGrL-2RJAV1910aclS=pDsau=PAZMt1aqtQ0nqSl3AS+iZlt0ND-wGZnjpPm8YFNnE4-abo5Wh63L06mw2t5hrpqWhUOFJiiXgvFP4wn8eOIjUvlU6mQjgRCWqNUaittPXqxhNgJg2=IubAhMa78tvQSj7dt-dSW2+y3+L1bipaHqk0dqLjDhwFx-b=IiK2WZXHG+jM8oxWhNiAUyXp+1blrGSJd7WLKUQlUn3cYRSHwuI50VZF5bHn5ZuBEwWEmQx6hzEw=zkQ9EmhhnajJ0-LZggbptI3JtZBCg9uCmbp0Moaamd03KKJ+RUug+l1hspLmf8Cf8G6aub3=uJLSwf8-ucSie3yNPSaX+guevM5OONG5FIiJp4PMHqrdCh+w-kr9-gVqHQRWjndLypryK1UYOxZQzUX8YRFVgBllA6SJlUxhX=OppWLRvK+A-mjz=zbI-7QlhZqUQBQnFmOdUAEFm8-OsQWh39r6PXxze2CXmlNgtlzXGXXdYHGzXIgfOde8FQCkTDljlvr4ZDJtZ+zHrFwxCwxSj4fiW3Ccsdl0NHw+LOZKTARpZT-3jVVqHimmAE7TZ5wOTYMcu9zUA4IRSwjksCPxEejtKUduOJrDLwzMW5EUJnWflobCsiyeIH33WO7SQ4fS53ZlHfyCDjrp7zfP=rrH9L8bJtBrh5GsmZHQX=-pJunvx=9FY++48ZqEbmRwsrVHv6XihyqtQeHjm-n1XVBNHMhhCSBcdJZgJa53CHlRGzyrkWsd12BVGV4DGQ0VDbiKrArm6Ef+nDiSIJH=yS0sAm5kQpgX1SMTgV3vkHREVTE9mA75JfmmDIDTIdcwgDzGXjIie-QTI3TmVGLWr90=vvzZXcvQyJnqFgaoqdVeZl9KhgQwPEjNg7lJWru2Zgply8HDw=7tsIaUJweMOECo2sN83IZluATTY39NSvw-gYK1MhL7w4r5JyRpomELUb8rNL=ydsr+BHvq+1TPsaz=9viJw39UstN=2tLtRXIbStu+JyS4lAK3kvIoALHQpCPB8FfB0WOeWYovPTaxv0QbZzAx+P4Kw4TiYflxkTiod2aRf5iSSr0Tde6RtolKrBOeWoybgTr5IcB2nAxxpRzKe7-sCL-Z=43rIr6BSWYKt7hzpnwoC+Kd8bLRREZiUslY3=5li4HznvAUtyWg7TadMmNX4ruhaXl4r8dsYfSCAdKx90h0j3HD8VKfxvz9WagIOEj4F2Z0yantapPtoi4cLmb+Cnb-F=6dsPNSzh+GmnR8EzJSJFriVzq-PE8oUQU5d9q4oH0Mzm3r+U-xO+AuwAtC5tRog3lzFHRPauAb-k3auqPFnCTNdRY4yP=cXHzBBDUneotvo3GEo5CnGBjsEPPBufr6p"
          ],
          "date_created": [
            "1431474266"
          ],
          "org_id": [
            "CIEMKHUCQ-34433-oUiAEA-797281-aoeUiAIii195763297"
          ],
          "device_type": [
            "aliquet molestie"
          ],
          "location": [
            "66.56991, 136.55521"
          ],
          "vin": [
            "FLDXF93707719aFX0MYIZEE24126"
          ],
          "category": [
            "Integer in magna. Phasellus dolor"
          ],
          "app_id": [
            "EEZ28VMY0VWoZFA"
          ],
          "current_location": [
            "-84.6035, -103.11874"
          ]
        },
7

Much better.

I certainly expect to see some performance degradation over 1.6.0 when you do single inserts because of fsync. See the release notes. But I don't expect to see that big a difference.

I'm curious what you see with reasonable bulk sizes - 50 or 100 docs at a time. The biggest performance gains on insertion are using bulk and raising the (refresh_interval)[Update index settings API | Elasticsearch Guide [8.11] | Elastic] to 30s or -1.

When we ran the numbers for the performance degradation for the fsync when you use large-ish bulks we saw around 7%.

8

currently performance degradation is substantial with identical setup except for ES version difference. From 1Hr40 minutes for 45m docs insert in 1.6 to nearly 6-7 hrs in 2.0. Any other settings that can tried to tweak the insert performance?

9

You'll need to use bulk inserts to get similar performance.