Okay, I changed 30m to 5s. Memory usage was in bounds, I checked it.
Your version does not explain why restarting scrolling makes it fast
again. I don't think es could guess that previous scrolling is over.
I didn't get this example from anywhere, I wrote it myself. The first
version was in php, but it was very slow, so I decided to practice
golang and wrote this one. This one as slow as previous one.
Btw, results for 5s (no surprise here):
first request: 5000
fetched: 10000 , time ms: 190
fetched: 15000 , time ms: 255
fetched: 20000 , time ms: 197
fetched: 25000 , time ms: 282
fetched: 30000 , time ms: 211
fetched: 35000 , time ms: 213
fetched: 40000 , time ms: 219
fetched: 45000 , time ms: 223
fetched: 50000 , time ms: 231
fetched: 55000 , time ms: 235
fetched: 60000 , time ms: 336
fetched: 65000 , time ms: 344
fetched: 70000 , time ms: 347
fetched: 75000 , time ms: 256
fetched: 80000 , time ms: 357
fetched: 85000 , time ms: 369
fetched: 90000 , time ms: 289
fetched: 95000 , time ms: 305
fetched: 100000 , time ms: 306
fetched: 105000 , time ms: 402
fetched: 110000 , time ms: 296
fetched: 115000 , time ms: 342
fetched: 120000 , time ms: 432
fetched: 125000 , time ms: 430
fetched: 130000 , time ms: 324
fetched: 135000 , time ms: 460
fetched: 140000 , time ms: 352
fetched: 145000 , time ms: 464
fetched: 150000 , time ms: 508
fetched: 155000 , time ms: 523
fetched: 160000 , time ms: 359
fetched: 165000 , time ms: 366
fetched: 170000 , time ms: 409
fetched: 175000 , time ms: 375
fetched: 180000 , time ms: 536
fetched: 185000 , time ms: 392
fetched: 190000 , time ms: 418
fetched: 195000 , time ms: 561
fetched: 200000 , time ms: 418
fetched: 205000 , time ms: 569
fetched: 210000 , time ms: 589
fetched: 215000 , time ms: 460
fetched: 220000 , time ms: 591
fetched: 225000 , time ms: 454
fetched: 230000 , time ms: 486
fetched: 235000 , time ms: 453
fetched: 240000 , time ms: 633
fetched: 245000 , time ms: 656
fetched: 250000 , time ms: 656
fetched: 255000 , time ms: 664
fetched: 260000 , time ms: 488
fetched: 265000 , time ms: 693
fetched: 270000 , time ms: 530
fetched: 275000 , time ms: 510
fetched: 280000 , time ms: 715
fetched: 285000 , time ms: 526
fetched: 290000 , time ms: 578
fetched: 295000 , time ms: 539
fetched: 300000 , time ms: 745
fetched: 305000 , time ms: 757
fetched: 310000 , time ms: 582
fetched: 315000 , time ms: 767
fetched: 320000 , time ms: 628
fetched: 325000 , time ms: 810
fetched: 330000 , time ms: 599
fetched: 335000 , time ms: 615
fetched: 340000 , time ms: 815
fetched: 345000 , time ms: 599
fetched: 350000 , time ms: 660
fetched: 355000 , time ms: 612
fetched: 360000 , time ms: 661
fetched: 365000 , time ms: 894
fetched: 370000 , time ms: 1058
fetched: 375000 , time ms: 1112
fetched: 380000 , time ms: 1128
fetched: 385000 , time ms: 1127
fetched: 390000 , time ms: 1153
fetched: 395000 , time ms: 981
fetched: 400000 , time ms: 912
fetched: 405000 , time ms: 958
fetched: 410000 , time ms: 969
fetched: 415000 , time ms: 982
fetched: 420000 , time ms: 727
fetched: 425000 , time ms: 974
fetched: 430000 , time ms: 734
fetched: 435000 , time ms: 843
fetched: 440000 , time ms: 730
fetched: 445000 , time ms: 1051
fetched: 450000 , time ms: 778
fetched: 455000 , time ms: 773
fetched: 460000 , time ms: 849
fetched: 465000 , time ms: 1088
fetched: 470000 , time ms: 875
fetched: 475000 , time ms: 807
fetched: 480000 , time ms: 832
fetched: 485000 , time ms: 1120
fetched: 490000 , time ms: 813
fetched: 495000 , time ms: 1154
fetched: 500000 , time ms: 806
fetched: 505000 , time ms: 930
fetched: 510000 , time ms: 861
fetched: 515000 , time ms: 1144
fetched: 520000 , time ms: 885
fetched: 525000 , time ms: 1197
fetched: 530000 , time ms: 1205
fetched: 535000 , time ms: 890
fetched: 540000 , time ms: 1199
fetched: 545000 , time ms: 1265
fetched: 550000 , time ms: 983
fetched: 555000 , time ms: 1184
fetched: 560000 , time ms: 954
fetched: 565000 , time ms: 976
fetched: 570000 , time ms: 1363
fetched: 575000 , time ms: 1310
fetched: 580000 , time ms: 943
fetched: 585000 , time ms: 1283
fetched: 590000 , time ms: 1401
fetched: 595000 , time ms: 985
fetched: 600000 , time ms: 973
fetched: 605000 , time ms: 993
fetched: 610000 , time ms: 1054
fetched: 615000 , time ms: 1004
fetched: 620000 , time ms: 994
fetched: 625000 , time ms: 1021
fetched: 630000 , time ms: 1406
fetched: 635000 , time ms: 1044
fetched: 640000 , time ms: 1051
fetched: 645000 , time ms: 1123
fetched: 650000 , time ms: 1080
fetched: 655000 , time ms: 1070
fetched: 660000 , time ms: 1037
fetched: 665000 , time ms: 1191
fetched: 670000 , time ms: 1466
fetched: 675000 , time ms: 1167
And after immediate ctrl+c and dumping restart:
first request: 5000
fetched: 10000 , time ms: 229
fetched: 15000 , time ms: 221
fetched: 20000 , time ms: 225
fetched: 25000 , time ms: 232
fetched: 30000 , time ms: 304
fetched: 35000 , time ms: 331
fetched: 40000 , time ms: 248
fetched: 45000 , time ms: 255
fetched: 50000 , time ms: 264
fetched: 55000 , time ms: 314
fetched: 60000 , time ms: 329
fetched: 65000 , time ms: 304
fetched: 70000 , time ms: 296
fetched: 75000 , time ms: 286
fetched: 80000 , time ms: 409
fetched: 85000 , time ms: 411
fetched: 90000 , time ms: 304
fetched: 95000 , time ms: 427
fetched: 100000 , time ms: 310
fetched: 105000 , time ms: 315
fetched: 110000 , time ms: 406
On 27 November 2013 01:48, joergprante@gmail.com joergprante@gmail.com wrote:
Why do you want to keep search results for 30 minutes on each node?
This is the cause of your program getting slower. In 30 minutes, you can
easily thrash the system memory of the ES nodes so they must getting slower
and slower.Just use a scan/scroll retain time of a few seconds or so, just as long as
you need to dump the result. But never use minutes (unless you have a good
reason).Where do you get this example from?
Jörg
--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RYVP-VH8BaA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFJwg9JP0dKaWDRYnkEbsJWZzV4DL3WicEvu%2BfFSt42OQ%40mail.gmail.com.For more options, visit https://groups.google.com/groups/opt_out.
--
Regards, Ian Babrou
http://bobrik.name http://twitter.com/ibobrik skype:i.babrou
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANWdNRC9Agq%2Btg%2BAuQtMd2pce6KWgtfMen8HwhOxdWAAzUPyQw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.