Elasticsearch 7.9.3 on Mac dies when inserting documents fast

Elastic Stack Elasticsearch
1

I use node-red to transform an existing JSON document and insert individual small JSON document using the POST command. Apparently i do this too fast because after a few seconds port 9200 stops responding.
What is best practice to send documents fast to elasticsearch? Can i tune my local install to accept faster document inserts?

Elasticsearch 7.9.3 native (no Docker, no VM) on MacBook Pro (32Gb memory and 1Tb NVMe drive so tuning is possible).

2

Could you share elasticsearch full logs please?

3

Not sure if they are of much use though, but you will let me know for sure.

4

Could you share them uncompressed on gist.GitHub.com?

5

I am now throttling the message flow to 250 messages/second and that seems to work fine. Slow, but ok.

6

That's not a good solution IMO. I mean that the fix is now probably hiding a problem

7

Agree. But i'm just starting using ES. I have no clue (yet) how to get around this problem. Aggregate the data and then bulk load maybe?

8

I look at your logs (please next time share them on gist.github.com so they can be read even on a mobile phone. It will help to get a faster answer).

Some comments:

[2020-10-29T08:46:23,622][INFO ][o.e.e.NodeEnvironment    ] [Laptop] heap size [1gb], compressed ordinary object pointers [true]

This is probably too low for the heap size, specifically if you are indexing a lot of data per second with huge bulks.

I can't see anything in logs between those 2 lines. That'd mean a hard stop of the process, like a JVM OOM Killer maybe...

[2020-10-31T01:00:10,733][INFO ][o.e.c.m.MetadataCreateIndexService] [Laptop] [.monitoring-es-7-2020.10.31] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0]
[2020-10-31T11:25:35,862][INFO ][o.e.n.Node               ] [Laptop] version[7.9.3], pid[993], build[default/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Mac OS X/10.15.7/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/15/15+36-1562]

Here we can see that something/someone stopped the node. Was that you?

[2020-11-02T14:29:18,805][INFO ][o.e.n.Node               ] [Laptop] stopping ...
[2020-11-06T15:31:18,402][INFO ][o.e.n.Node               ] [Laptop] stopping ...
[2020-11-07T16:25:49,959][INFO ][o.e.n.Node               ] [Laptop] stopping ...
[2020-11-09T18:32:42,186][INFO ][o.e.n.Node               ] [Laptop] stopping ...

If it was not done by you intentionally, it might indicate that a process kills elasticsearch for you.
Check that you don't have any oom jvm killer and increase the HEAP anyway.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.