Scale ingest node based on number of application

Elastic Stack Elasticsearch
1

Hello, I have a plan to use ingest pipeline on my cluster,
the plan is:
90 application will hit to elastic server, so I'll make around 200 pipeline on my cluster
How can I scale the ingest node I need on my cluster to make this possible

Thanks in advance

2

anyone know the method to scale the number of the ingest node pls

3

Hi @alfianaf Are you in Elastic Cloud or Self Managed.

You can scale the number of ingest nodes by deploying nodes with only ingest role only.

See here

But more importantly that how many pipelines is how complex they are and what is throughput in terms of Events / Min etc.

The number and size of the ingest nodes will depend on that. Also unless otherwise specified your other nodes probably are ingest nodes by default, if you want to isolate ingest processing to only the dedicated ingest only nodes you will need to remove the ingest role from the other nodes.

Ingest node do not need much disk but CPU and RAM are good. Depending on your ingest needs will help understand what you needs are.

If you provide a little perspective perhaps we could suggest a starting point.

Hope this helps...

1 Like
4

hello @stephenb, I'm on self managed server, if I can explain the complexity of pipeline, maybe each pipeline checks on document field to get different index output,
on example, if field == A, indexname == A, and the event throughput is about 1500 events / Min per pipeline

5

for additional information, I'm running my cluster on VM
my cluster have
1 load-balance node (this node now also act as ingest node)
1 load-balance node (not act as ingest node)
1 master node
4 data node

so right now I only have 1 ingest node

6

15O0 events / min I suspect 1 ingest node is fine depending on the RAM and CPU it is provided.

I am not sure what you mean by a load balance node.. I suspect you might mean a coordinator node?

7

yes I guess it is coordinator node,
so is it fine 1 ingest node handling 1500 events / min * 200 pipeline? does this mean my node can handle around 30k events / min total?

8

First I miss-understood I thought you meant 1500 EPM Total, but really you meant about 30K EPM Total, that should still be fine as that is really only about 500 Events / Sec... you should probably be able to scale a 4 CPU 8 GB RAM Ingest node to about 7-10x that ... about 3500 - 5000 Events / Sec for a low / medium level pipeline complexity. This is just an estimate.

Basically if it becomes CPU or RAM bound you can increase those... up to about what I said above then start scaling horizontally.

Again without close inspection these are just guesses... if you pipelines are not complex you might get more or more complex you might get less throughput.

1 Like
9

thank you for the detailed explanation, I guess I will stay on the current one ingest node then

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.