Our ELK stack does heavy logs per day, almost 8 billion per day.
and we have a hot/warm architecture where hot nodes index the todays logs and warm for retention, and a separate set of master nodes
What components can be separated from the hot nodes to improve cpu . Basically we use i3 instances which are optimised for disk but not for cpu.
Was thinking if I could separate the high cpu taking components out of the i3 instances then would be able to ingest more logs with less resources.
I haven't explicitly defined any ingest pipeline, is there a default ingest pipeline that es automatically does? If so one improvement that I can see is to create separate pool of ingest nodes and disable ingestion on i3 instances
Do you have any other improvements? also can you plz confirm regarding any default pipeline that auto applies?