Elastic Performance

Elastic Stack Elasticsearch
1

Hi all,
I wonder how\if elastic will handle my requirements:
20M records a day -> index 3 years backwards.

My worry is about query performance and not multiple requests. (more shards rather then more nodes)

will elastic handle this amount of data?
What is the best architecture to handle this load? ( index per day? week? )
what are the variables my performance will be effected by? my machines memory and size? will infinite memory and size support infinite data per day? (i exaggerate on purpose to learn&understands the relations between machine strength, architecture and performance of elastic)

Thanks in advance!

2

All of this depends on how large each record is.

3

It'll be an average document, not 2 large not 2 small. maybe even more leaning towards a small documents, it will have about dozen fields...

4

Average compared to what?

5

It will have 60~ish fields, the fields themselves will not have some base 64's or some very large data. you need the size of every document to give me a an answer?

6

Saying it's "average" means nothing when the uses for Elasticsearch range from metrics from the Mars rover through to DNA strands or even more "boring" stuff like logs, metrics, emails, PDFs, text.

You're asking a very general question and providing very vague details. Ultimately you will need to test with a set of your data to make sure your structures and queries are optimal for your use case.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.