Elastic Performance

[sharongur (sharon)]

Hi all,
I wonder how\if elastic will handle my requirements:
20M records a day -> index 3 years backwards.

My worry is about query performance and not multiple requests. (more shards rather then more nodes)

will elastic handle this amount of data?
What is the best architecture to handle this load? ( index per day? week? )
what are the variables my performance will be effected by? my machines memory and size? will infinite memory and size support infinite data per day? (i exaggerate on purpose to learn&understands the relations between machine strength, architecture and performance of elastic)

Thanks in advance!

[warkolm (Mark Walkom)]

All of this depends on how large each record is.

[sharongur (sharon)]

It'll be an average document, not 2 large not 2 small. maybe even more leaning towards a small documents, it will have about dozen fields...

[warkolm (Mark Walkom)]

Average compared to what?

[sharongur (sharon)]

It will have 60~ish fields, the fields themselves will not have some base 64's or some very large data. you need the size of every document to give me a an answer?

[warkolm (Mark Walkom)]

Saying it's "average" means nothing when the uses for Elasticsearch range from metrics from the Mars rover through to DNA strands or even more "boring" stuff like logs, metrics, emails, PDFs, text.

You're asking a very general question and providing very vague details. Ultimately you will need to test with a set of your data to make sure your structures and queries are optimal for your use case.