How to do summary indexing in Elasticsearch

kelk · August 18, 2020, 8:42pm

hi,
we have a massive index, few reference indexes and we wanted to summarise from these multiple indexes into a single "summary index" . This way we are planning to join the queries

So the questions

Can output of a GET/POST from elasticsearch be indexed into ElasticSearch (another index)?
Should it be sent to logstash again or is there a method to index without going outside elasticsearch?

warkolm · August 18, 2020, 8:51pm

Do you mean something like https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-rollup.html?

kelk · August 18, 2020, 9:33pm

can the rollup join data from multiple indexes or do we need to individually roll them up?
I'm trying to achieve is a "correlation" of data within multiple indexes by putting into a single summarised index

warkolm · August 18, 2020, 9:36pm

Elasticsearch cannot join data.

What sort of correlation are you looking at?

kelk · August 18, 2020, 9:39pm

I had a complex set of requirement whereby I've to aggregate and then alert based on a dynamic threshold (from a CSV file). So my thought was to put the Dynamic CSV && the summarised raw dataset into a single index and then alert from it.

warkolm · August 18, 2020, 9:41pm

Ah right, you might be able to use a chained input alert to do this - https://www.elastic.co/guide/en/elasticsearch/reference/current/input-chain.html

kelk · August 18, 2020, 9:45pm

very interesting. I will have a playaround with it. Meantime, I've upvoted. thanks again mate

system · September 15, 2020, 9:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.