Shrink, Reindex with wildcard? Replumbing the lot!

[gavin.mcmenemy]

Hi All,

Need a bit of advice. We've been testing ElasticSearch as a tool for aggregating and analysing Windows logs. Nothing controversial about that.

We have a single node implementation in our Dev environment. I'm going to move this server out into the wide world but ...

Before I do that I'd like to fix our index.

I've been using curator and it's been creating daily indices. We've ended up with with a lot of small indices with a lot of shards. Considering that this is write intensive use I'd like to reduce these in number and probably rollover based on size, or maybe creating monthly indexes.

In addition I'd like to change the mapping - so I'm looking at a reindex job but I was wondering:

Should I shrink the index first?
Create a new index and simply reindex into it?

The total space on disk is around 40GB. So it should fit (just) onto a single shard. We're not currently indexing anything other than windows logs in ES but in the near future I'd like to add more indexes for networking gear, file systems etc.

Because it's been in a dev environment we are not currently bothered about the data being unavailable for a period of time.

Also I upgraded us to ES 6.7.0.

[DavidTurner]

@gavin.mcmenemy

Create a new index and simply reindex into it?

Yes, this sounds like the best path. "Shrink" allows you to reduce the number of shards within a single index, but doesn't let you combine multiple indices e.g. from different days. If you want to do that, you will need to reindex them. But there's no real need to shrink anything if you are going to reindex all the data anyway.

[gavin.mcmenemy]

HI David,

Thanks for that. There's so many options it's not always clear to me which would be the best.

Many Thanks.