Should I run multiple indexes?

Elastic Stack Elasticsearch
1

I have been testing out ELK for monitoring our Windows Event Viewer logs, IIS logs and network switch logs. I have been having some serious performance issues. Not to mention when running queries I frequently get something like "Courier Fetch: 40 of 1410 shards failed.". Which upon further troubleshooting I am finding that this is due to: "nested: ExpressionScriptCompilationException[Field [cs_bytes] used in expression does not exist in mappings]" which I need to find a fix for.

My question is that is it better to separate log types out into different indexes? Such as syslog entries in one index (such as [syslog-]YYY.MM.DD), IIS logs in another ([iis-]YYYY.MM.DD) and event logs in another ([events-]YYYY.MM.DD)? Is there an "easy" way to re-index the old data all in one [logstash-]YYYY.MM.DD index to separate indexes?

The machine that I am running this on is a quad core VM with 65G of ram.

2

I would as it makes it easier to maintain.

I use Reindexing Elasticsearch with Logstash 2.0 · GitHub for reindexing.

3

Hi,

I'd recommend using a naming scheme like logstash-iis-YYYY.MM.DD, logstash-syslog-YYYY.MM.DD. There is a few advantages to this:

  1. Logstash adds a template called logstash, which matches any index called logstash-*. There are a few optimizations in there, so by starting your index names with logstash-* they will match and have these optimizations applied.
  2. When you have data that doesn't share the same field names or field types, you should use separate indices. To avoid field type conflicts.
  3. From the Kibana UI, you can still use logstash-* pattern and separate index patterns for each index.
  4. You can use logstash to re-index as Mark suggested or stream2es.
4

Would I be correct to determine that you put that config for Logstash in, then restart your Logstash instance and let it do its magic?

Once that is done then go ahead and edit your current rules? Or adjust current rules first?

5

Rules?

6

Sorry, current logstash filters/config and not rules.

7

Just run a manual instance of LS to do the reindex.