Multiple logs 1 index or 1 log per index?

[javadevmtl (None)]

Hi we are starting to index our applications logs into Elastic search using 5.4

All our logs have obviously common fields like time stamp, threads, log level etc...

But the log message is where it each log is different. We are storing the log message as json so we can index and aggregate on it as well.

The question is if each application has different "message" should we index all applications in the same index or use 1 index per application.

I'm assuming if we use 1 index for all logs we will have sparse fields on the message part. Does that matter?

If we go down the route of 1 log 1 index, how can we correlate say between the API gateway log and the microservice log.

[theuntergeek (Aaron Mildenstein)]

If all of the other fields are truly similar, then I would put all in one index. Sparse fields is a consideration, but it would be the same if not worse with multiple mapping types per index.

As a point to ponder: starting in Elasticsearch 6.0, only one mapping type per index will be supported.

[thn (tri-man)]

It depends on the number of log messages generated by each application per day and how you want to handle it. Here are a few scenarios that you might want to consider

• how long do you want to keep the log messages per application?
• how often do you query/aggregate info per application?
• does the message field has text in one language? do you need to support multi-language search?

Here are my suggestions regarding indexing data

• If you want to use 1 index for all log messages, I suggest to try / where is the document type used by elasticsearch

• If you want to use 1 index for each application, you can name the index like log-, this will allow you to search for info per index using log- as an index name or use the alias "logs" that points (or mapped) to all log- indices. In kibana, you can also use "log-*" to search for info in all indices with "log-" prefix