Apache and IIS Logs on single instance of ElasticSearch, recommended approach?

[hexpirate (Mr. Dave)]

Relatively new to Elastic products, hopefully this is a simple question.

I have Apache logs and I have IIS logs. Can I store both on Elastic at once, how?

Looking for advice on how to approach the problem but open to specific prescriptive guidance as well. =)

[leandrojmp (Leandro Pereira)]

Yes, you can, there is no limitation, you can even have the events from the Apache Logs and the events from the IIS logs stored in the same index, but I think that the best approach is to store each log type in its own index.

To do that you will need to add different type names for each log source (apache or iis) and filter your events using Logstash before sending them to elasticsearch.

[hexpirate (Mr. Dave)]

Perfect! Thank you sir!

[bhatch (Brandon Hatch)]

We have a lot of log sources across multiple product lines in our cluster. The question I ask myself is "Would I ever want to easily reference both data sources while searching or running visualizations?" If the answer is yes, then I put them into the same index using _type or something else to differentiate them. If the answer no then I keep them separate. Even if they are separate you can use things like aliases to make them appear as one in Kibana.

If you end up storing them in the same index, I would recommend trying to use the same field names across both sources though. This allows you to do things such as "Give me all requests that took more than 20 seconds", or "Give me the requests that returned more than 10 MB of data". If the field names are the same you can use one phrase and have it return results from both sources. Otherwise you have to use OR statements which can get messy.
For Visualizations it is even more important. For example "Give me the top 10 URL's that have the longest average duration" won't work if IIS uses the name "cs-uri-stem" and Apache uses the name "Url". You would have to run two separate visualizations.