Advice needed on configuration approach

Elastic Stack Logstash
1

Greetings all. I'm starting down a path of implementing ELK + NXLog for various reasons. Our current use case involves wanting to monitor:

  1. Certain Windows event log events (Really ASP.NET related ones, but I haven't figured out how to be that specific yet).
  2. IIS logs
  3. Various web application log files (many different apps, in many different environments, etc.).

I'm struggling to find examples of the best approach here...

My thought was to have each server send its logs via NXLog to our central logstash server. I'm ok with that except for one question: Is it better to run the logstash server with one input on one port that ALL of these logs get shipped to, or is it better to run multiple inputs on different ports and have each of those types of logs get separated later via a type field, etc.?

Then on the outputs, am I better off putting ALL of these logs into one elasticsearch index, or splitting them up, one index per type? Should I do a single index for ALL application logs, or separate those off into different indexes? If some of these logs will be in a more service oriented architecture, so transactions can bounce between multiple apps, would that change your answer for finding, for instance, a chain of logs with a "transactionId" on them?

Depending on the answer to that, what would the Kibana setups look like to allow for cross index queries?

I'll take pointers in Google Fu if I'm just missing where to find some of these answers!

2

My thought was to have each server send its logs via NXLog to our central logstash server. I'm ok with that except for one question: Is it better to run the logstash server with one input on one port that ALL of these logs get shipped to, or is it better to run multiple inputs on different ports and have each of those types of logs get separated later via a type field, etc.?

If you can use the same codec for all messages I don't see why it matters. You should be able to tag the messages with a type on the NXLog side.

Then on the outputs, am I better off putting ALL of these logs into one elasticsearch index, or splitting them up, one index per type?

How many types do you think you'll have, eventually? There is a fixed per-shard overhead so if you have too many types and too many days of logs (assuming daily indexes) you might consume a lot of cluster capacity just for keeping the shards alive.

Fields with a given name can only have one mapping in an index, i.e. if the field foo is an integer for type A it must also be an integer for type B. If you have multiple indexes this limitation won't exist. On the other hand, I find it less confusing when fields behave the same regardless of the event type.

3

Well, they wouldn't be the same types of messages at all. For instance, a Windows event log will not match an IIS log, nor one of our application logs. Our application logs should be relatively all the same in schema.

Are you saying that we should split them into different indexes based on the schema of the log? Ergo, one for IIS, one for windows events, and one for application logs, and then just make sure our application logs all have fields for "application", "environment", "host", etc.?

I'm feeling after some more research that I'd want to do multiple logstash inputs on different ports, one per log schema, and multiple ES indexes, again, one per log schema. There would only be three in my case this way.

4

Different indexes (or indexes series, actually) makes it much easier to implement fine-grained access control (i.e. only allowing some users to read messages of type X), but otherwise the only technical reason for splitting into different indexes that I know of is that one I stated earlier. Note that it's totally fine to have completely different document types in the same index as long as you don't try to have different mappings for a particular field.