Centralized logging solution for custom application logs

We are looking for a solution that could help us centralize custom logs from different applications that are all part of a single system. The applications are all different in nature, some are .NET C# apps on IIS, some are services running in Microsoft Biztalk Server (in C#) and some could be Ruby on Rails sites running under Apache on Linux.

All of these applications call each other in sequence to transform and deliver messages, and all have a single point in common: a request id.

We need a way to be able to track the logs related to the same request-id across all the different apps of the system. In other words, we would like to be able to search Kibana for all logs from all apps where the request-id is equal to a specific ID, and be able to follow the flow easily.

We thought of using the ELK stack to do this, but that means that all of those apps would have to have a common log format, or at least, a common schema once Logstash ships them to Elasticsearch, or else, it would be impossible to search the multiple indices at once since the fields wouldn't match. Is my assessment correct?

Ideally, we would like to explore the machine learning feature and use visualisations in Kibana, and from what I understand, it would only work properly if all our data is in the same index pattern. Correct?

From what I have seen in the past, provided the request id field has the same name across each log format they do not necessarily need to have a common schema. Being able to request all the logs related to a request id in a single call is a giant leap forward for the folks doing RCA. Of course the more commonality between the field names in the indexes the more interesting things you can do with the data.

You would need all the data in the same index pattern, but that does not mean the same index, and you can have multiple index patterns.

1 Like

Ok, that's what I thought. One of the issues is that the Biztalk Server application logs we currently produce are Windows Event Logs, and that using the winlogbeats agent to push them to Logstash uses the Beats protocol and schema. I tried to use * as an index pattern in Kibana to include the winlogbeats index and our other custom application log indexes, but it warned about field type mismatch right away. If we want to use the mode advanced analysis tools of Kibana, such as the Visualizations and the Machine Learning, wouldn't that prevent us from doing so properly?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.