Filebeat module for jboss logs

[Saloni_Vithalani]

We are using keycloak 5.0.0 and it gets started on JBOSS wildfly 7.0.0. What will be the recommended way of getting and parsig jboss logs in ELK. We are using filebeat, logstash, elasticsearch and kibana flow.

[ruflin]

I would recommend you to use Filebeat to read the log lines and then create an ingest pipeline to parse the logs: https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html

Best would be if in the future we have a jboss module that does the magic for you out of the box. If you are interested in this, perhaps file a feature request in the beats repository?

[Saloni_Vithalani]

Thanks for the reply @ruflin.

I managed to make jboss log in json format and pick log file location from environment variable. So, I think jboss gives the magic I was looking for. But the configuration of jboss to make it log in json was difficult to find online. Hence, here is the gist of standalone.xml that does this configuration.

I think If we get logs in json format from any application it makes it version agnostic and we can get rid of regex being maintained in pipeline.

What are your views on this understanding.

[ruflin]

Getting logs in JSON format is definitively an improvement in most cases. But json is not just solving the issue. Often I still see {"message": "foo"} which is json, but we still need to process the message field. The other part is the naming of fields and fields explosion. Some json logs have from my perspective too many different fields and they are not standardised. We are trying to solve this with https://github.com/elastic/ecs

So my guess is with JSON logs at least the input and groking part becomes easier, but we will still need renames and an understanding for the content of the log file.

[Saloni_Vithalani]

I looked at Elastic common schema and understood that common schema can give an upper hand to aggregate metrics and APM and logs into a common way of monitoring it.

But we are currently using 6.6.0 basic license and I am only concerned about logs not metric or APM, so is it fair to say that I do not need renames for the time being.