LogStash Configurations for Log4Net, Log4J etc

[Tomahawk]

Bit of a left field question…..

In a highly regulated space and restricted industry, log files coming from multiple apps (100-200) with Log4Net and Log4J, Python Native logging libraries. No real customisation done by the devs.

In the absence (it’ll just never happen for these in env they’re in) of being able to get them to adopt an ECS library or FileBeat (they just pushed out to a streaming service for us to sub and process)…..

Is there a standard set of LogStash configuration pipelines for popular logging libraries like Log4Net, Log4J, Python native logging libraries. Allowing us to just push with the same ECS schema we have for the less regulated apps to Elasticsearch.

In this situation what would you suggest…

[Tomahawk]

No bueno?

[leandrojmp (Leandro Pereira)]

@Tomahawk

Is there a standard set of LogStash configuration pipelines for popular logging libraries like Log4Net, Log4J, Python native logging libraries.

I don't think such thing exists, the definition of what will be logged using any of those libraries is done by the users according to its uses cases, so a pipeline would depends on the fields of each log.

If you can't change the format of the logs on the source, you will need to create a pipeline for each use case to transform the fields in ecs fields.

For example, one log may use srcip, other may use source_ip and others may use SourceIp, you would need to have a transformation for each one of those fields.

Something like this:

mutate {
    rename => {
        "srcip" => "[source][ip]"
        "source_ip" => "[source][ip]"
        "SourceIp" => "[source][ip]"
    }
}