In a highly regulated space and restricted industry, log files coming from multiple apps (100-200) with Log4Net and Log4J, Python Native logging libraries. No real customisation done by the devs.
In the absence (it’ll just never happen for these in env they’re in) of being able to get them to adopt an ECS library or FileBeat (they just pushed out to a streaming service for us to sub and process)…..
Is there a standard set of LogStash configuration pipelines for popular logging libraries like Log4Net, Log4J, Python native logging libraries. Allowing us to just push with the same ECS schema we have for the less regulated apps to Elasticsearch.
I don't think such thing exists, the definition of what will be logged using any of those libraries is done by the users according to its uses cases, so a pipeline would depends on the fields of each log.
If you can't change the format of the logs on the source, you will need to create a pipeline for each use case to transform the fields in ecs fields.
For example, one log may use srcip, other may use source_ip and others may use SourceIp, you would need to have a transformation for each one of those fields.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.