We have filebeat on our linux box which periodically sends logs to elastic search and from there kibana is used to display them. In the scenario our application generates two types of logs:
Type 1 - {"data": "some string log"}; - string
Type 2 - {"data": {"objectKey": "objectValue"}} - object
Now if type 1 is sent first to elastic search it creates the data field and type as keyword and when filebeat tries to send type 2 it breaks because of incompatible types. This happens vice versa as well.
Is there any solution for this problem? May be something on the line of dynamic templates. We can not change the logging as it comes from different applications and can be different. Any help will be really appreciated.
If it is possible to split the two types in to their own log files, that would be one way to go. You would have type1.log and type2.log. You could then add fields in filebeat to the two types of logs. (Then if you had Logstash you could modify "data" based on that added field...)
That is a very tricky situation then... Not sure if you can solve that with just Filebeat...
You could have something splitting the original log file and then writing new files that are picked up by Filebeat. I do something like that with logs we get from Stackdriver in GCP. One process pulls down the logs and does a few things with the JSON lines, removes some stuff, copies keys to new key names (to solve the same exact problem). That process then writes the new modified lines into new log files that Filebeat picks up. It is a python script that is managed by daemontools
Just a thought. Forward it to logstash. use grok pattern to differentiate/parse events and add some field based on the type and use that field in the output block to forward logs to different indexes.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
