I want to use filebeat to send logs to elasticsearch with simple structure (date in custom format, http code, processing time in ms, query text).
So log line should be parsed and these data should go to different fields in Index.
Also I want to add another field (length of query text in symbols, provided it is in UTF-8 encoding), and I want to truncate the actual text so it fits to 32Kb (because of ES limitation).
As far as I understand I can do all these things in Logstash (even add custom handler written in Ruby).
The question is: is it possible to avoid using logstash at all and achieve these transformations using filebeat only (and possibly in ES using ingest API, pipelining. etc).
I tried to load sample log file into ES vis LS. LS process consumed 2 times more CPU that ES. Is it normal? ES does complex job indexing data. And LS only parses lines against regexp. It feels that LS should be rather light process, but it is not case
Though the same logic for parsing log lines against regexps written in Python consumes like an order of magnitude less CPU that logstash does. So something looks broken here...
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.