I use filebeat to harvest lines including keywords and send it to logstash for post processing.
But the time filebeat searching for the string is much longer than running grep in Ubuntu shell
I don't have number to show up but I can definitely 'feel' it
Did the beat team compare the filebeat inlcude_lines performance v.s. grep?
Here is my environment -
- filebeat 6.4.1 in my Ubuntu docker container
- ELK and Ubuntu filebeat are under the same network (created through docker-compose) running in the PC
- each of message files size is around 1.1MB, around 12000 lines in it
filebeat configuration -
include_lines: ['waiting for mykeyword.*']