I've setup an ELK server with a number of "beats" agents, versions as follows:-
# rpm -qa | grep -E "logstash|elastic" logstash-2.1.1-1.noarch elasticsearch-2.1.1-1.noarch
# rpm -qa | grep beat filebeat-1.0.1-1.x86_64 packetbeat-1.0.1-1.x86_64 topbeat-1.0.1-1.x86_64
A couple of days ago I started getting timeout errors on the agents when they were trying to send messages to the ELK server. I read that the problem may be with logstash, so I've tried to remove the logstash from my pipeline so that the "beats" agents will send messages direct to elasticsearch. This seems to have sorted the errors.
My question is this, I had some custom grok patterns setup in logstash for my application logs. How do I replicate this and parse my logs direct in elastic search? Have I made a mess of things by removing logstash?
Any pointers would be most welcome.