What are the advantages of filebeat 1.2.2 over logstash 2.1.1?

(Stuart Cracraft) #1

I get a fair number of crashes of the latter which I have always assumed is because of bugs and the java runtime and that one of the advantages I might enjoy with filebeat is that it is written in Go and perhaps has better memory management, fewer leaks, and better architecture.

So I am searching for reasons as to how best to justify filebeat over logstash for a large installation.


(Mark Walkom) #2

Beats do not do geoip, they don't do translations, grok and much much more.
They are great for shippers, but analysis still belongs to LS.

(Also latest is 2.3.2 :slight_smile:

(Stuart Cracraft) #3

Do beats crash as mush as logstash?

(Stuart Cracraft) #4

I'll refine that question: DOES FILEBEAT CRASH AS MUCH AS LOGSTASH? The issue is that LOGSTASH 2.x crashes heavily here with a very plain, vanilla setup and I have CM restarting it regularly and want to get out from under the scenario above. FILEBEAT, if more stable, would be such a solution. Does anyone out there have filebeat running on a large number of servers and will they comment about the relative rate of their crashes of the filebeat client? Assume the clients are connecting to a logstash server rather than directly to elasticsearch.

(Steffen Siering) #5

why does logstash crash? Any errors?

What's your use-case. Any filters in logstash? Filebeat is no logstash replacement, but an addition to the stack.

(Mark Walkom) #6

If it is then that is a fixable problem.

(Stuart Cracraft) #7

Well, that's fresh out of the box, without any special rules or transformations, just collecting a bunch of logs and sending them off to an Elasticsearch cluster.

(Mark Walkom) #8

If you can provide more info then perhaps we can help :slight_smile:

(Stuart Cracraft) #9

Yes, I will be glad to do so on the next occurrence.

(Stuart Cracraft) #10

Okay, I have a crashed logstash-shipper.log. It is over a megabyte. Where do you want it placed?

(Mark Walkom) #11

Is this the LS forwarder or LS itself?

Try putting it on gist/pastebin/etc.

(Stuart Cracraft) #12

I could only put ~150kb there. Here is the pastebin: http://pastebin.ca/3597434

(Steffen Siering) #13

Is redis running and reachable?

First traces are basically saying: can not connect to redis

CannotConnectError: Error connecting to Redis on (Errno::EHOSTUNREACH)

The since_db info messages are info message to notify you about progress I'd say.

I see a very verbose log, but no crash.

(Stuart Cracraft) #14

scracraft@devops1:~$ cat redis-tester.py

import redis

rs = redis.Redis("")

response = rs.client_list()
print "redis server is up"
except redis.ConnectionError:
print "woops, redis server is down"
scracraft@devops1:~$ ./redis-tester.py
redis server is up

(Stuart Cracraft) #15

irb(main):007:0> r = Redis.new(:host => '')
=> #<Redis client v3.3.0 for redis://>
irb(main):008:0> r.ping
=> "PONG"
irb(main):009:0> r.ping
=> "PONG"

(Stuart Cracraft) #16

The only thing I can think of is that the single Redis server isn't capable of dealing with the traffic from the network and should be clustered up.

(Steffen Siering) #17

maybe this explains why the redis error messages stopped and we got the sincedb entries (sincedb is keeping track of file state). Is redis running stable? Regarding logstash maybe you're better of asking on logstash forum.

(system) #18