What are the advantages of filebeat 1.2.2 over logstash 2.1.1?

Elastic Stack Beats
1

I get a fair number of crashes of the latter which I have always assumed is because of bugs and the java runtime and that one of the advantages I might enjoy with filebeat is that it is written in Go and perhaps has better memory management, fewer leaks, and better architecture.

So I am searching for reasons as to how best to justify filebeat over logstash for a large installation.

Thoughts?

2

Beats do not do geoip, they don't do translations, grok and much much more.
They are great for shippers, but analysis still belongs to LS.

(Also latest is 2.3.2 :slight_smile:

3

Do beats crash as mush as logstash?

4

I'll refine that question: DOES FILEBEAT CRASH AS MUCH AS LOGSTASH? The issue is that LOGSTASH 2.x crashes heavily here with a very plain, vanilla setup and I have CM restarting it regularly and want to get out from under the scenario above. FILEBEAT, if more stable, would be such a solution. Does anyone out there have filebeat running on a large number of servers and will they comment about the relative rate of their crashes of the filebeat client? Assume the clients are connecting to a logstash server rather than directly to elasticsearch.

5

why does logstash crash? Any errors?

What's your use-case. Any filters in logstash? Filebeat is no logstash replacement, but an addition to the stack.

6

LS SHOULD NOT CRASH (heh).
If it is then that is a fixable problem.

7

Well, that's fresh out of the box, without any special rules or transformations, just collecting a bunch of logs and sending them off to an Elasticsearch cluster.

8

If you can provide more info then perhaps we can help :slight_smile:

9

Yes, I will be glad to do so on the next occurrence.

10

Okay, I have a crashed logstash-shipper.log. It is over a megabyte. Where do you want it placed?

11

Is this the LS forwarder or LS itself?

Try putting it on gist/pastebin/etc.

12

I could only put ~150kb there. Here is the pastebin: http://pastebin.ca/3597434

13

Is redis running and reachable?

First traces are basically saying: can not connect to redis

CannotConnectError: Error connecting to Redis on 10.10.5.116:6379 (Errno::EHOSTUNREACH)

The since_db info messages are info message to notify you about progress I'd say.

I see a very verbose log, but no crash.

14

scracraft@devops1:~$ cat redis-tester.py
#!/usr/bin/python

import redis

rs = redis.Redis("10.10.5.116")

try:
response = rs.client_list()
print "redis server is up"
except redis.ConnectionError:
print "woops, redis server is down"
scracraft@devops1:~$ ./redis-tester.py
redis server is up
scracraft@devops1:~$

15

irb(main):007:0> r = Redis.new(:host => '10.10.5.116')
=> #<Redis client v3.3.0 for redis://10.10.5.116:6379/0>
irb(main):008:0> r.ping
=> "PONG"
irb(main):009:0> r.ping
=> "PONG"
irb(main):010:0>

16

The only thing I can think of is that the single Redis server isn't capable of dealing with the traffic from the network and should be clustered up.

17

maybe this explains why the redis error messages stopped and we got the sincedb entries (sincedb is keeping track of file state). Is redis running stable? Regarding logstash maybe you're better of asking on logstash forum.