Which codec has the best performance?

Elastic Stack Logstash
1

Hi Team,

In one of our new projects we are trying to use ELK stack with Redis as the buffering interface. I was going through the documentation and different examples over the internet and saw that at most of the places shipper is pushing the data in JSON format to redis channel/list etc. Even the default value of codec plugin used in input plugin of logstash used for reading the logs from redis is JSON.

My question revolves around is there any specific reason for using JSON as the default data representation for redis and is it easy for logstash to decode the json style log data into events as compared to other codec plugins?

Also, it might not be a best question but just out of curiosity which codec plugin has the best performance at logstash level in terms of input plugin parsing the raw log data and converting it to events?

Cheers
Piyush

2

Which shipper is this?

3

Well the documents I looked into were:
a.) http://www.logstash.net/docs/1.3.2//tutorials/getting-started-centralized - I know this document is old, but if you look at the redis list output, it shows a JSON content.
b.) https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html - The default value of codec is JSON (why default?)
c.) https://github.com/logstash/logstash-logback-encoder/ - Last but not the least. All appenders, layouts and encoders are JSON specific.

All these links gave me a view(hint) with logstash is much happy with JSON format as input and that's where my question comes. Why logstash prefers JSON? And also which other codec is recommendable in terms of performance? :slight_smile:

Regards
Piyush

4

Most of those are so old you shouldn't be using them!

But yes, LS prefers json because ES prefers it. json is simple to parse.

5

@Piyush_Goyal
On the shipper side in the Redis ouput, a LS Event (a kind of Map/Dictionary) needs to be made into a string for storage in Redis.
On the ingester side, the Redis input needs to take the string and make a LS Event from it.

We use JSON for the to-string and from-string conversion by default.

Why?

  • We use a very efficient JSON Processor (Jackson wrapped for JRuby). Most of the event that pass through the Processor will have a similar structure where most keys and some strings are repeated. Jackson makes heavy use of caching when it sees the same thing again and again. The LS process runs for a long time and Jackson will process many similar events.
  • The format is easy for a human to read.

Why not something else?

  • Frankly, we have not tried.
  • the msgpack codec would probably work, YMMV. However, it does not use any caching.