Data not coming in through redis

Elastic Stack Logstash
1

I have a basic ELK setup on a single box that is fed data from several inputs like syslog and lumberjack. It started dying from too much log data coming in so I installed redis server and added it to config, but now the logs never make it to elastic search.

output.conf:

output {
  if [type] == "redis" {
    elasticsearch {
      protocol => "http"
      host => [ "127.0.0.1" ]
      user => "logstash"
      password => "nope"
    }
  } else {
    redis {
      data_type => "list"
      key => "logstash:cache:dev"
    }
  }
}

redis.conf:

input {
  redis {
    host => "127.0.0.1"
    data_type => "list"
    key => "logstash:cache:dev"
    type => "redis"
  }
}

syslog.conf:

input {
  syslog {
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    dns {
      reverse => [ "host" ]
    }
  }
}

redis seems to be working and is receiving data:

$ redis-cli LLEN logstash:cache:dev
(integer) 68

Am I missing something obvious?
Thanks.

2

Are you sure type => "syslog" in the redis input actually changes and overwrites the existing type field of incoming events? By reading the code it's not what I'd expect. I'm guessing that message are just looping back and forth between Redis and Logstash.

3

I couldn't find any place where @type variable gets set in the code of the redis input plugin so I assume it gets overwritten correctly since it isn't set.

Just in case I also tried to use a custom tag which also doesn't work.

redis.conf:

input {
  redis {
    host => "127.0.0.1"
    data_type => "list"
    key => "logstash:cache:dev"
    type => "redis"
    tags => ["redis"]
  }
}

output.conf:

output {
  if ([type] == "redis") or ("redis" in [tags]) {
    elasticsearch {
      protocol => "http"
      host => [ "localhost" ]
      user => "logstash"
      password => "nope"
    }
  } else {
    redis {
      data_type => "list"
      key => "logstash:cache:dev"
    }
  }
}
4

I couldn't find any place where @type variable gets set in the code of the redis input plugin so I assume it gets overwritten correctly since it isn't set.

You make dangerous assumptions. The @type attribute is set by code inherited into the redis input plugin, but what ends up in the event depends on the I quoted above.

Just in case I also tried to use a custom tag which also doesn't work.

Could you be more specific than "doesn't work"? Anything interesting in the logs if you start Logstash with --verbose or --debug? To remove a potential source of errors, what if you replace the elasticsearch output with a simple stdout output?

5

Doesn't work = logs never make it to Elastic Search.

Nothing obvious in the logs with either verbosity level. Just a bunch of received events and filters parsing them.

Strangely, I don't see any mention of redis in the logs except that the redis Grok filters got loaded.

6

More strange info:
Two random entries from input type exec did show up in elastic search over a period of a ~4 hours. These are setup with a 10 second interval so about 1440 of these are missing.

Any ideas?

7

Problem solved by moving Redis to another server. Seems like tagging is broken at least on the Redis input, so I don't think there's a way to get this working on a single server.