# Data not coming in through redis

**URL:** https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957
**Category:** Logstash
**Created:** [July 20, 2015, 11:40pm UTC](https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957 "2015-07-20T23:40:30Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)
#### Post date: [July 20, 2015, 11:40pm UTC](https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957/1 "2015-07-20T23:40:30Z")

</div>

I have a basic ELK setup on a single box that is fed data from several inputs like syslog and lumberjack. It started dying from too much log data coming in so I installed redis server and added it to config, but now the logs never make it to elastic search.

output.conf:

```
output {
  if [type] == "redis" {
    elasticsearch {
      protocol => "http"
      host => ["127.0.0.1"]
      user => "logstash"
      password => "nope"
    }
  } else {
    redis {
      data_type => "list"
      key => "logstash:cache:dev"
    }
  }
}

```

redis.conf:

```
input {
  redis {
    host => "127.0.0.1"
    data_type => "list"
    key => "logstash:cache:dev"
    type => "redis"
  }
}

```

syslog.conf:

```
input {
  syslog {
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    dns {
      reverse => ["host"]
    }
  }
}

```

redis seems to be working and is receiving data:

```
$ redis-cli LLEN logstash:cache:dev
(integer) 68

```

Am I missing something obvious?  
Thanks.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 21, 2015, 9:59am UTC](https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957/2 "2015-07-21T09:59:43Z")

</div>

Are you sure `type => "syslog"` in the redis input actually changes and overwrites the existing `type` field of incoming events? By reading the code it's not what I'd expect. I'm guessing that message are just looping back and forth between Redis and Logstash.

> <https://github.com/elastic/logstash/blob/v1.5.3/lib/logstash/inputs/base.rb#L109-L111>

---

<div class="post-metadata">

### Author: ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)
#### Post date: [July 21, 2015, 6:26pm UTC](https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957/3 "2015-07-21T18:26:57Z")

</div>

I couldn't find any place where @type variable gets set in the code of the redis input plugin so I assume it gets overwritten correctly since it isn't set.

Just in case I also tried to use a custom tag which also doesn't work.

redis.conf:

```
input {
  redis {
    host => "127.0.0.1"
    data_type => "list"
    key => "logstash:cache:dev"
    type => "redis"
    tags => ["redis"]
  }
}

```

output.conf:

```
output {
  if ([type] == "redis") or ("redis" in [tags]) {
    elasticsearch {
      protocol => "http"
      host => ["localhost"]
      user => "logstash"
      password => "nope"
    }
  } else {
    redis {
      data_type => "list"
      key => "logstash:cache:dev"
    }
  }
}
```

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 21, 2015, 8:10pm UTC](https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957/4 "2015-07-21T20:10:02Z")

</div>

> I couldn't find any place where @type variable gets set in the code of the redis input plugin so I assume it gets overwritten correctly since it isn't set.

You make dangerous assumptions. The @type attribute is set by code inherited into the redis input plugin, but what ends up in the event depends on the I quoted above.

> Just in case I also tried to use a custom tag which also doesn't work.

Could you be more specific than "doesn't work"? Anything interesting in the logs if you start Logstash with `--verbose` or `--debug`? To remove a potential source of errors, what if you replace the elasticsearch output with a simple stdout output?

---

<div class="post-metadata">

### Author: ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)
#### Post date: [July 21, 2015, 8:57pm UTC](https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957/5 "2015-07-21T20:57:20Z")

</div>

Doesn't work = logs never make it to Elastic Search.

Nothing obvious in the logs with either verbosity level. Just a bunch of received events and filters parsing them.

Strangely, I don't see any mention of redis in the logs except that the redis Grok filters got loaded.

---

<div class="post-metadata">

### Author: ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)
#### Post date: [July 21, 2015, 10:23pm UTC](https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957/6 "2015-07-21T22:23:08Z")

</div>

More strange info:  
Two random entries from input type exec did show up in elastic search over a period of a ~4 hours. These are setup with a 10 second interval so about 1440 of these are missing.

Any ideas?

---

<div class="post-metadata">

### Author: ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)
#### Post date: [July 29, 2015, 6:57pm UTC](https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957/7 "2015-07-29T18:57:54Z")

</div>

Problem solved by moving Redis to another server. Seems like tagging is broken at least on the Redis input, so I don't think there's a way to get this working on a single server.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:33am UTC](https://discuss.elastic.co/t/data-not-coming-in-through-redis/25957/8 "2017-07-06T05:33:28Z")

</div>


