# Not able to index the logs coming from Shipper through redis into the ELK **URL:** \ **Category:** Logstash\ **Created:** [September 13, 2016, 10:31am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367 "2016-09-13T10:31:50Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![ANU\_SARA\_VARGHESE](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anu_sara_varghese/32/51487_2.png) [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Post date:** [September 13, 2016, 10:31am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/1 "2016-09-13T10:31:50Z")
Hi, I am deploying ELK for POC. In that, I have pushed the logs into the logstash shipper(I am able to see that using tcpdump). But then I am not able to see them in Elasticsearch and kibana. I am able to see them at Redis. This issue came after I applied filter plugin in the Logstash indexer. Could anybody help me with this? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 13, 2016, 10:55am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/2 "2016-09-13T10:55:06Z")
So it appears that the Logstash instance that reads from Redis and posts to ES stopped working when you added a filter? Does it start to work again if you remove the filter? ---
**Author:** ![ANU\_SARA\_VARGHESE](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anu_sara_varghese/32/51487_2.png) [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Post date:** [September 13, 2016, 12:09pm UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/3 "2016-09-13T12:09:49Z")
Hi Magnus, It did not work. I reinstalled logstash, thus removing the logstash patterns folder and just using one pattern with the grok filter . It started working. Earlier I had tried to use the patterns ( [https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns](https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns) ) by installing the Jruby and run the Gemfile but this did not work. I want to use these logstash patterns. How can I use them without editing the Gemfile? It appears to me that by editing the Gemfile the indexer had stopped working. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 13, 2016, 12:19pm UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/4 "2016-09-13T12:19:25Z")
I don't understand exactly what you want to do, but it sounds like your should either - upgrade the logstash-patterns-core plugin or - add the wanted pattern file(s) in e.g. /etc/logstash/patterns and point your grok filter to that directory. ---
**Author:** ![ANU\_SARA\_VARGHESE](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anu_sara_varghese/32/51487_2.png) [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Post date:** [September 14, 2016, 2:13pm UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/5 "2016-09-14T14:13:53Z")
Hey Magnus, I have applied multiple grok patterns as follows in the logstash filter { ``` grok { match => ["message", " %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}"] } grok { match => { "message" => "Accepted %{WORD:auth_method} for %{USER:username} from %{IP:src_ip} port %{INT:src_port} ssh2" } } grok { match => { "message" => "Invalid user %{USER:username} from %{IP:src_ip}" } } grok { match => { "message" => "Failed %{WORD:auth_method} for %{USER:username} from %{IP:src_ip} port %{INT:src_port} ssh2" } } ``` } The logstash indexer has extracted the fields from the first two grok patterns but not from the last two. Could you help me on this? How can I extract fields for different logs? What would be the appropriate way to do the filtering and indexing in the logstash for different types of logs? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 14, 2016, 2:22pm UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/6 "2016-09-14T14:22:59Z")
> The logstash indexer has extracted the fields from the first two grok patterns but not from the last two. Could you help me on this? Yes, if you show me an example message that's currently not processed correctly. > How can I extract fields for different logs? What would be the appropriate way to do the filtering and indexing in the logstash for different types of logs? Use conditionals on e.g. the `type` field. See [Accessing event data and fields | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html). ---
**Author:** ![ANU\_SARA\_VARGHESE](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anu_sara_varghese/32/51487_2.png) [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Post date:** [September 14, 2016, 8:47pm UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/7 "2016-09-14T20:47:30Z")
Hi Magnus, Thanks for the reply. The sample logs which is not being processed correctly are of these 3 types 1. Invaild user username from 123.123.123.123 2. Failed password for username from 123.123.123.123 port 1234 ssh2 3. Failed password for invaild username from 123.123.123.123 port 1234 ssh2 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 15, 2016, 5:35am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/8 "2016-09-15T05:35:23Z")
Please show the raw output from a `stdout { codec => rubydebug }` output for a message that isn't processed correctly (and presumably has a `_grokparsefailure` tag). ---
**Author:** ![ANU\_SARA\_VARGHESE](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anu_sara_varghese/32/51487_2.png) [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Post date:** [September 15, 2016, 6:15am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/9 "2016-09-15T06:15:30Z")
Hi Mangus, The output for the message " Invaild user username from 123.123.123.123 " { "message" =\> "Invaild user username from 123.123.123.123", "@version" =\> "1", "@timestamp" =\> "2016-09-15T06:11:37.151Z", "host" =\> "ip-10-0-0-10", "tags" =\> [ [0] "\_grokparsefailure" ] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 15, 2016, 6:56am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/10 "2016-09-15T06:56:30Z")
"Invalid" is misspelled in the log message but not in your grok expression. ---
**Author:** ![ANU\_SARA\_VARGHESE](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anu_sara_varghese/32/51487_2.png) [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Post date:** [September 15, 2016, 7:16am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/11 "2016-09-15T07:16:46Z")
Hi Mangus, It has worked. But the tag \_grokparsefailure is still there. Will this cause any problem during the parsing.? Invalid user username from 123.123.123.123 { "message" =\> "Invalid user username from 123.123.123.123", "@version" =\> "1", "@timestamp" =\> "2016-09-15T07:06:54.759Z", "host" =\> "ip-10-0-0-10", "tags" =\> [ [0] "\_grokparsefailure" ], "username" =\> "username", "src\_ip" =\> "123.123.123.123" } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 15, 2016, 7:19am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/12 "2016-09-15T07:19:38Z")
You have multiple grok filters but only one of them matches the input, so the remaining filters obviously fail and tag the event with `_grokparsefailure`. Note that you can list multiple grok expressions in a single grok filter which helps in this situation. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:38am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367/13 "2017-07-06T04:38:22Z")