# Logstash does not process multiple filebeat inputs

**URL:** <https://discuss.elastic.co/t/logstash-does-not-process-multiple-filebeat-inputs/164712>\
**Category:** Logstash\
**Created:** [January 17, 2019, 10:44pm UTC](https://discuss.elastic.co/t/logstash-does-not-process-multiple-filebeat-inputs/164712 "2019-01-17T22:44:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![victoravr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/victoravr/32/39901_2.png) [@victoravr](https://discuss.elastic.co/u/victoravr)\
**Post date:** [January 17, 2019, 10:44pm UTC](https://discuss.elastic.co/t/logstash-does-not-process-multiple-filebeat-inputs/164712/1 "2019-01-17T22:44:33Z")

</div>

G'day!

I'm new to ELK - please forgive me if it's something obvious.

I'd like filebeat to ship two different type of logs to logstash, logstash to process it with different grok filters and output to elasticsearch in two different indices.

The problem I'm having is that filebeat ships all logs, logstash receives them, but only one index is getting created, i.e. app-catalina-log-

My filebeat inputs are as follows:  
filebeat.inputs:

- type: log  
paths:
  - /var/log/tomcat/\*.log  
fields:  
app: app  
log\_type: catalina-log  
multiline.pattern: '^[[:space:]]'  
multiline.pattern: '^([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?):'  
multiline.negate: false  
multiline.match: after

- type: log  
enabled: true  
paths:
  - /var/log/tomcat/\*.txt  
fields:  
app: app  
log\_type: tomcat-access-log  
output.logstash:  
hosts: [":5044"]  
...

My logstash config:  
input {  
beats {  
port =\> 5044  
}  
}  
filter {  
if ([fields][app] == "mapleta") {

```
if ([fields][log_type] == "catalina-log") {
   grok {
        match => ["message", "%{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{NOTSPACE:methodName}\n%{LOGLEVEL:logLevel}: %{GREEDYDATA:logMessage}"]
        remove_field => ["message"]
   }
   date {
     timezone => "Australia/Sydney"
     match => ["timestamp", "MMM dd, YYYY KK:mm:ss a"]
     target => "@timestamp"
     remove_field => ["timestamp"]
   }   
}
if ([fields][log_type] == "tomcat-access-log") {
   grok {
        match => ["message" , "%{IPV4:clientIP} - %{NOTSPACE:user} \[%{DATA:timestamp}\] \"%{WORD:method} %{NOTSPACE:request} HTTP/1.1\" %{NUMBER:status} %{NUMBER:bytesSent}" ]
        remove_field => ["message"]
    }
    grok{
        match => ["request", "/%{USERNAME:app}/"]
        tag_on_failure => []
    }
    date {
        match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
        remove_field => ["timestamp"]
    }
    mutate {
        lowercase => ["user"]
        convert => ["bytesSent", "integer", "duration", "float"]
        update => { "host" => "%{[beat][hostname]}" }
        remove_field => ["beat","type","geoip","input_type","tags"]
    }
    if [user] == "-" {
        mutate {
            remove_field => ["user"]
        }
    }
    # drop unmatching message (like IPv6 requests)
    if [message] =~ /(.+)/ {
        drop { }
    }    	
}
}

```

}  
output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
manage\_template =\> false  
index =\> "%{[fields][app]}-%{[fields][log\_type]}-%{+YYYY.MM.dd}"  
}  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 17, 2019, 10:59pm UTC](https://discuss.elastic.co/t/logstash-does-not-process-multiple-filebeat-inputs/164712/2 "2019-01-17T22:59:40Z")

</div>

> [@victoravr](#):
>
> if [message] =~ /(.+)/ { drop { } }

It appears to me that that drops all tomcat-access-log messages.

---

<div class="post-metadata">

**Author:** ![victoravr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/victoravr/32/39901_2.png) [@victoravr](https://discuss.elastic.co/u/victoravr)\
**Post date:** [January 17, 2019, 11:07pm UTC](https://discuss.elastic.co/t/logstash-does-not-process-multiple-filebeat-inputs/164712/3 "2019-01-17T23:07:29Z")

</div>

Yes, that was it! I've removed this drop portion of the filter and now it works as intended! Many thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 14, 2019, 11:07pm UTC](https://discuss.elastic.co/t/logstash-does-not-process-multiple-filebeat-inputs/164712/4 "2019-02-14T23:07:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.