Manage multiple beats on a single Logstash instance

Elastic Stack Logstash
1

Hello, I have multiple porspectors on a filebeat.yml configuration file. Each of these prospector has a unique type:

filebeat:
  prospectors:
    -
      paths:
        - file*.csv
      input_type: log
      document_type: type1
    -
      paths:
        - file*.log
      input_type: log
      document_type: type2
[...]

While the Logstash configuration is like this:

input {
  beats {
    port => 5044
	}
       }

filter {
  if [type] == "type2" {
    grok { [some-grok-configuration-here] }
  }
  else if [type] == "type1" {
    csv { [some-csv-configuration-here] }

output {
  if [type] == "type2" {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "logstash-type2-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
  }
  else if [type] == "type1" {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "logstash-type1%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
  }
}

From the Filebeat log I see that all the files are correctly sent with the correct "type" field to Logstash. However, Logstash pipeline is only showing type2 and obviously I can't find type1 on Elasticsearch.
Could you help me figuring out why this is happening?

2

Are they showing in stdout at all?

3

In Logstash stdout only one type is showing

4

I have noticed that when there is only one beat type (hence when I don't use conditionals in Logstash configuration) the stdout of the CSV has a pre-defined "Normal" type:

"@version" => "1",
"@timestamp" => "2016-09-19T05:49:56.269Z",
"type" => "Normal",

Even if I didn't defined one. Maybe this is the cause, because in the conditionals I'm referring to the type set by Filebeat.

5

Yes, issue solved! :slight_smile:

I've added a

mutate { replace => { "type" => "type2" } }

Inside the Logstash filter section and now it works like a charm.
Was this approach the correct one?