If conditional with multiple outputs

Elastic Stack Logstash
1

Hi,
I have the following configuration in my logstash atm.

input {
   redis {
        host => "[elasticb.0001.us1.cache.amazonaws.com]"
        port => 6379
        key => "filebeat"
        data_type => "list"
    }
}

output {
   elasticsearch {
      if [type] == "nginx_access" {
         pipeline => "weblog_combined"
       }
       if [type] == "nginx_error" {
         pipeline => "weblog_nginx_error"
       }
       hosts => ["https://search-56ebnsnzz6dq.eu-west-1.es.amazonaws.com:443"]
       index => "filebeat-%{+YYYY.MM.dd}"
     }
tcp {
    host => "[listener.logz.io]"
    port => 5050
    codec => json_lines
   }
}

What I want to achieve is to be able to send all logs to both endpoints (elasticsearch and tcp one), however if type is nginx_access then it a log should be sent only to a given pipeline in elasticsearch and tcp endpoint. When I test the above config I get the following error:
[LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of #, => at line 14, column 8 (byte 330) after output {

What is wrong here?

3

The previous topic was flagged as spam automatically and I couldn't properly format it. It was hidden for few hours until I deleted it and it showed up.

4

Could be like this:

output {
	if "nginx_access" in [type] {
		elasticsearch { pipeline => "weblog_combined" }
		}
	if "nginx_error" in [type] {
		elasticsearch { pipeline => "weblog_nginx_error" }
		}
   else { ... }
	}
5

While the configuration file is working fine now the logs are not being sent to pipelines. Is there a way to check this?

6

I'm not sure, but did you add all the outputs?

 output {
    	if "nginx_access" in [type] {
    		elasticsearch { pipeline => "weblog_combined" }
    		hosts => ["https://search-56ebnsnzz6dq.eu-west-1.es.amazonaws.com:443"]
            index => "filebeat-%{+YYYY.MM.dd}"
         }

    	if "nginx_error" in [type] {
    		elasticsearch { pipeline => "weblog_nginx_error" }
    		hosts => ["https://search-56ebnsnzz6dq.eu-west-1.es.amazonaws.com:443"]
            index => "filebeat-%{+YYYY.MM.dd}"
         }
       else { tcp {
        host => "[listener.logz.io]"
        port => 5050
        codec => json_lines
       }
    }
7

This is my current config:

input {
  redis {
    host => "elasticsearch-logs.d3cexb.0001.euw1.cache.amazonaws.com"
    port => 6379
    key => "filebeat"
    data_type => "list"
  }
}

output {
  if "nginx_access" in [type] { 
     elasticsearch { 
        pipeline => "weblog_combined"
    }
  }
  if "nginx_access" in [type] {
     elasticsearch {
        pipeline => "weblog_nginx_error"
    }
  }
  elasticsearch {
    hosts => ["https://search-zego-es-56ebnsnz55kywk62l53h7kz6dq.eu-west-1.es.amazonaws.com:443"]
    index => "filebeat-%{+YYYY.MM.dd}"
   }
  tcp {
    host => "listener.logz.io"
    port => 5050
    codec => json_lines
   }
}

What I want to get to work is to send all logs to the tcp endpoint(regardless of type) and if log is of type in if statement then send it to a pipeline in elasticsearch. If log is not of type specified in if statements then still send it to elasticsearch as it is.

8

I tested this again and now for some reason logstash cannot see this config file and keeps trying to use the default config, which connects to 127.0.0.1:9200.
When I test the above config file
docker run -it -v /etc/logstash/conf.d/:/usr/share/logstash/config/ docker.elastic.co/logstash/logstash:7.0.0 /usr/share/logstash/bin/logstash -t -f /usr/share/logstash/config/logstash.conf
I get the below:
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

[INFO ] 2019-04-24 11:22:25.399 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}

[INFO ] 2019-04-24 11:22:25.407 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}

[WARN ] 2019-04-24 11:22:25.691 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified

Configuration OK

[INFO ] 2019-04-24 11:22:29.922 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.