[Solved] Logstash sends logs to the same elasticsearch index even after I configured not to do so

Elastic Stack Logstash
1

I got two logs, one is nginx access log, another is gunicorn access log.
The two logs has similar contents, but I want them to be two different elasticsearch indices.
Here are my conf files:

# /etc/logstash/conf.d/nginx-access-01.conf
input {
  file {
    path => "/home/deploy/log/fresh/nginx_access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => { "message" => "%{DATA:log_host} %{IPORHOST:remote_ip} -%{DATA:remote_user}- \[%{HTTPDATE:timestamp}\] \"%{WORD:request_method} %{DATA:request_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:response_length} \"%{DATA:request_referer}\" \"%{DATA:user_agent}\""}
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
}

# /etc/logstash/conf.d/gunicorn-access-01.conf
input {
  file {
    path => "/home/deploy/log/fresh/gunicorn_access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => { "message" => "%{DATA:log_host} %{IPORHOST:remote_ip} -%{DATA:remote_user}- \[%{HTTPDATE:timestamp}\] \"%{WORD:request_method} %{DATA:request_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:response_length} \"%{DATA:request_referer}\" \"%{DATA:user_agent}\""}
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "gunicorn-access-%{+YYYY.MM.dd}"
  }
}

There are mainly two differences in these two conf files: the input file path and the output elasticsearch index.
I think these are enough for logstash to send different logs to different elasticsearch indices.
But when I check elasticsearch, the two logs is mixed up with each other, some logs in nginx access log file are found in gunicorn-access-* index and some gunicorn nginx.
So what is wrong with my config files?

2

Hi John,

You need to explicitly specify Logstash to use those two conf files as separate entities. By default, both the files will work in a piped configuration. That means, logstash will treat them as a single configuration.

You can either use conditionals or Multiple Pipeline configuration in Logstash to parse as two seperate indices.

https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

Hope this helps.

1 Like
3

Thank you @NerdSec, the multiple-pipelines solution works !

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.