[Solved] Logstash sends logs to the same elasticsearch index even after I configured not to do so

I got two logs, one is nginx access log, another is gunicorn access log.
The two logs has similar contents, but I want them to be two different elasticsearch indices.
Here are my conf files:

# /etc/logstash/conf.d/nginx-access-01.conf
input {
  file {
    path => "/home/deploy/log/fresh/nginx_access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => { "message" => "%{DATA:log_host} %{IPORHOST:remote_ip} -%{DATA:remote_user}- \[%{HTTPDATE:timestamp}\] \"%{WORD:request_method} %{DATA:request_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:response_length} \"%{DATA:request_referer}\" \"%{DATA:user_agent}\""}
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
}
# /etc/logstash/conf.d/gunicorn-access-01.conf
input {
  file {
    path => "/home/deploy/log/fresh/gunicorn_access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => { "message" => "%{DATA:log_host} %{IPORHOST:remote_ip} -%{DATA:remote_user}- \[%{HTTPDATE:timestamp}\] \"%{WORD:request_method} %{DATA:request_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:response_length} \"%{DATA:request_referer}\" \"%{DATA:user_agent}\""}
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "gunicorn-access-%{+YYYY.MM.dd}"
  }
}

There are mainly two differences in these two conf files: the input file path and the output elasticsearch index.
I think these are enough for logstash to send different logs to different elasticsearch indices.
But when I check elasticsearch, the two logs is mixed up with each other, some logs in nginx access log file are found in gunicorn-access-* index and some gunicorn nginx.
So what is wrong with my config files?

Hi John,

You need to explicitly specify Logstash to use those two conf files as separate entities. By default, both the files will work in a piped configuration. That means, logstash will treat them as a single configuration.

You can either use conditionals or Multiple Pipeline configuration in Logstash to parse as two seperate indices.

https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

Hope this helps.

1 Like

Thank you @NerdSec, the multiple-pipelines solution works !

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.