Hi,
I'm logging Apache logs using the file input, and my Logstash is logging events both to the logstash-yyyy.mm.dd index and the custom one I specify (sometimes also creating duplicates in one of the indexes).
Here's the relevant config:
output {
if "localhost" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "logs-localhost-%{+YYYY.MM.dd}"
}
}
else if [log_custom_index] {
elasticsearch {
hosts => ["localhost:9200"]
index => "logs-%{log_custom_index}-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
}
}
}
When one between the "localhost" tag or "log_custom_index" field is found, a copy of the log is always sent to the logstash-yyyy.mm.dd index
Logstash is running as a service on an up-to-date Debian Stretch, installed via the Elastic apt repository
root@Server /etc/logstash/conf.d # dpkg -l | grep -E "elastic|logstash|kibana"
ii elasticsearch 6.4.0 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
ii elasticsearch-curator 5.5.4 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices.
ii kibana 6.4.0 amd64 Explore and visualize your Elasticsearch data
ii logstash 1:6.4.0-1 all An extensible logging pipeline
root@Server /etc/logstash/conf.d # apt-cache policy logstash
logstash:
Installed: 1:6.4.0-1
Candidate: 1:6.4.0-1
Version table:
*** 1:6.4.0-1 500
500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
100 /var/lib/dpkg/status
root@Server /etc/logstash/conf.d # apt-cache policy elasticsearch
elasticsearch:
Installed: 6.4.0
Candidate: 6.4.0
Version table:
*** 6.4.0 500
500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
100 /var/lib/dpkg/status
Things I tried:
Both suggestions from Log duplicated in multiple indexes
- "Try without the else clause"
If I comment out the else (and then restart logstash), logs are still sent to logstash-yyyy.mm.dd.
This caught me by surprise - "you should not see any log messages with this text in the log line
retrying failed action with response code"
I looked into both logstash-plain.log and the elasticsearch log; can't find it in either
Making sure that my if-else worked fine (from Problem with output elasticsearch data duplicate on index)
To do this, I redirected all the outputs to rubydebug, used stdin as the input to feed it log lines, and made sure that:
- with my normal configuration, the events were only parsed once and the tags that were supposed to be there, were.
- removing the "else" from "else if" the logs were parsed multiple times (which is consistent with what seems to be happening to me)
My filters (including the mutate that checks cidr for the "localhost" tag) work fine: the log lines stored in the logstash-yyyy.mm.dd index contain the tags/fields that are added manually, so this seems an issue with either the elasticsearch output sending the logs where it's not supposed to, which I think is probable, because the other cause could be a bug in elasticsearch itself storing the logs twice on receipt. I am unlikely to believe this because some (quick) testing with custom indexes (test-something) did not reproduce the issue.
Here's the full (redacted) config:
input {
file {
type => "apache2-accesslog"
path => [ "/var/log/apache2/example.org.access.log", "/var/log/apache2/example.org.access.log.1" ]
tags => [ "apache2", "example.org"]
}
file {
type => "apache2-accesslog"
path => [ "/var/log/apache2/elastic.example.org.access.log", "/var/log/apache2/elastic.example.org.access.log.1" ]
tags => [ "apache2", "elastic.example.org"]
add_field => { "log_custom_index" => "elasticsearch" }
}
file {
type => "apache2-accesslog"
path => [ "/var/log/apache2/access.log", "/var/log/apache2/access.log.1","/var/log/apache2/other_vhosts_access.log" , "/var/log/apache2/other_vhosts_access.log.1" ]
tags => [ "apache2", "other_vhosts"]
}
}
filter {
grok {
match => { "message" => [ "%{COMBINEDAPACHELOG}" ] }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
if [clientip] {
cidr {
add_tag => [ "localhost" ]
address => [ "%{clientip}" ]
network => ["127.0.0.0/8", "xxx.xxx.xxx.xxx/32", "yyy.yyy.yyy.yyy/32", "2a01:xxx:xxx:xxx::2/64", "fe80::/10", "::1/128" ]
}
}
}
output {
if "localhost" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "logs-localhost-%{+YYYY.MM.dd}"
}
}
else if [log_custom_index] {
elasticsearch {
hosts => ["localhost:9200"]
index => "logs-%{log_custom_index}-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
}
}
}
Any ideas on how to fix it?
Thanks!
