Separate ELK pattern for log files

Elastic Stack Elasticsearch
1

Hi team,
Can any one help me to find the solution for my below requirement.

I have two apache server and I want to send the apache access and error logs to elk server via filebeat apache module to logstash. I configured apache.yml and apache.conf file in logstash.
example:
in logstash,
apache1.conf - srever1
apache2.conf - server2. with different index name.
These configuration was working fine, but the issue is it sending the logs to both index name.
see my apache.conf file below,( apache2.conf file also same , only the difference is file path and index name)

input {
  beats {
     port => 5044
 }
}

filter {
    if [log][file][path] in ["/var/log/apache2/sfsite-access_log","/var/log/apache2/sfapi-access_log"] {
      grok {
        match => { "message" => "\[%{HTTPDATE:time_stamp}\] %{HOSTNAME:domain_name} \"%{WORD:method} /%{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} (?:%{NUMBER:response_bytes}|-)  %{QS:agent} %{NOTSPACE:page_url} %{QS:agent_type}" }

 }
      grok {
        match => { "message" => "%{IP:client_ip}" }
 }
      geoip {
        source => "[client_ip]"
        ecs_compatibility => disabled
      }
}

else if [log][file][path] in ["/var/log/apache2/sfsite-error_log","/var/log/apache2/sfapi-error_log"] {
      grok {
        match => { "message" => "\[%{HTTPDERROR_DATE:timestamp-error}\] \[%{DATA:loglevel}%{SPACE}\] \[%{DATA:process-id}%{SPACE}\]%{SPACE}\[%{DATA:client-ip}\] %{GREEDYDATA:message} (\[%{NOTSPACE:Index}\]\[%{NUMBER:shards}\])?%{GREEDYDATA} %{GREEDYDATA:referred_url}" }
   }
 }
}


output {
  elasticsearch {
    hosts => ["https://a.a.a.a:9200"]
    index => "index-testing-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "pppppp"
    ssl => true
    cacert => "/etc/logstash/http_ca.crt"
  }
}

can any one help me out this.

2

You need to have conditionals in the output as well, the same ones that you are using in the filter section.

3

Hi @Leandrojmp

Can you give any examples for configure the conditional statements in output sections as well. It will more helpful.

Also is there any documentation..!

4

It is exactly the same ones you are using in your filter.

It would be something like this:

output {
  if [log][file][path] in ["/var/log/apache2/sfsite-access_log","/var/log/apache2/sfapi-access_log"] {
    elasticsearch { your output for this indice}
  } else if [log][file][path] in ["/var/log/apache2/sfsite-error_log","/var/log/apache2/sfapi-error_log"] {
    elasticsearch { your output for this indice}
  }
}

The documentation about conditionals is this one.

5

@leandrojmp

Thank you so much for your guidance.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.