Configuration Validation

Elastic Stack Logstash
1

I have a web server. I have a two-server es cluster. I want to pipe the access logs to the es cluster using logstash.

Is this the right configuration for the web server? How do I configure the ES cluster to accept the data?

web server

input {
  file {
       path => "/var/log/httpd/miss*log"
       type => "apache"
       }
     }
filter {
  if [type] == "apache" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }

    if [clientip] in ["10.1.88.11", "10.1.88.12", "10.1.88.13", "10.1.88.14", "10.1.88.15", "10.1.88.16", "10.1.42.117", "10.1.42.118", "10.1.42.119", "10.1.88.21", "10.1.88.22", "10.1.88.23", "10.1.88.24", "10.1.88.25", "10.1.88.26", "10.1.42.127", "10.1.42.128", "10.1.42.129"]  {
       drop {}
    }
  }
}

output {
    elasticsearch {
        cluster => "muostats"
        host => "10.210.2.98:9300"
        protocol => "node"
        index_type => "apache"
	workers => "1"
    }
}
  • Output host => is 'a' IP from my current configuration while I troubleshoot.
2

The configuration looks pretty reasonable. Give it a shot. If you haven't locked down your ES cluster in any particular way you don't have to do anything special to have it accept the messages.

3

See Edit - I may have un-jugged myself.

It's not working. Nothing falling into the cluster. I did redir the output to file and it doesn't create the file, either.

output {
        file { path =>"/var/log/logstash/output.txt" }
}

This is a valid path .. I'm watching apache write files to it like gangbusters

/var/log/httpd/miss*log"

I even touched, and told the file it belonged to user:group logstash ...

 ls -al /var/log/logstash/output.txt
-rw-r--r-- 1 logstash logstash 0 Jul  7 12:33 /var/log/logstash/output.txt

If I'm doing component isolation right there is a thing wrong with input, but I don't see it ..

# cat logstash.conf
input {
  file {
       path => "/var/log/httpd/miss*log"
       type => "apache"
       }
     }
filter {
  if [type] == "apache" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }

    if [clientip] in ["10.1.88.11", "10.1.88.12", "10.1.88.13", "10.1.88.14", "10.1.88.15", "10.1.88.16", "10.1.42.117", "10.1.42.118", "10.1.42.119", "10.1.88.21", "10.1.88.22", "10.1.88.23", "10.1.88.24", "10.1.88.25", "10.1.88.26", "10.1.42.127", "10.1.42.128", "10.1.42.129"]  {
       drop {}
    }
  }
}
output {
        file { path =>"/var/log/logstash/output.txt" }
}

EDIT

I went back to the basics, put them in a conf file called test.conf and manually executed logstash

/opt/logstash/bin/logstash -f test.conf

and LO the apache access logs, correctly formatted, started filling up the file. So a problem with the startup script? I can work around that.