HTTPD_COMBINEDLOG not found with docker


(Luis) #1

Hi all,

I'm using the ELK stack with Docker, for Logstash I'm using Logstash:latest (that is Logstash 2.4 at the moment, https://hub.docker.com/_/logstash/), researching a little in Github and code I see that the patterns for this version were moved to logstash-patterns-core (more info here). Not problem until here.

The problems comes when I checked the repository for logstash-patterns-core and the patterns for httpd I saw that the COMBINEDAPACHELOG is deprecated and now the correct is use the HTTPD_COMBINEDLOG, so I tried to use it on my Logstash with the next configuration:

if [type] == "nginx" and [input_type] == "access" {
    grok {
        match => [ "message" , "%{HTTPD_COMBINEDLOG}+%{GREEDYDATA:extra_fields}"]   
        overwrite => [ "message" ]
    }

   mutate {
       convert => ["response", "integer"]
       convert => ["bytes", "integer"]
       convert => ["responsetime", "float"]
    }

   geoip {
       source => "clientip"
       target => "geoip"
       add_tag => [ "nginx-geoip" ]
    }

   date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
        remove_field => [ "timestamp" ]
    }

   useragent {
        source => "agent"
    }
} 

But it fails and show me the next error (quite unreadable):

{:timestamp=>"2016-10-26T09:32:59.288000+0000", :message=>"Pipeline aborted due to error", :exception=>"Grok::PatternError", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:123:in 'compile'", "org/jruby/RubyKernel.java:1479:in 'loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:93:in 'compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:in 'register'", "org/jruby/RubyArray.java:1613:in 'each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in 'register'", "org/jruby/RubyHash.java:1342:in 'each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in 'register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in 'start_workers'", "org/jruby/RubyArray.java:1613:in 'each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in 'start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in 'run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in 'start_pipeline'"], :level=>:error}

If I try with the COMBINEDAPACHELOG don't show the error but it fails parsing the log (I can put an example if is required, is not the main of the question but it can be a possible workaround)

Any clue about this error? thanks in advance!


(Magnus Bäck) #2

The problems comes when I checked the repository for logstash-patterns-core and the patterns for httpd I saw that the COMBINEDAPACHELOG is deprecated and now the correct is use the HTTPD_COMBINEDLOG, so I tried to use it on my Logstash with the next configuration:

The HTTPD_COMBINEDLOG pattern isn't available in the logstash-patterns-core plugin that ships with Logstash 2.4. You might be able to upgrade the plugin though.

If I try with the COMBINEDAPACHELOG don't show the error but it fails parsing the log (I can put an example if is required, is not the main of the question but it can be a possible workaround)

If you want help with the parse failure we need to see what the input looks like.


(Luis) #3

Hi,
I'm moving to 5.0, so this is not a problem now, but thanks for the help :smiley:


(Magnus Bäck) #4

I'd expect HTTPD_COMBINEDLOG and COMBINEDAPACHELOG to be identical, so if you're having parse errors with the last one I'd be surprised if upgrading to Logstash 5 will help.


(David Česal) #5

Even with Logstash 5.4 default installation, HTTPD_COMBINEDLOG is not available. Solution is to update plugin with command /usr/share/logstash/bin/logstash-plugin update and restart logstash (systemctl restart logstash).


#6

Hi,
I am currently using logstash 5.4, as per documentation I used "COMBINEDAPACHELOG" for parsing my apache logs and it's working fine. When read the file mentioned below *, I saw "COMBINEDAPACHELOG" stated under "Deprecated" section. I know for sure it's not deprecated yet as it's parsing my logs correctly.

Questions:

  1. Is this feature going to be deprecated in the near future?
  2. If not the it would be nice if the response time can also be added to this feature's parsing abilities as it's quite common in production logs.

I am new to ELK and would like to head in the right direction keeping in mind future minor and major upgrades.

THANKS!!

*/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.0/patterns/httpd


(Magnus Bäck) #7

Is this feature going to be deprecated in the near future?

Probably not, but if there's a replacement pattern you might as well start using it.

If not the it would be nice if the response time can also be added to this feature's parsing abilities as it's quite common in production logs.

Sure, but it's not part of the Combined log format. Additionally, different web servers express response time in different units (Apache microseconds, Tomcat milliseconds, Nginx seconds). I suggest you use COMBINEDAPACHELOG (or something equivalent) together with additions that match your particular format.


#8

Thanks for the prompt reply.