HTTPD_COMBINEDLOG not found with docker

Elastic Stack Logstash
1

Hi all,

I'm using the ELK stack with Docker, for Logstash I'm using Logstash:latest (that is Logstash 2.4 at the moment, https://hub.docker.com/_/logstash/), researching a little in Github and code I see that the patterns for this version were moved to logstash-patterns-core (more info here). Not problem until here.

The problems comes when I checked the repository for logstash-patterns-core and the patterns for httpd I saw that the COMBINEDAPACHELOG is deprecated and now the correct is use the HTTPD_COMBINEDLOG, so I tried to use it on my Logstash with the next configuration:

if [type] == "nginx" and [input_type] == "access" {
    grok {
        match => [ "message" , "%{HTTPD_COMBINEDLOG}+%{GREEDYDATA:extra_fields}"]   
        overwrite => [ "message" ]
    }

   mutate {
       convert => ["response", "integer"]
       convert => ["bytes", "integer"]
       convert => ["responsetime", "float"]
    }

   geoip {
       source => "clientip"
       target => "geoip"
       add_tag => [ "nginx-geoip" ]
    }

   date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
        remove_field => [ "timestamp" ]
    }

   useragent {
        source => "agent"
    }
}

But it fails and show me the next error (quite unreadable):

{:timestamp=>"2016-10-26T09:32:59.288000+0000", :message=>"Pipeline aborted due to error", :exception=>"Grok::PatternError", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:123:in 'compile'", "org/jruby/RubyKernel.java:1479:in 'loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:93:in 'compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:in 'register'", "org/jruby/RubyArray.java:1613:in 'each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in 'register'", "org/jruby/RubyHash.java:1342:in 'each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in 'register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in 'start_workers'", "org/jruby/RubyArray.java:1613:in 'each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in 'start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in 'run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in 'start_pipeline'"], :level=>:error}

If I try with the COMBINEDAPACHELOG don't show the error but it fails parsing the log (I can put an example if is required, is not the main of the question but it can be a possible workaround)

Any clue about this error? thanks in advance!

2

The problems comes when I checked the repository for logstash-patterns-core and the patterns for httpd I saw that the COMBINEDAPACHELOG is deprecated and now the correct is use the HTTPD_COMBINEDLOG, so I tried to use it on my Logstash with the next configuration:

The HTTPD_COMBINEDLOG pattern isn't available in the logstash-patterns-core plugin that ships with Logstash 2.4. You might be able to upgrade the plugin though.

If I try with the COMBINEDAPACHELOG don't show the error but it fails parsing the log (I can put an example if is required, is not the main of the question but it can be a possible workaround)

If you want help with the parse failure we need to see what the input looks like.

3

Hi,
I'm moving to 5.0, so this is not a problem now, but thanks for the help :smiley:

4

I'd expect HTTPD_COMBINEDLOG and COMBINEDAPACHELOG to be identical, so if you're having parse errors with the last one I'd be surprised if upgrading to Logstash 5 will help.

5

Even with Logstash 5.4 default installation, HTTPD_COMBINEDLOG is not available. Solution is to update plugin with command /usr/share/logstash/bin/logstash-plugin update and restart logstash (systemctl restart logstash).

6

Hi,
I am currently using logstash 5.4, as per documentation I used "COMBINEDAPACHELOG" for parsing my apache logs and it's working fine. When read the file mentioned below *, I saw "COMBINEDAPACHELOG" stated under "Deprecated" section. I know for sure it's not deprecated yet as it's parsing my logs correctly.

Questions:

  1. Is this feature going to be deprecated in the near future?
  2. If not the it would be nice if the response time can also be added to this feature's parsing abilities as it's quite common in production logs.

I am new to ELK and would like to head in the right direction keeping in mind future minor and major upgrades.

THANKS!!

*/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.0/patterns/httpd

7

Is this feature going to be deprecated in the near future?

Probably not, but if there's a replacement pattern you might as well start using it.

If not the it would be nice if the response time can also be added to this feature's parsing abilities as it's quite common in production logs.

Sure, but it's not part of the Combined log format. Additionally, different web servers express response time in different units (Apache microseconds, Tomcat milliseconds, Nginx seconds). I suggest you use COMBINEDAPACHELOG (or something equivalent) together with additions that match your particular format.

8

Thanks for the prompt reply.

9

Using elastic stack 7.3.0 and when trying to use Kibana Grok Debugger with "%{HTTPD_COMBINEDLOG}" pattern I get the following error:

[parse_exception] [patterns] Invalid regex pattern found in: [%{HTTPD_COMBINEDLOG}]. Unable to find pattern [HTTPD_COMBINEDLOG] in Grok's pattern dictionary, with { property_name="patterns" & processor_type="grok" }

Using %{COMBINEDAPACHELOG} works.
Shouldn't HTTPD_COMBINEDLOG be available after more than 2 years of deprecating HTTPD_COMBINEDLOG?

10

In logstash both work, and one is defined in terms of the other

patterns/httpd:COMBINEDAPACHELOG %{HTTPD_COMBINEDLOG}

Not sure why it is different in kibana.