Grok Pattern for Apache Error Logs

Elastic Stack Logstash
1

Below is the error log:

[Mon Nov 28 04:38:24 2016] [error] [client 10.114.34.43] File does not exist: /fep10/oraapps/appl/fep10comn/portal/FEP10_j201s648/favicon.ico

I had written grok pattern as
"message"=> "[(?%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})] [%{LOGLEVEL:loglevel}] [client %{IP:clientip}:.*] %{GREEDYDATA:errormsg}"

It gives an grokparsefailure in ES. Please suggest the appropriate and let me know where i am making mistake.

TIA

2

Please edit your post and format the grok expression as preformatted text (e.g. using the toolbar button) so that it doesn't get mangled.

3

I'm trying out with nginx and even I get _grokparsefailure error along with _geoip_lookup_failure.
Here is my conf.

filter
{
    if [type] == "nginx" 
    {
        grok {
            match => {
                "message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] "%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'
            }
        }
        date {
            match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]
            locale => en
        }

        geoip {
            source => "remote_ip"
            target => "geoip"
        }

        useragent {
            source => "agent"
            target => "user_agent"
        }
    }
}
4

But that's your access log filters and you wanted help with your error log.

5

Oh Sorry I forgot to mention were the errors were shown. Actually it shows up in KIBANA.

image
image.png1584×148 11.4 KB

6
  • What do you want help with, error logs or access logs?
  • For the kind of log you want help with, please show
    • the configuration used (formatted as preformatted text) and
    • an example message processed by Logstash (use a stdout { codec => rubydebug } output).
7

Can you be more clear how to do it, As i am very new to this . I dont understand it easily

8

How to do what?

9

[Mon Nov 28 04:38:24 2016] [error] [client 10.114.34.43] File does not exist: /fep10/oraapps/appl/fep10comn/portal/FEP10_j201s648/favicon.ico

Grok Pattern for the above error.

10

Okay. Then please provide the things I asked for in the bullet list a few posts up.

11

The output which i get,

message:[Mon Nov 28 04:42:21 2016] [error] [client 10.114.34.43] client denied by server configuration: /fep10/oraapps/appl/fep10comn/java/oracle/forms/engine
@version:1
@timestamp:14/9/2017 23:02:06 PM
host:j051s319.jci.com
path:/data/Ops_analytics_EBS/oracle_ebs_eu/j201s648/apacheerror.log
type:logs
tags:_grokparsefailure
_id:AV6Bcqn7o5C-nkf485it
_type:logs
_index:logstash_parsing_error_index
_score:1

12

Sorry, I don't have time for this when you never provide all information I ask for. Maybe someone else has more patience.

13

Code :

input {
file {
path => "/data/Ops_analytics_EBS/oracle_ebs_eu/j201s648/error_20170913.log"
start_position => "beginning"
type => "apache_error"
}
}

filter {
grok {
match =>["message","[(?%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})] [%{LOGLEVEL:loglevel}] [client %{IP:clientip}:.*] %{GREEDYDATA:errormsg}"]

add_field => {
	"eventName"=> "grok"
	}
}
geoip {
source => "clientip"
}

}

output {
elasticsearch {
hosts => "localhost:9200"
index => "logstash_parsing_error_index"
}
}

The Error log which i want to push in ES is

[Mon Aug 22 02:41:35 2016] [error] [client 10.96.159.140] File does not exist: /fep10/oraapps/appl/fep10comn/portal/FEP10_j201s648/favicon.ico

The output which is being indexed in ES or logstash screen is

message:[Mon Nov 28 04:42:21 2016] [error] [client 10.114.34.43] client denied by server configuration: /fep10/oraapps/appl/fep10comn/java/oracle/forms/engine
@version:1
@timestamp:14/9/2017 23:02:06 PM
host:j051s319.jci.com
path:/data/Ops_analytics_EBS/oracle_ebs_eu/j201s648/apacheerror.log
type:logs
tags:_grokparsefailure
_id:AV6Bcqn7o5C-nkf485it
_type:logs
_index:logstash_parsing_error_index
_score:100:

I want the output in a format where each part of error should be aligned to the data_name which is mentioned in grok pattern .

n would like to know the cause of _grokparsefailure as well.

14

You have a : after the client pattern which does not seem to match the entry. What happens if you remove this?

1 Like
15

It does cleared grokparsefailure error. tnx for that .

I want my error-time to converted in time format. It is getting stored as string. Can you plz guide me where i am going wrong.

Is there any regex pattern where i can convert string into timestamp format?

16

You should use a date filter to process the date and get it into the right format.

17

Thanks for the suggestion , I used date filter before only , now added target in date filter and it works fine :slight_smile:

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.