Grok isn't getting parsed (grok debugger is parsing it fine but it's throwing error in logstash)

sudhir_singh · November 22, 2023, 9:05am

 - "22/Nov/2023:12:21:04 +0530" 196.24.23.101 GET "GET /api/status HTTP/1.1" 191 200 439 83 - "nginx/1.23.4 (health check server_103.225.61.177_Pool-1_http_ok)" 127.0.0.1:8080 200 0.002 0.002 0.000 0.002 - pauth.mumbcms.hydtimes.com - - - - - - - -
103.18.141.27, 23.45.91.202, 23.212.50.121 "22/Nov/2023:12:21:05 +0530" 196.21.108.44 POST "POST /oauth/api/merchant/BOI/token HTTP/1.1" 1447 200 2004 867 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36" 127.0.0.1:8080 200 0.943 0.942 0.000 0.942 - pauth.mumbcms.hydtimes.com vaCo1Dl9SGo2wVmzE78jo0FdSZShl9flZ0v0fO9hgVRCv - b2a8e883ec676f417520f422068a4742 04e2b3dc222d26d7ffa0ed3d3285cce6 iti219571 - {\x22grantType\x22:\x22refresh_token\x22,\x22ticketId\x22:\x223e90c3acad414c2bb5b6d5be4a543ed4\x22,\x22allMerchant\x22:true} -
93.112.40.146 "22/Nov/2023:12:21:05 +0530" 196.21.246.35 POST "POST /oauth/api/merchant/ET/token?ref=desktop HTTP/1.0" 4479 200 2550 835 https://mumbcms.hydtimes.com/ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" 127.0.0.1:8080 200 0.598 0.598 0.001 0.598 - oauth2.mumbcms.hydtimes.com 66911495-ea51-4e3f-babb-8a5d5ae41e80 - b2a8e883ec676f417520f422068a4742 undefined 2o3hbq2uwdcly1f5bd1io03r1 2o3hbq2uwdcly1f5bd1io03r1 {\x22grantType\x22:\x22refresh_token\x22,\x22ticketId\x22:\x22e9a7e69fc8714bf1ad894ed13e60701c\x22,\x22deviceDetail\x22:\x22x64\x22,\x22allMerchant\x22:true} e9a7e69fc8714bf1ad894ed13e60701c
103.18.142.26, 23.45.91.202, 104.124.54.69 "22/Nov/2023:13:55:34 +0530" 196.21.108.44 POST "POST /api/token/generate HTTP/1.1" 761 200 1284 928 - "-" 127.0.0.1:8080 200 1.119 1.120 0.001 1.119 - pauth.mumbcms.hydtimes.com SM-G611FF_3ecaa1dd7ee49883 827f918a54aa4545ac439148275e772d 3bf38ae9028de021754be41c7b61170f - - - grant_type=refresh_token&restrict_input=0&merchantCode=NBTGOLD& -

And Below is my pattern

{ "message" => "(?:%{IP:ip}|)%{SPACE:remoteip}(, %{IP})* \"%{HTTPDATE:timestamp}\" %{IP:clientip} %{WORD:method} "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:request_length} %{NUMBER:status} %{NUMBER:bytes_sent} (?:%{NUMBER:body_bytes_sent}|-) %{DATA:referer} %{QS:agent} %{DATA:upstream_addr} %{DATA:upstream_status} %{DATA:request_time} %{DATA:upstream_response_time} %{DATA:upstream_connect_time} %{DATA:upstream_header_time} %{DATA:upstream_cache_status} %{NOTSPACE:domain} %{DATA:X_DEVICE_ID} (?:%{DATA:X_TICKET_ID}|-) %{DATA:X_CLIENT_ID} %{DATA:X_SITE_APP_CODE} %{DATA:X_SSO_ID} %{DATA:ssoid} %{GREEDYDATA:additional_info}"}

sudhir_singh · November 22, 2023, 11:32am

Below is the error which logstash is throwing and there is no missing brackets or such errors.

 Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\", \"}\" at line 14, column 130 (byte 328) after filter {\nif [type] == \"accesslogs\" {\n    grok {\nmatch => { \"message\" => \"(?:%{IPORHOST:ip}|)%{SPACE:remoteip}(, %{IP})* \\\"%{HTTPDATE:timestamp}\\\" %{IP:clientip} %{WORD:method} \"", :backtrace=>["/opt/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:383:in `block in converge_state'"]}

leandrojmp · November 22, 2023, 12:29pm

Your error is not in grok, Logstash is not able to start your pipeline because of some configuration error.

Please share your entire configuration file.

sudhir_singh · November 22, 2023, 1:49pm

Below configuration I'm using

input {
file {
    path => "/log/et-oauth/app.log"
    type => "applog"
}
file {
    path => "/var/log/nginx/access.log"
    type => "accesslogs"
}
}
filter {
if [type] == "accesslogs" {
    grok {
match => { "message" => "(?:%{IP:ip}|)%{SPACE:remoteip}(, %{IP})* "%{HTTPDATE:timestamp}" %{IP:clientip} %{WORD:method} "%{WORD:verb} %{DATA:request}%{NUMBER:httpversion}" %{NUMBER:request_length} %{NUMBER:status} %{NUMBER:bytes_sent} (?:%{NUMBER:body_bytes_sent}|-) %{DATA:referer} %{QS:agent} %{DATA:upstream_addr} %{DATA:upstream_status} %{DATA:request_time} %{DATA:upstream_response_time} %{DATA:upstream_connect_time} %{DATA:upstream_header_time} %{DATA:upstream_cache_status} %{NOTSPACE:domain}%{DATA:X_DEVICE_ID} (?:%{DATA:X_TICKET_ID}|-) %{DATA:X_CLIENT_ID} %{DATA:X_SITE_APP_CODE} %{DATA:X_SSO_ID} %{DATA:ssoid} %{GREEDYDATA:additional_info}" }

remove_field => ["tags","timestamp"]
 }


  if [size] == "-" {
    mutate {
       replace => [ "size", "0" ]
     }
    }
  }
}

output {
if [type] == "applog" {
  elasticsearch {
    hosts => ["logelastic.nytimes.com:80"]
    index => "et-oauth-applog-%{+YYYY.MM.dd}"
  }
  }
}

output {
if [type] == "accesslogs" {
  elasticsearch {
    hosts => ["logelastic.nytimes.com:80"]
    http_compression => true
    index => "et-oauth-webservernew-%{+YYYY.MM.dd}"
  }
  }
}

sudhir_singh · November 22, 2023, 1:50pm

But if I just replace the grok pattern with below pattern it works fine but it doesn't parse the fields properly

%{DATA:xforwardedfor} \"%{HTTPDATE:timestamp}\" +%{IP:clientip} %{WORD:method} \"+%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" +%{NUMBER:request_length} %{NUMBER:status} (?:%{NUMBER:bytesSent}|-) (?:%{NUMBER:BodybytesSent}|-) %{DATA:referrer} %{QS:agent} %{DATA:upstream_addr} %{DATA:upstreamstatus} %{DATA:request_time} %{DATA:upstream_response_time} %{DATA:upstream_connect_time} %{DATA:upstream_header_time} %{DATA:upstream_cache_status} %{NOTSPACE:domain} %{DATA:X_DEVICE_ID} (?:%{DATA:X_TICKET_ID}|-) %{DATA:X_CLIENT_ID} %{DATA:X_SITE_APP_CODE} %{DATA:X_SSO_ID} %{DATA:ssoid} \{%{DATA:request_body}\}\s+%{GREEDYDATA:TicketId}

leandrojmp · November 22, 2023, 2:14pm

This last grok worked for me for two of your sample messages and failed for other two.

You need to investigate why it is failing for those specific messages, maybe you will need to build two different groks depending on your messages.

Your initial error was unrelated to grok, it was related to Logstash not being able to start your pipeline, the configuration you shared had some unescaped double quotes in your grok, if always need to escape the double quotes.

sudhir_singh · November 22, 2023, 2:35pm

Yes I understand I tried to do that but I couldn't do as it's not clear to me at the moment. But the initial error is concerning if it works then it will parse everything.
I have two grok pattern which solves the two different messages but I fail to apply that through if else block. I will provide the grok pattern. If you could help in the if else block would be great.

leandrojmp · November 22, 2023, 2:43pm

This error:

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main

Means that it is something wrong in your configuration and Logstash cannot start the pipeline.

It was probably caused by this:

match => { "message" => "(?:%{IP:ip}|)%{SPACE:remoteip}(, %{IP})* "%{HTTPDATE:timestamp}"

You have double quotes in your pattern without escaping it, it will not work, you need to escape all double quotes, that is why the last pattern you shared worked, the double quotes are escaped.

sudhir_singh · November 22, 2023, 2:45pm

To escape I just have to use backward slash () before double quotes right ?

leandrojmp · November 22, 2023, 2:47pm

Yeah, just use \".

sudhir_singh · November 22, 2023, 3:06pm

This particular log is getting parsed in debugger but not in logstash

2402:8100:2456:eab1:1:2:226b:58f7, 23.45.91.202, 23.212.50.121 "22/Nov/2023:20:31:38 +0530" 172.29.108.47 OPTIONS "OPTIONS /api/token/generate HTTP/1.1" 961 200 915 0 https://navbharattimes.indiatimes.com/ "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Mobile Safari/537.36" 127.0.0.1:8080 200 0.003 0.003 0.001 0.003 - oauth.economictimes.indiatimes.com - - - - - - - -

This is what it is showing in kibana

message
2402:8100:2456:eab1:1:2:226b:58f7, 23.45.91.202, 23.212.50.121 "22/Nov/2023:20:31:38 +0530" 172.29.108.47 OPTIONS "OPTIONS /api/token/generate HTTP/1.1" 961 200 915 0 https://navbharattimes.indiatimes.com/ "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Mobile Safari/537.36" 127.0.0.1:8080 200 0.003 0.003 0.001 0.003 - oauth.economictimes.indiatimes.com - - - - - - - -

This is what it is showing kibana

system · December 20, 2023, 3:07pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.