Invalid Configuration Error

Elastic Stack Logstash
1

Hi Team,
I am working on parsing apache access logs along with error logs to elastic via logstash index and facing invlaid configuration error. I am copying my filebeat input and logstash input for reference.

Filebeat Input:

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

  • type: log

enabled: true

paths:

  • /home/patcat/roddock_webservices/error*. .
  • /home/patcat/paddock_webservices/error*. .
    log_type: apache_error

paths:

  • /home/patcat/roddock_webservices/access*.*
  • /home/patcat/paddock_webservices/access*.*
    log_type: apache_access

exclude_lines: ['a10hm/1.0']

Logstash Conf file:

input {
beats {
port => "5068"
}
}
filter {
if [log_type] == "apache_error" {
grok {
match => { "message" => '[%{DATA:ApachErrortime}] [%{NOTSPACE:ApachErrorloglevel}] (?:[client %{IPORHOST:clientip}]) {0,1}%{GREEDYDATA:ApachErrorMessage}' }
}
overwrite => [ "message" ]
}
else if [log_type] == "apache_access" {
grok {
match => { "message" => '%{NUMBER:ResponseTime}\s%{COMBINEDAPACHELOG}\s%{GREEDYDATA:Ignore}' }
}
mutate {
rename => { "timestamp" => "Apachetimestamp" }
}
date {
match => [ "Apachetimestamp" , "ISO8601"]
}
mutate {
convert => { "bytes" => "integer" }
}
mutate {
convert => { "response" => "integer" }
}
mutate {
convert => { "ResponseTime" => "integer" }
}
mutate {
remove_field => [ "Ignore" , "message" ]
}
overwrite => [ "message" ]
}
}

Quick assistance is of high value. Thanks.

2

Always quote the full error message. You're not even saying if it's Logstash or Filebeat that's complaining.

overwrite => [ "message" ]

This belongs inside a grok filter.

3

Sorry, My bad the problem occurs while executing the logstash conf file. Also tried including the overwrite within grok filter...

[2018-08-13T00:01:10,736][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, => at line 10, column 32 (byte 306) after filter {
if [log_type] == "apache_error" {
grok {
match => { "message" => "[%{DATA:ApachErrortime}] [%{NOTSPACE:ApachErrorloglevel}] (?:[client %{IPORHOST:clientip}] ){0,1}%{GREEDYDATA:ApachErrorMessage}" }
overwrite => { "message"

4

overwrite => { "message"

This indicates that your configuration contains overwrite => { "message" } which is wrong. overwrite => [ "message" ] is correct.

5

Yes i agree, but i have done that change but still very similar issue with output section.

[2018-08-13T01:56:28,376][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, => at line 35, column 11 (byte 938) after filter {
if [log_type] == "apache_error" {
grok {
match => { "message" => "[%{DATA:ApachErrortime}] [%{NOTSPACE:ApachErrorloglevel}] (?:[client %{IPORHOST:clientip}] ){0,1}%{GREEDYDATA:ApachErrorMessage}" }
overwrite => [ "message" ]
}
}
else if [log_type] == "apache_access" {
grok {
match => { "message" => "%{NUMBER:ResponseTime}\s%{COMBINEDAPACHELOG}\s%{GREEDYDATA:Ignore}" }
overwrite => [ "message" ]
}
mutate {
rename => { "timestamp" => "Apachetimestamp" }
}
mutate {
convert => { "bytes" => "integer" }
}
mutate {
convert => { "response" => "integer" }
}
mutate {
convert => { "ResponseTime" => "integer" }
}
mutate {
remove_field => [ "Ignore" ]
}
}
output {
stdout
[2018-08-13T01:56:28,387][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit

6

Looks like you're not closing the if [log_type] == "apache_access" { conditional.

7

Yes closing the if condition resolved it , but the overwrite filter didn't have any impact i could only see the message filed being present in the output logs.

{
"source" => "/home/patcat/roddock_webservices/access_log.20180522",
"prospector" => {
"type" => "log"
},
"message" => "0 10.65.19.201 - - [22/May/2018:00:08:50 -0700] "GET /assets/css/print.css HTTP/1.1" 200 509 "https://roddoc.patqa.corp.rrk.com/profile-web/signin" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) li
ke Gecko" GET /assets/css/print.css "-" "" "HTTP/1.1" xconnx7660 "-" "-"",
"beat" => {
"hostname" => "xconnx7660.corp.rrk.com",
"version" => "6.2.4",
"name" => "xconnx7660.corp.rrk.com"
},
"@version" => "1",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"@timestamp" => 2018-08-13T09:58:43.635Z,
"offset" => 4386,
"host" => "xconnx7660.corp.rrk.com"
}

8

None of your filters execute since the log_typefield isn't set.

9

Yes, how do i fix it, is it the right forum or should i raise a separate request for this!

10

We can continue here. Please post your Filebeat configuration formatted as preformatted text (use the </> toolbar button or markdown notation) so we can see exactly what it looks like.

11

Only apache_access got picked up here and not apache_errors...

12

As I said I want to see the configuration formatted as preformatted text.

13 
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- type: log

  enabled: true

  paths:
    - /home/patcat/roddock_webservices/error*.*.*
    - /home/patcat/paddock_webservices/error*.*.*
  fields:
    log_type: apache_error
  fields_under_root: true


  paths:
    - /home/patcat/roddock_webservices/access*.*
    - /home/patcat/paddock_webservices/access*.*
  fields:
    log_type: apache_access
  fields_under_root: true

  exclude_lines: ['a10hm/1.0']

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

#setup.template.settings:
  #index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5070"]

Does it suffice!
Sorry didn't get your requirement earlier. Thanks.

14

Hi,
I am able to resolve this problem with home work. Thank you very much Magnus...

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.