Error to run logstash

Elastic Stack Logstash
1

indent preformatted text by 4 spacesI got following error on console when tried to start the logstash.

root@111:/opt/logstash/bin# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/10-syslog-filter.conf --configtest
Error: Expected one of #, => at line 4, column 18 (byte 39) after filter {

grok {
PATHELEM {:level=>:error}

Here is my logstash conf file:

filter {

grok {
PATHELEMindent preformatted text by 4 spaces [^/]+
TASKPATH ^/var/lib/mesos/slave/slaves/%{PATHELEM:agent}/frameworks/%{PATHELEM:framework}/executors/%{PATHELEM:executor}/runs/%{PATHELEM:run}
match => ["file" , "%{TASKPATH}" ]
}
}

2

Can you paste the entire config. Please use the code formatting. It is very difficult to read the JSON.

3

grok {
PATHELEMindent preformatted text by 4 spaces [^/]+
TASKPATH ^/var/lib/mesos/slave/slaves/%{PATHELEM:agent}/frameworks/%{PATHELEM:framework}/executors/%{PATHELEM:executor}/runs/%{PATHELEM:run}
match => ["file" , "%{TASKPATH}" ]
}
}

Is this actually what your configuration looks like? You simply can't define new grok patterns inside the grok filter. You have to define them in a separate file.

4

thank you so much. I will make this change and try.

5

I crate a separate pattern file under pattern directory and content of this file is :

THELEM [^/]+
TASKPATH ^/var/lib/mesos/slave/slaves/%{PATHELEM:agent}/frameworks/% indent preformatted text by 4 spaces{PATHELEM:framework}/executors/% indent preformatted text by 4 spaces{PATHELEM:executor}/runs/%indent preformatted text by 4 spaces{PATHELEM:run}

and now logstash conf file look like this

filter {
grok {
patterns_dir => "/pattern"
match => { "file" => "%{TASKPATH}" }
}
}

but got following error when tried to start the logstash.

{:timestamp=>"2016-08-15T10:28:03.118000-0500", :message=>"An unexpected `error occurred!", :error=>#<Grok::PatternError: pattern %{TASKPATH} not defined>, :class=>"Grok::PatternError"}

I followed this https://docs.mesosphere.com/1.7/administration/logging/filter-elk/ link as guide line.

Any help is highly appreciated.

6

even I got the same error after providing the complete path of pattern file like
patterns_dir => "/etc/logstash/pattern/"

Thanks
Rai

7

With all the "indent preformatted text by 4 spaces" garbage it's hard to understand what the file actually looks like.

Where's the definition of PATHELEM that you're using in multiple places in TASKPATH? Did you mean THELEM instead?

8

Hi Magnu
I am really sorry for formatting issue. As you mentioned before to use </> to format the text and that's what I did.

pattern file

PATHELEM [^/]+

TASKPATH ^/var/lib/mesos/slave/slaves/%{PATHELEM:agent}/frameworks/%{PATHELEM:framework}/executors/%{PATHELEM:executor}/runs/%{PATHELEM:run}

beats-input.conf file

input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

mesosphere-filter.conf

filter {
grok {
patterns_dir => "/etc/logstash/pattern/"
match => { "file" => "%{TASKPATH}" }
}
}

elasticserach-out.conf

output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Hope it will be in formatted form.