# Logstash configuration file error filtering

**URL:** <https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338>\
**Category:** Logstash\
**Created:** [January 30, 2019, 11:01am UTC](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338 "2019-01-30T11:01:57Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![althair34](https://avatars.discourse-cdn.com/v4/letter/a/898d66/32.png) [@althair34](https://discuss.elastic.co/u/althair34)\
**Post date:** [January 30, 2019, 11:01am UTC](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338/1 "2019-01-30T11:01:58Z")

</div>

Hello, I'm starting to use ELK and I'm attempting to start logstash with a custom logstash.conf file but I've an error :

```
 Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, \", ', -, [, { at line 10, column 37 (byte 149) after filter {\n if \"HTTP\" in [message] {\n grok {\n mapping => { \"message\" => ", :backtrace=>["/opt/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:42:in `block in execute'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:92:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:92:in `exclusive'", "/opt/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:317:in `block in converge_state'"

```

Here my configuration file :

```
input {

```

beats {  
port =\> 5044  
codec =\> "json"  
}  
}  
filter {  
if [message] =~ "HTTP" {  
grok {  
mapping =\> { "message" =\> %{TIMESTAMP\_ISO8601:timestamp} %{WORD:type} %{LOGLEVEL:level} "%{WORD:method} %{URIPATHPARAM:url}" %{INT:code} %{INT:bytes} - %{GREEDYDATA:response\_time} }  
}  
}  
else if [message] =~ "APP" {  
grok {  
mapping =\> { "message" =\> %{TIMESTAMP\_ISO8601:timestamp} %{WORD:type} %{LOGLEVEL:level} %{GREEDYDATA:jsonstring} }  
}  
json {  
source =\> "jsonstring"  
target =\> "doc"  
}  
mutate {  
add\_field =\> {  
"code" =\> "%{[doc][code]}"  
"message" =\> "%{[doc][message]}"  
}  
}  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost"]  
}  
}

Logs sample I want to get back (docker stdout logs) :

1. HTTP request logs  
2019-01-29T18:35:15.423Z HTTP INFO "POST /myroute/?param1=test" 201 41 - 44.014 ms

2. APP logs  
2019-01-29T18:48:19.657Z APP ERROR : {"code":201,"message":"ok"}

discriminator : APP or HTTP

What is wrong with this config ? I tried a change a lot things specially around line 37 but i don't understand why it doesn't work.

Thank you very much for your help 🙂

---

<div class="post-metadata">

**Author:** ![danhermann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danhermann/32/33024_2.png) [@danhermann](https://discuss.elastic.co/u/danhermann)\
**Post date:** [January 30, 2019, 11:14am UTC](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338/2 "2019-01-30T11:14:33Z")

</div>

@althair34, you need to enclose your grok patterns in quotes. E.g.:

```auto
grok {
mapping => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:type} %{LOGLEVEL:level} %{GREEDYDATA:jsonstring}" }
}

```

---

<div class="post-metadata">

**Author:** ![althair34](https://avatars.discourse-cdn.com/v4/letter/a/898d66/32.png) [@althair34](https://discuss.elastic.co/u/althair34)\
**Post date:** [January 30, 2019, 11:50am UTC](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338/3 "2019-01-30T11:50:41Z")

</div>

Thank you for your help. I just have a question :

How can escape me double quotes in message ? Should I use simple quote to enclose my grok pattern ?

Thank you again

---

<div class="post-metadata">

**Author:** ![danhermann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danhermann/32/33024_2.png) [@danhermann](https://discuss.elastic.co/u/danhermann)\
**Post date:** [January 30, 2019, 11:58am UTC](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338/4 "2019-01-30T11:58:25Z")

</div>

You can use either single quotes around your grok pattern or a backslash to escape special characters within your grok pattern.

---

<div class="post-metadata">

**Author:** ![althair34](https://avatars.discourse-cdn.com/v4/letter/a/898d66/32.png) [@althair34](https://discuss.elastic.co/u/althair34)\
**Post date:** [January 30, 2019, 12:08pm UTC](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338/5 "2019-01-30T12:08:30Z")

</div>

I've almost fixed my configuration file. I have an error of deprecation on the output :

You are using a deprecated config setting "document\_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=\>"document\_type", :plugin=\>\<LogStash::Outputs::ElasticSearch index=\>"%{[@metadata][beat]}-%{+YYYY.MM.dd}", manage\_template=\>false, id=\>"6a469e616cda88c3ab1205d3def17747cd86ab278aa8b4bcabba84d12b7accf2", hosts=\>[[//localhost](https://localhost)], document\_type=\>"%{[@metadata][type]}", enable\_metric=\>true, codec=\>\<LogStash::Codecs::Plain id=\>"plain\_33572ebe-e2b0-4f11-97b6-e0bca5239b31", enable\_metric=\>true, charset=\>"UTF-8"\>, workers=\>1, template\_name=\>"logstash", template\_overwrite=\>false, doc\_as\_upsert=\>false, script\_type=\>"inline", script\_lang=\>"painless", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_initial\_interval=\>2, retry\_max\_interval=\>64, retry\_on\_conflict=\>1, action=\>"index", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5, timeout=\>60, pool\_max=\>1000, pool\_max\_per\_route=\>100, resurrect\_delay=\>5, validate\_after\_inactivity=\>10000, http\_compression=\>false\>}  
[2019-01-30T12:04:01,099][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50

Here my configuration file updated

```
input {

```

beats {  
port =\> 5044  
codec =\> "json"  
}  
}  
filter {  
if [message] =~ "HTTP" {  
grok {  
mapping =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp} %{WORD:type} %{LOGLEVEL:level} "%{WORD:method} %{URIPATHPARAM:url}" %{INT:code} %{INT:bytes} - %{GREEDYDATA:response\_time}" }  
}  
}  
else if [message] =~ "APP" {  
grok {  
mapping =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp} %{WORD:type} %{LOGLEVEL:level} %{GREEDYDATA:jsonstring}" }  
}  
json {  
source =\> "jsonstring"  
target =\> "doc"  
}  
mutate {  
add\_field =\> {  
"code" =\> "%{[doc][code]}"  
"message" =\> "%{[doc][message]}"  
}  
}  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost"]  
}  
}

---

<div class="post-metadata">

**Author:** ![danhermann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danhermann/32/33024_2.png) [@danhermann](https://discuss.elastic.co/u/danhermann)\
**Post date:** [January 30, 2019, 1:41pm UTC](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338/6 "2019-01-30T13:41:22Z")

</div>

That's just a warning that in the next major release of Elasticsearch, the `document_type` option will be removed. You can read more about that here: [https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 27, 2019, 1:41pm UTC](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338/7 "2019-02-27T13:41:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.