# Log stash filters log message - need help

**URL:** https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076
**Category:** Logstash
**Created:** [March 4, 2020, 12:50pm UTC](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076 "2020-03-04T12:50:55Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)
#### Post date: [March 4, 2020, 12:50pm UTC](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076/1 "2020-03-04T12:50:56Z")

</div>

Below are the log stash fields

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/1/f1d23c9411ecec0bdb5b943b2a8e437edd00abb2.png)

Log stash error :

> [2020-03-02T16:48:48,354][WARN][logstash.filters.json] Error parsing json {:source=\>"message", :raw=\>"WITH CORRESPONDING LINE(S)", "HoldReason": "Invoice price exceeds purchase order price", "Supplier": "BIRLA CARBON BRASIL LTDA", "Requestor": "Curi, Henrique", "Buyer": "da Cruz, Breno", "OperatingUnit": "414\_OU", "LogType": "Business\_KPI", "KPI\_Description": "HumanTimeEquivalent", "Process\_Step": "Email-Sent", "HumanTimeEquivalent": 1200 }", :exception=\>#\<LogStash::Json::ParserError: Unrecognized token 'WITH': was expecting ('true', 'false' or 'null')

Log stash pipe line config :  
[http://dpaste.com/198PTBC](http://dpaste.com/198PTBC)

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 4, 2020, 6:01pm UTC](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076/2 "2020-03-04T18:01:50Z")

</div>

> [@Seetharaman\_K](#):
>
> :raw=\>"WITH CORRESPONDING LINE(S)", "HoldReason": "Invoice price exceeds purchase order price", ...

That is not valid JSON, so the json filter logs an error.

Your grok filter is not anchored, so if the JSON field before HoldReason contains something that matches LOGLEVEL grok may match it. I suggest you use a different field name and do not overwrite message, then look at the original [message] field to see what is going on.

---

<div class="post-metadata">

### Author: ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)
#### Post date: [March 5, 2020, 5:43am UTC](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076/3 "2020-03-05T05:43:14Z")

</div>

i really thank you for your reply. i was even thinking in those lines but my problem is how to over come this ? as i mentioned before below is my Gork filter ,

> filter {  
> grok {  
> match =\> { "message" =\> "%{LOGLEVEL:loglevel} %{GREEDYDATA:message}"}  
> overwrite =\> ["message"]  
> }
> 
> ```
> json{
> source => "message"
> }
> 
> if [message] != "" {
> mutate {
> add_tag => ["spam_log"]
> }
> }........
> 
> ```

and below is the value of the log field before the hold reason  
field name : category  
value : EMAIL BUYER WITH CORRESPONDING LINE(S)  
how come this gork filter is preventing this ?  
can i process the same logs again ? coz there is another important feild in the same log which i don't want to loose and because of this error these logs are not processed for months now!  
i want to process these omitted logs .  
i read your solution of giving a different field name , but if i change the field and lets assume it works , still i will loose the old logs !! more than everything what is problem with the field name "category" according to this gork ? is the field name of the value of the field

---

<div class="post-metadata">

### Author: ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)
#### Post date: [March 5, 2020, 6:03am UTC](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076/4 "2020-03-05T06:03:40Z")

</div>

> # Log Levels
> 
> LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)

above is the task of the log leve lgork match. may be because of the braces ? "()"

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 5, 2020, 2:09pm UTC](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076/5 "2020-03-05T14:09:23Z")

</div>

The ERR?(?:OR)? part of that regexp matches the ER in BUYER.

---

<div class="post-metadata">

### Author: ![Seetharaman\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seetharaman_k/32/52542_2.png) [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)
#### Post date: [March 14, 2020, 11:10am UTC](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076/6 "2020-03-14T11:10:21Z")

</div>

investigating the same through command line. changed the output to std out , now facing

> [@Log-stash not showing output in stdout](https://discuss.elastic.co/t/log-stash-not-showing-output-in-stdout/223610):
>
> [config](http://dpaste.com/1DNAAT2) -\> my config file logstash -f "C:\busapps\drsb\gbl0\logstash\7.0.0\bin\pipelines\logstash.conf" above is my command below is the output : [2020-03-14T10:43:12,670][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-03-14T10:43:12,701][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.0.0"} [2020-03-14T10:43:19,873][INFO][logstash.javapipeline] Starting pipeline {:…

**this problem !! ☹**

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 11, 2020, 11:10am UTC](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076/7 "2020-04-11T11:10:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
