# Reading Data From Kafka and use filter json fails with ParserError

**URL:** https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179
**Category:** Logstash
**Created:** [May 16, 2018, 4:38pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179 "2018-05-16T16:38:12Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [May 16, 2018, 4:38pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/1 "2018-05-16T16:38:12Z")

</div>

Version: 6.2.3

Operating System: RHEL

Config File :

```auto
    input {
        kafka {
            bootstrap_servers => "127.0.0.1:9092"
            auto_offset_reset => earliest
            topics => ["test"]
            group_id => "test"
        }
    }

    filter {
      json {
        source => "message"
      }
    }

    filter {
        if ([fields][topic]=="test") {
            grok {
                match => ["message", "%{DATE:Date}\s+%{TIME:Time}\s+Caller=(%{DATA:Caller})?\s+Operator=(%{DATA:Operator})?\s+Message=(%{DATA:Message})?\s+Version=(%{DATA:Version})?\s+Service=(?<Service>(%{DATA}%{SPACE}%{DATA})|(%{DATA}))?\s+Server=(%{DATA:Server})?\s+KeyCnt=(%{NUMBER:KeyCnt:int})?\s+Key=(%{DATA:Key})?\s+(MsgId=%{DATA:MsgId}\s+)?ElapsedTime=(%{NUMBER:ElapsedTime:int})?"]
        }
      }
    }

```

Sample Data:  
`05/15/2018	13:21:28:464	Caller=WORK	Operator=work	Message=GetRelationship	Version=005 Service=RelationshipServer	Server=localhost	KeyCnt=1	Key=D1231231-3-2,	MsgId=6e3e3f84-3f49-496d-a287-3b277a598538	ElapsedTime=64`

Error:

```auto
[2018-05-16T00:44:29,119][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"05/15/2018\t13:21:28:464\tCaller=WORK\tOperator=work\tMessage=GetRelationship\tVersion=005\tService=RelationshipServer\tServer=localhost\tKeyCnt=1\tKey=D1231231-3-2,\tMsgId=6e3e3f84-3f49-496d-a287-3b277a598538\tElapsedTime=64", :exception=>#<LogStash::Json::ParserError: Invalid numeric value: Leading zeroes not allowed

```

Steps to Reproduce:

Filebeat reads the log files, and publishes to topic in kafka. Logstash is reading the kafka topic, and the filter is applied. During the filter, json throws an error.

When kafka is not used, and filebeat sends directly to logstash there is no failure.

Cannot go to the log/app owner and ask for correction.

How do we make logstash parse the data as string?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 16, 2018, 4:54pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/2 "2018-05-16T16:54:33Z")

</div>

> [@paano](#):
>
> 05/15/2018 13:21:28:464 Caller=WORK Operator=work Message=GetRelationship Version=005 Service=RelationshipServer Server=localhost KeyCnt=1 Key=D1231231-3-2, MsgId=6e3e3f84-3f49-496d-a287-3b277a598538 ElapsedTime=64

That is not valid JSON. I would handle that event with dissect and kv. The separators in dissect and the field\_split character are tabs.

```
  dissect { mapping => { "message" => "%{ts} %{+ts} %{restOfLine}" } }
  kv { source => "restOfLine" field_split => " " }

```

---

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [May 16, 2018, 8:25pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/3 "2018-05-16T20:25:52Z")

</div>

@Badger Yep, my Bad. The Data which i had pasted was RAW which filebeat is pushing. Kafka adds the json parser, and logstash fails on filter.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 16, 2018, 8:28pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/4 "2018-05-16T20:28:41Z")

</div>

OK, so with the kafka input and no filters, what does output { stdout { codec =\> rubydebug } } look like?

---

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [May 20, 2018, 11:04pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/5 "2018-05-20T23:04:27Z")

</div>

@Badger below is the raw data logstash is reading from `kafka`. Output from `rubydebug`

{  
"@timestamp" =\> 2018-05-19T22:56:24.065Z,  
"message" =\> "{"@timestamp":"2018-05-19T22:56:11.222Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.3","topic":"test"},"fields":{"logsource":"test","topic":"test"},"prospector":{"type":"log"},"beat":{"name":"elk","hostname":"elk","version":"6.2.3"},"source":"/var/log/dummy.log","offset":215,"message":"05/15/2018\t13:21:28:464\tCaller=WORK\tOperator=work\tMessage=GetRelationship\tVersion=005 Service=RelationshipServer\tServer=localhost\tKeyCnt=1\tKey=D1231231-3-2,\tMsgId=6e3e3f84-3f49-496d-a287-3b277a598538\tElapsedTime=64"}",  
"@version" =\> "1"  
}

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 21, 2018, 1:30pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/6 "2018-05-21T13:30:07Z")

</div>

A json filter turns that into

```auto
{
    "prospector" => {
        "type" => "log"
    },
        "source" => "/var/log/dummy.log",
    "@timestamp" => 2018-05-19T22:56:11.222Z,
        "fields" => {
            "topic" => "test",
        "logsource" => "test"
    },
       "message" => "05/15/2018\t13:21:28:464\tCaller=WORK\tOperator=work\tMessage=GetRelationship\tVersion=005 Service=RelationshipServer\tServer=localhost\tKeyCnt=1\tKey=D1231231-3-2,\tMsgId=6e3e3f84-3f49-496d-a287-3b277a598538\tElapsedTime=64",
        "offset" => 215,
          "beat" => {
            "name" => "elk",
         "version" => "6.2.3",
        "hostname" => "elk"
    }
}

```

I can reproduce the error using

```
  json { source => message }
  json { source => message }

```

i.e. over-writing the message field with a string that is not JSON, then trying to parse that with a json filter. What does rubydebug produce if you remove the json filter?

---

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [May 21, 2018, 8:13pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/7 "2018-05-21T20:13:08Z")

</div>

I do not have the filter when `rubydebug` was enabled. @Badger any specific setting i need to add to make the message a string and try json filer

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 22, 2018, 12:02pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/8 "2018-05-22T12:02:48Z")

</div>

I don't think you need a json filter. The fact that _message_ refers to "05/15/2018\t13:21:28:464\tCaller=WORK\tOperator=work\tMessage=GetRelationship\tVersion=005 Service=RelationshipServer\tServer=localhost\tKeyCnt=1\tKey=D1231231-3-2,\tMsgId=6e3e3f84-3f49-496d-a287-3b277a598538\tElapsedTime=64" indicates it is already being parsed as JSON.

---

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [May 23, 2018, 8:29pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/9 "2018-05-23T20:29:52Z")

</div>

@Badger i tried reading the JSON msg directly, but it failed as we have a grok pattern.  
Basically, Extracting the message out from Kafka Message using JSON filter, and then grok pattern the result.

What needs to be done to stop the JSONparse failure? any suggestion?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 23, 2018, 8:35pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/10 "2018-05-23T20:35:57Z")

</div>

> [@paano](#):
>
> What needs to be done to stop the JSONparse failure? any suggestion?

If you are talking about this

```
Error parsing json {:source=>"message", :raw=>"05/15/2018\t13:21:28:464\tCaller=WORK\tOperator=work\tMessage=GetRelationship\tVers[...]

```

The answer is to remove the json filter. The incoming JSON has already been parsed and does not need to be parsed a second time. If you are talking about some other error please show the error, the configuration that produces it, and the 'stdout { codec =\> rubydebug }' output.

---

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [June 4, 2018, 8:15pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/11 "2018-06-04T20:15:23Z")

</div>

@Badger I have been trying different options. And what i see is, if we remove json filter, when inserted into elastic, key-values do not show up. It is the complete json message itself.

`{"@timestamp":"2018-06-04T20:01:55.429Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.3","topic":"dummy"},"prospector":{"type":"log"},"fields":{"topic":"dummy","logsource":"dummy"},"beat":{"name":"elk","hostname":"elk","version":"6.2.3"},"message":"04/24/2018\t00:00:18:390\tCaller=WORK\tOperator=work\tMessage=GetClientPreference\tVersion=v7\tService=ClientPreferenceService\tServer=localhost\tKeyCnt=1\tKey=2/32115004\tMsgId=438cc172-f99b-4cf0-83a0-2ff48d0f44ce\tElapsedTime=28","source":"/var/log/dummy.log","offset":913}`

Any better way to get the json key-value into elastic.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 4, 2018, 8:52pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/12 "2018-06-04T20:52:19Z")

</div>

What does 'stdout { codec =\> rubydebug }' show?

---

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [June 6, 2018, 3:16am UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/13 "2018-06-06T03:16:48Z")

</div>

@Badger adding the json filter as  
source =\> "[message]" was part of the solution. DO not know by adding the Brackets helped in resolving.

Also noticed, if we have multiple conf file with json filter, and if any one of the conf file would fail parsing , rest would fail.

Example:  
app1.conf which has a JSON filter  
app2.conf which has GROK and JSON.

If app2.conf fails on the filter, it would throw jsonparsefailure.

How do we resolve Filter sequence? And how does logstash read multiple ".conf" files?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 6, 2018, 12:17pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/14 "2018-06-06T12:17:39Z")

</div>

> [@paano](#):
>
> How do we resolve Filter sequence? And how does logstash read multiple ".conf" files?

This is commonly misunderstood. Unless you are using pipelines.yml to point individual pipelines at different configuration files, logstash concatenates all the configuration files. If you point -f at a directory that contains app1.conf, app2. conf and app3. conf, it will read events from all the inputs, put them through every filter in every file (with the files ordered in dictionary sort order) then send the events to every output.

So if app1.conf has 'json { source =\> message }' it will overwrite the message field with the contents of "message": "05/15/2018\t13:21:28:464\tCaller=WORK...". Then if app2.conf also has 'json { source =\> message }' it will fail to parse that, since it is not valid JSON. You do not need the json filter in both files, unless you are using conditionals to make sure the filters are only applied to specific messages.

---

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [June 6, 2018, 2:33pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/15 "2018-06-06T14:33:31Z")

</div>

@Badger is there a way to have logstash read pipelines in sequence. Like to name the files as 001app1.conf 002app2.conf ...

Any documentation/best practices we need to follow.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 6, 2018, 2:59pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/16 "2018-06-06T14:59:46Z")

</div>

Yes, you can number the files to make the order clearer. It will still read from all the inputs, apply all the filters, and write the same events to every output unless you use conditionals.

---

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [June 6, 2018, 6:25pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/17 "2018-06-06T18:25:54Z")

</div>

Thanks. This helped.  
ANy documentation on how to name logstash conf files.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 6, 2018, 6:29pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/18 "2018-06-06T18:29:36Z")

</div>

I am not aware of any best practices documented. We all have our own way doing things.

---

<div class="post-metadata">

### Author: ![paano](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@paano](https://discuss.elastic.co/u/paano)
#### Post date: [June 7, 2018, 3:13pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/19 "2018-06-07T15:13:57Z")

</div>

Thanks @Badger. Will close this issue.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2018, 3:13pm UTC](https://discuss.elastic.co/t/reading-data-from-kafka-and-use-filter-json-fails-with-parsererror/132179/20 "2018-07-05T15:13:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.