# Json data from Filebeat to Logstash

**URL:** <https://discuss.elastic.co/t/json-data-from-filebeat-to-logstash/322672>\
**Category:** Logstash\
**Created:** [January 7, 2023, 11:00pm UTC](https://discuss.elastic.co/t/json-data-from-filebeat-to-logstash/322672 "2023-01-07T23:00:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hamburglar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamburglar/32/78037_2.png) [@Hamburglar](https://discuss.elastic.co/u/Hamburglar)\
**Post date:** [January 7, 2023, 11:00pm UTC](https://discuss.elastic.co/t/json-data-from-filebeat-to-logstash/322672/1 "2023-01-07T23:00:05Z")

</div>

**Json data to be ingested:** /var/log/file.json

```auto
{"event_type":"temp","time":"2023-01-07 23:30:12","temp":64.0,"fan":2291}
{"event_type":"temp","time":"2023-01-07 23:30:22","temp":63.0,"fan":2308}
{"event_type":"temp","time":"2023-01-07 23:30:32","temp":63.0,"fan":2295}
{"event_type":"temp","time":"2023-01-07 23:30:42","temp":67.0,"fan":2299}
{"event_type":"temp","time":"2023-01-07 23:30:52","temp":61.0,"fan":2291}

```

**logstash config:**

```auto
input {
  beats {
        port => 5044
        }
}

filter {
        json {
        #skip_on_invalid_json => true
        source => "message"
        target => "json"
        }
}
output {
       elasticsearch {
                hosts => "<private_ip>:9200"
                user => "<redacted>"
                password => "<redacted>"
        }  
        stdout { }
}

```

Filebeat config:

```auto
filebeat.inputs:
- type: log
  enabled: true
    - /var/log/file.json
    
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true

output.logstash:
  hosts: ["localhost:5044"]

```

Errors while using filebeat:

```auto
[WARN][logstash.filters.json][main][20c071f94122e35758b5f80f63972b111cecff39a62dbe8a583f702663a206fa] Error parsing json {:source=>\\\"message\\\", :raw=>\\\"[2023-01-07T13:36:45,326] 

Sending a new message for the listener, sequence: 540\\\", :exception=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): was expecting comma to separate Array entries\\n at [Source: (byte[])\\\"[2023-01-07T13:36:45,326]

```

If i **dont** use filebeat, and just use the following logstash config instead it works perfect:

```auto
input {
        file {
                start_position=>"beginning"
                path => "/var/log/file.json"
                sincedb_path => "/dev/null"
        }
}

filter {
        json {
        source => "message"
        target => "json"
        }
}

output {
       elasticsearch {
                hosts => "<private_ip>:9200"
                user => "<redacted>"
                password => "<redacted>"
        }
        stdout {}
}

```

Why is Filebeat producing these Json issues?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 7, 2023, 11:16pm UTC](https://discuss.elastic.co/t/json-data-from-filebeat-to-logstash/322672/2 "2023-01-07T23:16:29Z")

</div>

Hi @Hamburglar Welcome to the community!

What version are you on.. always tell us 🙂

> [@Hamburglar](#):
>
> Why is Filebeat producing these Json issues?

Leave filebeat in ...Take out the JSON filter in logstash and see what the log lines / messages look like coming out of logstash and share.

My suspicion is that you are trying to decode the JSON twice...

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 8, 2023, 11:51am UTC](https://discuss.elastic.co/t/json-data-from-filebeat-to-logstash/322672/3 "2023-01-08T11:51:05Z")

</div>

> [@Hamburglar](#):
>
> `[WARN][logstash.filters.json]`

Your error is in Logstash, not filebeat.

> [@Hamburglar](#):
>
> `Error parsing json {:source=>\\\"message\\\", :raw=>\\\"[2023-01-07T13:36:45,326]`

Are you sure your message is a JSON? From this line it says that your message is not a json, that's why you got that warning log line.

---

<div class="post-metadata">

**Author:** ![Hamburglar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamburglar/32/78037_2.png) [@Hamburglar](https://discuss.elastic.co/u/Hamburglar)\
**Post date:** [January 8, 2023, 1:51pm UTC](https://discuss.elastic.co/t/json-data-from-filebeat-to-logstash/322672/4 "2023-01-08T13:51:04Z")

</div>

Hi

Running versions:

```auto
filebeat 7.15.0
logstash 8.5.3

```

You were spot on, removing the json filter in the logstash config fixed the issue. I forgot that filebeat actually decodes the JSON, so it was decoding twice. Thanks for the warm welcome and help!

Working logstash config:

```auto
input {
  beats {
        port => 5044
        }
}

output {
       elasticsearch {
                hosts => "<private_ip>:9200"
                user => "<redacted>"
                password => "<redacted>"
        }  
        stdout { }
}

```

---

<div class="post-metadata">

**Author:** ![Hamburglar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamburglar/32/78037_2.png) [@Hamburglar](https://discuss.elastic.co/u/Hamburglar)\
**Post date:** [January 8, 2023, 1:54pm UTC](https://discuss.elastic.co/t/json-data-from-filebeat-to-logstash/322672/5 "2023-01-08T13:54:35Z")

</div>

Hi

The initial JSON was valid (line by line) as pasted in the first post, but as the JSON was getting decoded twice logstash was having issues accepting the data.

Thanks for the help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2023, 1:54pm UTC](https://discuss.elastic.co/t/json-data-from-filebeat-to-logstash/322672/6 "2023-02-05T13:54:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.