# Logstash can't process all JSON logs

**URL:** <https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003>\
**Category:** Logstash\
**Created:** [July 13, 2018, 7:15pm UTC](https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003 "2018-07-13T19:15:38Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [July 13, 2018, 7:15pm UTC](https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003/1 "2018-07-13T19:15:38Z")

</div>

Hi,

We are sending logs directly from django based python application to logstash over TCP/9111 port. We found that some logs are missing like

> {"@timestamp": "2018-07-13T14:38:23.741Z", "remote\_addr": "10.0.2.2", "host": "ubuntu-xenial", "message": "Successful multi-factor authentication step", "path": "/home/ubuntu/themis\_airflow/profiles/views.py", "@version": "1", "stack\_info": null, "logger\_name": "profiles.views", "auth": {"user": 152390415851211, "step": 1, "mfa": "REQ"}, "type": "flow-django-local", "tags": ["mfa-success"], "level": "INFO"}
> 
> {"@timestamp": "2018-07-13T14:39:28.506Z", "remote\_addr": "10.0.2.2", "host": "ubuntu-xenial", "message": "Successful multi-factor authentication step", "path": "/home/ubuntu/themis\_airflow/profiles/views.py", "@version": "1", "stack\_info": null, "logger\_name": "profiles.views", "auth": {"user": 152390415851211, "step": 1, "mfa": "REQ"}, "type": "flow-django-local", "tags": ["mfa-success"], "level": "INFO"}

We found the first log in Kibana but the second one didn't appeared.

Here are out logstash configuration:

```
input{
  tcp {
    host => "0.0.0.0"
    port => 9111
    codec => "json"
    tags => ["django"]
  }
}

filter {
  if "django" in [tags] {
      json {
        source => "message"
      }
  }
}

output {

  if "django" in [tags] {
    amazon_es {
      hosts => ["<AWS-ES>"]
      region => "<region>"
      index => "django-%{+YYYY.MM.dd}"
    }
  }

  else {
    null {}
  }

}

output {
  if "django" in [tags] {
    s3{
      region => "<region>"
      bucket => "dotdash-qa-application-logs"
      prefix => "app/django/%{+YYYY.MM.dd}"
      size_file => 2146304
      time_file => 5
      codec => json
      canned_acl => "private"
    }
  }
  else {
    null {}
  }
}

```

We found the following warning log in logstash

> [2018-07-13T15:07:04,097][WARN][logstash.filters.json] Error parsing json {:source=\>"message", :raw=\>"Login failed.", :exception=\>#\<LogStash::Json::ParserError: Unrecognized token 'Login': was expecting ('true', 'false' or 'null')  
> at [Source: (byte)"Login failed."; line: 1, column: 7]\>}

Any help would be appreciated.

Ferdous Shibly

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 13, 2018, 7:20pm UTC](https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003/2 "2018-07-13T19:20:14Z")

</div>

If you are using a json codec on the input the "message" field will be something like "Successful multi-factor authentication step" and that is not valid JSON. You do not need a json filter if you use a codec.

---

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [July 13, 2018, 7:38pm UTC](https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003/3 "2018-07-13T19:38:00Z")

</div>

After removing filter pipeline, we found some tags are missing. We got only tags which are being added in the input chain. And still now some logs are missing.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 13, 2018, 8:00pm UTC](https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003/4 "2018-07-13T20:00:52Z")

</div>

If that is your complete configuration it is hard to see how that could be happening. But I would suggest replacing those null outputs with something like file to see if there are events going through those branches.

---

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [July 16, 2018, 3:18am UTC](https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003/5 "2018-07-16T03:18:55Z")

</div>

Only message I found is

> [2018-07-15T23:16:31,025][WARN][logstash.filters.json] Error parsing json {:source=\>"message", :raw=\>"Login failed.", :exception=\>#\<LogStash::Json::ParserError: Unrecognized token 'Login': was expecting ('true', 'false' or 'null')  
> at [Source: (byte)"Login failed."; line: 1, column: 7]\>}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 13, 2018, 3:19am UTC](https://discuss.elastic.co/t/logstash-cant-process-all-json-logs/140003/6 "2018-08-13T03:19:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.