# Parsed JSON object/hash requires a target configuration option

**URL:** <https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354>\
**Category:** Logstash\
**Created:** [August 30, 2016, 6:57pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354 "2016-08-30T18:57:29Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [August 30, 2016, 6:57pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/1 "2016-08-30T18:57:29Z")

</div>

I'm seeing these messages in my logstash.log file:

```
{:timestamp=>"2016-08-30T11:36:51.811000-0700", :message=>"Parsed JSON object/hash requires a target configuration option", :source=>"message", :raw=>"", :level=>:warn}
{:timestamp=>"2016-08-30T11:36:57.453000-0700", :message=>"Parsed JSON object/hash requires a target configuration option", :source=>"message", :raw=>"", :level=>:warn}

```

According to [https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) the `target` setting is not required.

So, why am I getting messages saying that the `target` setting IS required?

I believe these messages are triggered by log lines like:

```
{"key":"ldap:\/\/1.1.1.1", "foo": "bar"}

```

jsonlint says:

```auto
    jsonlint-py -vf jsontest.json 
    jsontest.json:1:20: Warning: String escape code is not allowed in strict JSON: u'\\/'
    | At line 1, column 20, offset 20
    | String started at line 1, column 14, offset 14
     jsontest.json:1:22: Warning: String escape code is not allowed in strict JSON: u'\\/'
    | At line 1, column 22, offset 22
    | String started at line 1, column 14, offset 14

```

So, is logstash erroring on syntax, but logging the wrong message?

logstash is version 2.3.4 on Ubuntu 14.04.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 30, 2016, 7:12pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/2 "2016-08-30T19:12:02Z")

</div>

Here's the relevant piece of code:

> <https://github.com/logstash-plugins/logstash-filter-json/blob/v2.0.6/lib/logstash/filters/json.rb#L80-L87>

Judging by the `:raw=>""` in the log message I'd say you're trying to parse an empty string.

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [August 30, 2016, 7:35pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/3 "2016-08-30T19:35:49Z")

</div>

As in something like:

```
{"blah":""}

```

?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 30, 2016, 7:39pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/4 "2016-08-30T19:39:29Z")

</div>

No, an empty string. I think your `message` field was empty.

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [August 30, 2016, 8:00pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/5 "2016-08-30T20:00:08Z")

</div>

As in Filebeat is sending an empty line?

What if just the source field was empty? As in, I groked a "jsonfieldfoo" field out of a line that is not just json. But that field doesn't always exist. Would grok create an empty "jsonfieldfootarget" in that case?

For example, for [gitlab production logs](http://docs.gitlab.com/ce/administration/logs.html#production.log), I do this:

```auto
filebeat prospector:
    - paths:
        - /var/log/gitlab/gitlab-rails/production.log
      encoding: plain
      document_type: gitlab_production
      multiline:
        pattern: ^Started
        negate: true
        match: after

logstash filter:

grok {
           match => ["message", "Started\s%{WORD:verb}\s\"%{NOTSPACE:request}\"\sfor\s%{IPORHOST:clientip}\s(at)\s.*(?<timestamp>%{TIMESTAMP_ISO8601}\s[-+]{1}\d{4})((?<main_message>.*)Parameters:\s(?<project_parameters>\{.*\})(?<main_message_two>.*\n)*(Completed?\s?%{NUMBER:response}(?<response_message>.*))?)?"]
           add_field => {"combinestuff" => "blah" }
         }
         mutate {
           gsub => ["project_parameters", "=>",":"]
         }
         {
         json {
           source => "project_parameters"
           target => "project_parameters"
         }

```

`project_parameters` does not always exist. So would the json filter throw that error if that happened?

I did try wrapping the json filter in `if [project_parameters]{ }`, but did not stop the error messages from showing up.

Also, just to double check, when Filebeat sends a line to Logstash using the plain encoding, the actual log line is in the `message` field, correct?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 30, 2016, 8:38pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/6 "2016-08-30T20:38:50Z")

</div>

> As in Filebeat is sending an empty line?

I don't know how your `message` field is populated so I can't answer, but I suppose that's a possibility.

> What if just the source field was empty?

That's exactly what I'm talking about. According to the log message the `message` field is your source field. This is however not consistent with the configuration you posted.

> project\_parameters does not always exist. So would the json filter throw that error if that happened?

No, only if the field is empty.

> Also, just to double check, when Filebeat sends a line to Logstash using the plain encoding, the actual log line is in the message field, correct?

Yes.

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [August 30, 2016, 9:03pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/7 "2016-08-30T21:03:58Z")

</div>

> [@magnusbaeck](#):
>
> That's exactly what I'm talking about. According to the log message the message field is your source field. This is however not consistent with the configuration you posted.

The only other json calls that use `message` should never have an empty line. They are a `{ "foo": "bar", ... , "bar": "foo" }` object per line. So the filter looks like:

```auto
json { source => "message" }

```

Well, I was trying to avoid posting a bunch of code like this, but here is the full config for the only json logs I have:

```auto
if [source] == "/var/log/apache2/access_json.log" {
          json {
            source => "message"
          }
          mutate {
            gsub => [
              "forwardedip","127.0.0.1","",
              "forwardedip",",",""
            ]
            strip => ["forwardedip"]
          }
          mutate { convert => { "bytes" => "integer" } }
          if [forwardedip] !~ /(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/ {
            if (([clientip] == "-") or ([clientip] == [proxyip])) and ([forwardedip] != "-") {
              mutate {
                replace => { "clientip" => "%{[forwardedip]}" }
              }
            }
          }
          if [proxyip] == [clientip] { mutate { replace => { "proxyip" => "-" } } }
          if [clientip] !~ /(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/ {
            geoip {
              source => "clientip"
            }
          }
        }

```

My apache log format:

```auto
LogFormat "{\"timestamp\": \"%t\",\"forwardedip\": \"%{X-Forwarded-For}i\",\"clientip\": \"%a\",\"proxyip\": \"%{c}a\",\"serverip\": \"%A\",\"servername\": \"%v\",\"request_log_id\": \"%L\",\"port\": \"%p\",\"ident\": \"%l\",\"auth\": \"%u\",\"verb\": \"%m\",\"request\": \"%U\",\"query\": \"%q\",\"httpversion\": \"%H\",\"response\": \"%>s\",\"bytes\": \"%b\",\"referrer\": \"%{Referer}i\",\"agent\": \"%{User-Agent}i\"}" access_json

```

```auto
if [type] == "ldapwrangler" {
          json {
            source => "message"
          }
          mutate {
            convert => { "timestamp" => "integer" }
          }
          date {
            match => ["timestamp", "UNIX"]
            target => "@timestamp"
          }
        }

```

ldapwrangler logs are a php array encoded to json and written to a text file.

```auto

if [type] == "gitlab_production" {
         grok {
           match => ["message", "Started\s%{WORD:verb}\s\"%{NOTSPACE:request}\"\sfor\s%{IPORHOST:clientip}\s(at)\s.*(?<timestamp>%{TIMESTAMP_ISO8601}\s[-+]{1}\d{4})((?<main_message>.*)Parameters:\s(?<project_parameters>\{.*\})(?<main_message_two>.*\n)*(Completed?\s?%{NUMBER:response}(?<response_message>.*))?)?"]
           add_field => {"combinestuff" => "blah" }
         }
         mutate {
           gsub => ["project_parameters", "=>",":"]
         }
         if [project_parameters]{
         json {
           source => "project_parameters"
           target => "project_parameters"
         }}
         if [main_message_two]{
           mutate {
             update => {"combinestuff" => "%{main_message} (project_parameters) %{main_message_two}"}
             remove_field => ["main_message_two", "main_message"]
           }
         }
         if [response_message] and [response] {
           mutate {
             update => {"combinestuff" => "%{combinestuff} Response %{response} %{response_message}"}
             remove_field => ["response_message"]
           }
         }
         if [combinestuff] != "blah"{
           mutate {
             update => {"message" => "%{combinestuff}"}
             remove_field => ["combinestuff"]
           }
         }
         if [combinestuff] == "blah"{
           mutate {
             remove_field => ["combinestuff"]
           }
         }
         date {
           match => ["timestamp", "YYYY-MM-dd HH:mm:ss Z"]
           target => "@timestamp"
         }
      }

```

See my previous post about the gitlab production logs.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 30, 2016, 9:11pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/8 "2016-08-30T21:11:24Z")

</div>

i see nothing obviously wrong with your filters, but it all comes down to the inputs. Perhaps adding a `stdout { codec => rubydebug }` output for events with the `_jsonparsefailure` tag would be helpful?

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [August 30, 2016, 11:45pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/9 "2016-08-30T23:45:46Z")

</div>

Well, this is fun. I added:

```auto
      if "_jsonparsefailure" in [tags] {
        file {
          path => "/var/log/logstash/jsonparsefailure.debug.log"
          codec => "rubydebug"
        }
      }

```

to my output. That let me know that the ldapwrangler logs are the issue.

The fun part is that I'm tailing both ldapwrangler files while also tailing my debug output. The debug file keeps getting entries added to it, but the actual log files are not adding new logs at the same time.

So, either, filebeat is reading old entries very slowly, or it's just gone crazy....

Ah well, almost time to go home. I'll work on it more tomorrow.

Oh, a sub field in the ldapwrangler logs is called "message". That shouldn't cause any issues, right? It'd just be considered message.message by logstash?

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [August 31, 2016, 8:57pm UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/10 "2016-08-31T20:57:56Z")

</div>

I modified the ldapwrangler logs to use "msg" instead of "message". That has significatnely reduced the number json errors I see.

Is there a way to make the debug codec echo the actual log line? All I'm getting is:

```auto
{
       "message" => "",
      "@version" => "1",
    "@timestamp" => "2016-08-31T20:50:11.104Z",
    "input_type" => "log",
         "count" => 1,
          "beat" => {
        "name" => "node01"
    },
        "fields" => nil,
          "tags" => [
        [0] "filebeat",
        [1] "beats_input_codec_plain_applied",
        [2] "_jsonparsefailure"
    ],
        "source" => "/var/www/ldapwrangler/application/logs/cronUserSync.log",
        "offset" => 35208123,
          "type" => "ldapwrangler",
          "host" => "node01"
}
{
       "message" => "",
      "@version" => "1",
    "@timestamp" => "2016-08-31T20:50:11.108Z",
          "type" => "ldapwrangler",
    "input_type" => "log",
         "count" => 1,
        "fields" => nil,
          "tags" => [
        [0] "filebeat",
        [1] "beats_input_codec_plain_applied",
        [2] "_jsonparsefailure"
    ],
        "offset" => 35385677,
          "beat" => {
        "name" => "node01"
    },
        "source" => "/var/www/ldapwrangler/application/logs/cronUserSync.log",
          "host" => "node01"
}

```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 1, 2016, 5:40am UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/11 "2016-09-01T05:40:51Z")

</div>

Logstash doesn't maintain the original input separately. If you want to keep it you just have to avoid overwriting it. Or, introduce a clone filter as the very first filter to make a copy of each event and log _that_ somewhere (and make sure you have conditionals in place so that the clones aren't processed by the other filters and outputs).

---

<div class="post-metadata">

**Author:** ![crags](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crags/32/15112_2.png) [@crags](https://discuss.elastic.co/u/crags)\
**Post date:** [March 2, 2017, 10:46am UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/12 "2017-03-02T10:46:00Z")

</div>

Hi. Appreciate that this is an old thread, but I'm having the same issue. I've followed the same steps above to log to a file, but all i'm getting is the following in the log file:

```json
{
    "@timestamp" => 2017-03-02T10:44:02.306Z,
      "@version" => "1",
          "tags" => [
        [0] "_jsonparsefailure"
    ]
}

```

Is there any further way I can find out what the original message is / was and why it's not parsing. We are reading data in from Kafka as opposed to filebeats

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:28am UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/59354/13 "2017-07-06T04:28:09Z")

</div>