# JSON Issues

**URL:** <https://discuss.elastic.co/t/json-issues/39769>\
**Category:** Logstash\
**Created:** [January 21, 2016, 1:12pm UTC](https://discuss.elastic.co/t/json-issues/39769 "2016-01-21T13:12:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tgdesrochers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tgdesrochers/32/51322_2.png) [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)\
**Post date:** [January 21, 2016, 1:12pm UTC](https://discuss.elastic.co/t/json-issues/39769/1 "2016-01-21T13:12:19Z")

</div>

I am using the JSON plugin to parse bro logs I have being printed in JSON format. My current setup is:

1. distributed sensor collecting data recording it in logs in JSON format.
2. logstash picks up the logs with the following input:

```
input {
  file {
    path => "/nsm/bro/logs/current/*.log"
    exclude => [
               "/nsm/bro/logs/current/stderr.log",
               "/nsm/bro/logs/current/stdout.log",
               "/nsm/bro/logs/current/communication.log",
               "/nsm/bro/logs/current/loaded_scripts.log"
               ]
    codec => "json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => "BRO"
  }
}
```

Filter stuff

```
output {
  if [type] == "BRO" {
    redis {
      host => "X.X.X.X"
      data_type => "list"
      key => "bro"
    }
  }
}
```

1. events get sent to central Redis Queue and forwarded with the following:

```
input {
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro"
    add_field => { "[@metadata][stage]" => "bro_redis" }
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro"
    add_field => { "[@metadata][stage]" => "bro_redis" }
  }
}

output {
  if [@metadata][stage] == "bro_redis" {
    redis {
      host => ["X.X.X.X"]
      data_type => "list"
      key => "bro_redis"
    }
  }
}
```

1. Data is read from redis with the following on a number of Logstash Indexer nodes and output to the ES cluster:

```
input {
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
  redis {
    host => ["X.X.X.X"]
    data_type => "list"
    key => "bro_redis"
  }
}

Custom filtering to add fields and some GEOIP stuff

output {
  if [type] == "BRO" {
    if [sensor] == "SENSOR1" {
      elasticsearch {
        hosts => [list of elasticsearch nodes]
        manage_template => false
        index => "sensor1-bro-%{+YYYY.MM.dd}"
      }
    }
    if [sensor] == "SENSOR2" {
      elasticsearch {
        hosts => [list of elasticsearch nodes]
        manage_template => false
        index => "sensor2-bro-%{+YYYY.MM.dd}"
      }
    }
  }
}
```

I am getting some strange errors in the /var/log/logstash/logstash.log file.

1. :message=\>"IP Field contained invalid IP address or hostname - I get this when the IP address has a "-" in the field.

2. :message=\>"Trouble parsing csv

:exception=\>#\<CSV::MalformedCSVError: Illegal quoting in line 1.\>, :level=\>:warnq

I get this with a lot of records. My problem is the data is not a CSV as indicated by the error. As you can see from the conf files above the data is in JSON. Why would logstash think its CSV. On my indexer nodes (Step 4 above) do I need to tell redis that the data is in JSON format. I thought according to the DOCS that the redis plugin reads json by default?

I know this is a very long post but any help would be greatly appreciated.

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 21, 2016, 6:45pm UTC](https://discuss.elastic.co/t/json-issues/39769/2 "2016-01-21T18:45:05Z")

</div>

What filters are you using?

As for the "IP Field contained invalid IP address or hostname" message, perhaps you should delete IP address fields that don't actually contain IP addresses? You'll probably want to map such fields as the "ip" type in Elasticsearch, and such fields only accept IPv4 addresses.

---

<div class="post-metadata">

**Author:** ![tgdesrochers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tgdesrochers/32/51322_2.png) [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)\
**Post date:** [January 25, 2016, 4:08pm UTC](https://discuss.elastic.co/t/json-issues/39769/3 "2016-01-25T16:08:52Z")

</div>

Solved. It appeared I was ingesting a log that wasn't in JSON format to begin with.

Thanks for the help

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:14am UTC](https://discuss.elastic.co/t/json-issues/39769/4 "2017-07-06T05:14:25Z")

</div>