# Trouble with Json filtering

**URL:** <https://discuss.elastic.co/t/trouble-with-json-filtering/117735>\
**Category:** Logstash\
**Created:** [January 31, 2018, 4:49am UTC](https://discuss.elastic.co/t/trouble-with-json-filtering/117735 "2018-01-31T04:49:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![gymmynzl](https://avatars.discourse-cdn.com/v4/letter/g/ccd318/32.png) [@gymmynzl](https://discuss.elastic.co/u/gymmynzl)\
**Post date:** [January 31, 2018, 4:49am UTC](https://discuss.elastic.co/t/trouble-with-json-filtering/117735/1 "2018-01-31T04:49:44Z")

</div>

Hi,

Im having some issues with trying to get some json into ES.

Im using filebeat to pass the json data over tcp to logstash ... this is a sample of the JSON, the JSON format is correct its just not feeding in the data, i can see what its doing in with the output, but what i see in the debug output isnt what going into ES.

* * *

{"classification.taxonomy": "abusive content", "raw": "MS4zMi4xMjguMC8xOCA7IFNCTDI4NjI3NQ==", "feed.accuracy": 100.0, "classification.type": "spam", "feed.provider": "Spamhaus", "feed.url": "[https://www.spamhaus.org/](https://www.spamhaus.org/)", "[feed.name](http://feed.name)": "Spamhaus Drop", "time.source": "2017-12-29T19:49:07+00:00", "time.observation": "2018-01-10T04:05:38+00:00", "extra": "{"blocklist": "SBL286275"}", "source.network": "1.32.128.0/18"}  
{"classification.taxonomy": "abusive content", "raw": "NS44LjM3LjAvMjQgOyBTQkwyODQwNzg=", "feed.accuracy": 100.0, "classification.type": "spam", "feed.provider": "Spamhaus", "feed.url": "[https://www.spamhaus.org/](https://www.spamhaus.org/)", "[feed.name](http://feed.name)": "Spamhaus Drop", "time.source": "2017-12-29T19:49:07+00:00", "time.observation": "2018-01-10T04:05:38+00:00", "extra": "{"blocklist": "SBL284078"}", "source.network": "5.8.37.0/24"}

* * *

input {  
beats {  
port =\> 9515  
codec =\> json  
type =\> data  
}  
}  
filter {  
if [type] == "data" {  
kv{  
}  
}  
}  
output {  
if [type] == "data" {  
stdout { codec =\> rubydebug }  
elasticsearch { hosts =\> ["127.0.0.1:9200"]  
index =\> "idata-%{+[YYYY.MM](http://YYYY.MM)}"  
}  
}  
}

* * *

Debug output

* * *

{  
"feed.url" =\> "[https://www.spamhaus.org/](https://www.spamhaus.org/)",  
"feed.provider" =\> "Spamhaus",  
"offset" =\> 7238348,  
"time.observation" =\> "2018-01-27T01:24:43+00:00",  
"input\_type" =\> "log",  
"raw" =\> "MjIzLjE3My4wLjAvMTYgOyBTQkwyMDQ5NTQ=",  
"[feed.name](http://feed.name)" =\> "Spamhaus Drop",  
"source" =\> "/opt/file-output/spamhous.txt",  
"source.network" =\> "223.173.0.0/16",  
"type" =\> "data",  
"tags" =\> [  
[0] "beats\_input\_codec\_json\_applied"  
],  
"@timestamp" =\> 2018-01-31T04:40:40.344Z,  
"time.source" =\> "2018-01-25T14:32:35+00:00",  
"classification.type" =\> "spam",  
"extra" =\> "{"blocklist": "SBL204954"}",  
"@version" =\> "1",  
"beat" =\> {  
"name" =\> "blar",  
"hostname" =\> "blar",  
"version" =\> "5.4.1"  
},  
"host" =\> "blar",  
"classification.taxonomy" =\> "abusive content",  
"feed.accuracy" =\> 100.0  
}

* * *

## Kibana

## name type format searchable aggregatable excluded controls \_id string \_index string \_score number \_source \_source \_type string

So none of the fields have arrived, which means there is no TIME field either so i can only creat an index which have no time reliance.

In Cerebro the entire index is only 810b ☹ the file which its importing is over 100megs

any help would be appreciated.

Thanks.

* * *

## { "\_shards": { "total": 5, "successful": 5, "failed": 0 }, "\_all": { "primaries": { "docs": { "count": 0, "deleted": 0 }, "store": { "size": "810b", "size\_in\_bytes": 810, "throttle\_time": "0s", "throttle\_time\_in\_millis": 0

---

<div class="post-metadata">

**Author:** ![mnaveen\_m](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@mnaveen\_m](https://discuss.elastic.co/u/mnaveen_m)\
**Post date:** [January 31, 2018, 5:02am UTC](https://discuss.elastic.co/t/trouble-with-json-filtering/117735/2 "2018-01-31T05:02:32Z")

</div>

You try removing if condition in output, you will see arriving input to ES/kibana as below.  
output {  
elasticsearch {  
hosts =\> "127.0.0.1:9200"  
}  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![gymmynzl](https://avatars.discourse-cdn.com/v4/letter/g/ccd318/32.png) [@gymmynzl](https://discuss.elastic.co/u/gymmynzl)\
**Post date:** [February 3, 2018, 1:23am UTC](https://discuss.elastic.co/t/trouble-with-json-filtering/117735/3 "2018-02-03T01:23:50Z")

</div>

I have the If condition there so that i see only the output from that ingest, i have a number of other things being ingested . the output above is a sample of the rubydebug. I know this is working as when i delete the index it gets recreated, i can also feed other json samples in using that filter and these work fine, I have validated the Json for its structure and this too is fine, it just seems to be ignoring it ☹

but thanks for the suggesting Mnaveen\_m

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2018, 1:24am UTC](https://discuss.elastic.co/t/trouble-with-json-filtering/117735/4 "2018-03-03T01:24:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.