# Logstash json filter not sending to root of event
**URL:** https://discuss.elastic.co/t/logstash-json-filter-not-sending-to-root-of-event/223332
**Category:** Logstash
**Created:** [March 12, 2020, 1:01pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-sending-to-root-of-event/223332 "2020-03-12T13:01:58Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Axl](https://discuss.elastic.co/u/Axl)
#### Post date: [March 12, 2020, 1:01pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-sending-to-root-of-event/223332/1 "2020-03-12T13:01:58Z")
I see that my event has a field "message", and that field is a JSON object, with several values, e.g `message: {"eventid":"cowrie.login.failed","username":"uno85",...}`.
I would like all subvalues of `message` to be put at the root of the event (i.e at the same level as @timestamp for example).
According to the [Logstash doc](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html), this should happen by default when I use a JSON filter and do not specify the `target` field. However, this is my configuration (below), and it's obviously not working because `username` is within message, not at top level...
```
filter {
if [type] == "cowrie" {
json {
source => message
}
```
---
### Author:  [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)
#### Post date: [March 12, 2020, 3:11pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-sending-to-root-of-event/223332/2 "2020-03-12T15:11:19Z")
You should use a ruby filter to extract your nested fields and put them at root level. Something like:
```
ruby {
code => "
event.get('message').each { |k, v| event.set(k,v) }
event.remove('message')
"
}
```
Anyway, if you post here the output of the pipeline as it is now (with only your filter applied), it'd be easier to help you.
---
### Author:  [@Axl](https://discuss.elastic.co/u/Axl)
#### Post date: [March 12, 2020, 4:03pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-sending-to-root-of-event/223332/3 "2020-03-12T16:03:03Z")
Ok, let me detail better because I'm not sure that `ruby` line will do the trick. This is what I get. Notice the **message** field? Inside, it has **eventId** , **username** etc. I'd like eventId, username to be at the same level as `_index`.
```auto
{
"_index": "filebeat-7.6.1-2020.03.10-000001",
"_type": "_doc",
"_id": "T_dsz3ABuJmadUjSadLZ",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2020-03-12T15:45:52.624Z",
"log": {
"offset": 7772652,
"file": {
"path": "/xxx/cowrie.json"
}
},
"message": "{\"eventid\":\"cowrie.login.success\",\"username\":\"test\",\"password\":\"xxxx\",\"message\":\"login attempt [test/xxxx] succeeded\",\"sensor\":\"instance-39\",\"timestamp\":\"2020-03-12T16:45:49.274837Z\",\"src_ip\":\"IP address\",\"session\":\"c176278d5630\"}",
"input": {
"type": "log"
},
"ecs": {
"version": "1.4.0"
},
"host": {
"name": "instance-39"
},
"agent": {
"type": "filebeat",
"ephemeral_id": "07441667-3997-4039-b3bc-e48209f22e69",
"hostname": "instance-39",
"id": "f505e637-ece6-4dc5-ad66-5f986536eae4",
"version": "7.6.1"
}
},
"fields": {
"suricata.eve.timestamp": [
"2020-03-12T15:45:52.624Z"
],
"@timestamp": [
"2020-03-12T15:45:52.624Z"
]
},
"sort": [
-9223372036854776000,
1584027952624
]
}
```
Now, there are several strange things. Mostly it's just as if my configuration file is not read. Though the `/var/log/logstash/logstash-plain.log` mentions the right file `"/etc/logstash/conf.d/logstash-cowrie.conf`
```auto
[2020-03-12T15:58:17,679][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/logstash-cowrie.conf"], :thread=>"#"}
```
Here is my `/etc/logstash/conf.d/logstash-cowrie.conf`. The commented lines are things that I tried.
```auto
input {
beats {
port => 5044
type => "cowrie"
}
}
filter {
# if [type] == "cowrie" {
json {
source => message
}
date {
match => ["timestamp", "ISO8601"]
}
if [src_ip] {
mutate {
add_field => { "src_host" => "%{src_ip}" }
}
dns {
reverse => ["src_host"]
nameserver => ["8.8.8.8", "8.8.4.4"]
action => "replace"
hit_cache_size => 4096
hit_cache_ttl => 900
failed_cache_size => 512
failed_cache_ttl => 900
}
geoip {
source => "src_ip"
target => "geoip"
database => "/opt/logstash/vendor/geoip/GeoLite2-City.mmdb"
#add_field => ["[geoip][location]", "%{[geoip][longitude]}" ]
#add_field => ["[geoip][location]", "%{[geoip][latitude]}" ]
}
}
mutate {
# convert => ["[geoip][coordinates]", "float" ]
remove_tag => ["beats_input_codec_plain_applied"]
remove_field => ["log.file.path", "log.offset"]
}
# }
}
output {
# if [type] == "cowrie" {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{type}"
}
file {
path => "/tmp/cowrie-logstash.json"
codec => json
}
stdout {
codec => rubydebug
}
# }
}
```
Several things are not normal:
- `message` is not at the right place
- I have a `log.file.path` though I ask to **remove** it
- I get **no** `src_host` field though it should be created as I have a `src_ip`
- I have **no** `/tmp/cowrie-logstash.json`
- should the line `index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"` be creating a new index each day? or only when I restart logstash? Anyway, I only have `filebeat-2020.03.10` and `filebeat-2020.03.09` (and we're March 12).
This is getting me mad! ☹
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 9, 2020, 4:03pm UTC](https://discuss.elastic.co/t/logstash-json-filter-not-sending-to-root-of-event/223332/4 "2020-04-09T16:03:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.