# Logstash не удаляет вложенные поля из json

**URL:** <https://discuss.elastic.co/t/logstash-json/275113>\
**Category:** Вопросы на русском языке\
**Created:** [June 7, 2021, 7:37am UTC](https://discuss.elastic.co/t/logstash-json/275113 "2021-06-07T07:37:34Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [June 7, 2021, 7:37am UTC](https://discuss.elastic.co/t/logstash-json/275113/1 "2021-06-07T07:37:34Z")

</div>

Привет. Возникла вопрос по работе logstash.  
Хочу сохранить в ES ответ nginx. Ответ представляет из себя json.  
Натравливаю на него json плагин и получаю структуру полей в ES.

```auto
json {
      source => "response_body"
      target => "response_fields"
      remove_field => ["response_body"]
    }

```

Из этой структуры хочется удалить некоторые ненужные вложенные поля, и тут возникает проблема. Они не хотят удаляться, ни через плагин json, ни через mutate.

```auto
json {
      source => "response_body"
      target => "response_fields"
      remove_field => ["response_body","[response_fields][contents][agg_content_gid]" ]
    }

```

Что может быть не так?

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [June 7, 2021, 6:23pm UTC](https://discuss.elastic.co/t/logstash-json/275113/2 "2021-06-07T18:23:08Z")

</div>

А у вас этот объект json в каком поле находиться? Он не в `message`? Пытался воспроизвести вашу ситуацию. Не получается - у меня все работает, как ожидается

test.txt:

```auto
{"foo":"bar", "contents": {"agg_content_gid": "killme", "agg": "keep_me"}}

```

config/test.conf:

```auto
input {
  stdin {
    id => "my_plugin_id"
  }
}

filter {
  json {
    source => "message"
    target => "response_fields"
    remove_field => ["message","[response_fields][contents][agg_content_gid]" ]
  }
}

output {
  stdout {
    codec => json
  }
}

```

Запускаю:

```auto
bin/logstash -f config/test.conf < test.txt

```

Результат:

```auto
{"response_fields":{"contents":{"agg":"keep_me"},"foo":"bar"},"@version":"1","host":"bee","@timestamp":"2021-06-07T18:18:32.646Z"}

```

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [June 7, 2021, 6:35pm UTC](https://discuss.elastic.co/t/logstash-json/275113/3 "2021-06-07T18:35:06Z")

</div>

response\_body появляется в результате парсинга гроком. И существует как отдельное поле.

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [June 7, 2021, 6:47pm UTC](https://discuss.elastic.co/t/logstash-json/275113/4 "2021-06-07T18:47:19Z")

</div>

Все работает. Попробуйте сами

test.txt:

```auto
here:{"foo":"bar", "contents": {"agg_content_gid": "killme", "agg": "keep_me"}}

```

config:

```auto
input {
  stdin {
    id => "my_plugin_id"
  }
}

filter {
  grok {
   match => { "message" => "here:(?<response_body>.*)"}
   remove_field => "message"
  }
  json {
    source => "response_body"
    target => "response_fields"
    remove_field => ["response_body","[response_fields][contents][agg_content_gid]" ]
  }
}

output {
  stdout {
    codec => json
  }
}

```

результат:

```auto
{"response_fields":{"contents":{"agg":"keep_me"},"foo":"bar"},"host":"bee","@timestamp":"2021-06-07T18:45:21.642Z","@version":"1"}

```

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [June 8, 2021, 7:34am UTC](https://discuss.elastic.co/t/logstash-json/275113/5 "2021-06-08T07:34:29Z")

</div>

Увы, но нет. response\_fields пишется в неизменном виде.

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [June 8, 2021, 8:00am UTC](https://discuss.elastic.co/t/logstash-json/275113/6 "2021-06-08T08:00:01Z")

</div>

Не может же быть проблемы в том, что это поле многократно повторяется в структуре json-а?

 ![123](https://us1.discourse-cdn.com/elastic/original/3X/c/7/c79f827ea7ab047905c584bd50bdbddeb31866b3.png)

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [June 8, 2021, 4:48pm UTC](https://discuss.elastic.co/t/logstash-json/275113/7 "2021-06-08T16:48:01Z")

</div>

test.txt:

```auto
here:{"foo":"bar", "contents": [{"agg_content_gid": "killme", "agg": "keep_me"}, {"agg_content_gid": "killmesoftly", "agg": "keep_me_please"}]}

```

test.conf:

```auto
input {
  stdin {
    id => "my_plugin_id"
  }
}

filter {
  grok {
   match => { "message" => "here:(?<response_body>.*)"}
   remove_field => "message"
  }
  json {
    source => "response_body"
    target => "response_fields"
    remove_field => ["response_body"]
  }
  ruby {
    code => '
        fields = event.get("response_fields")
        contents = fields["contents"]
        if contents
            newContents = []
            contents.each { |c|
                c.delete("agg_content_gid")
                newContents << c
            }
            fields["contents"] = newContents
            event.set("response_fields", fields)
        end
    '
  }
}

output {
  stdout {
    codec => json
  }
}

```

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [June 8, 2021, 5:12pm UTC](https://discuss.elastic.co/t/logstash-json/275113/8 "2021-06-08T17:12:21Z")

</div>

Спасибо, проверю. А все-таки почему json не отрабатывает?

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [June 8, 2021, 5:28pm UTC](https://discuss.elastic.co/t/logstash-json/275113/9 "2021-06-08T17:28:58Z")

</div>

> [@74db36a597f21b891b3f](#):
>
> А все-таки почему json не отрабатывает?

Потому что путь задан не правильно. Правильный путь был бы

```auto
["response_body", "[response_fields][contents][0][agg_content_gid]" , "[response_fields][contents][1][agg_content_gid]", .... и т.д. ]

```

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [June 8, 2021, 6:40pm UTC](https://discuss.elastic.co/t/logstash-json/275113/10 "2021-06-08T18:40:46Z")

</div>

Ругается на "no implicit conversion of string to integer" но направление я понял. Спасибо.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2021, 6:41pm UTC](https://discuss.elastic.co/t/logstash-json/275113/11 "2021-07-06T18:41:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.