# Помогите разобраться с grok filter

**URL:** <https://discuss.elastic.co/t/grok-filter/121988>\
**Category:** Вопросы на русском языке\
**Created:** [March 1, 2018, 6:48am UTC](https://discuss.elastic.co/t/grok-filter/121988 "2018-03-01T06:48:03Z")\
**Posts on this page:** 19\
**Page:** 2

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 12, 2018, 3:38pm UTC](https://discuss.elastic.co/t/grok-filter/121988/21 "2018-03-12T15:38:13Z")

</div>

Да я ваши предыдущие рекомендации держу в уме, просто пока не добрался до них. Попробую разобраться с этим, если получится.  
Меня смутило, что %{[text][0]} и %{[created\_at][0]} в выводе.

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 13, 2018, 6:33am UTC](https://discuss.elastic.co/t/grok-filter/121988/22 "2018-03-13T06:33:36Z")

</div>

Попробовал последовать Вашей рекомендации, и сделал такой конфиг:

```
input {
             http_poller {
               urls => {
                  test => {
                   method => get
                   url => "https://plesk.uservoice.com/api/v1/forums/184549/suggestions.json?sort=newest"
                   headers => {
                      Accept => "application/json"
                      Authorization => "Bearer xxxxxxxxxx"
                   }

                 }
               }
               request_timeout => 60
               # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
               # schedule => { cron => "* * * * * UTC"}
               schedule => { every => "1h"}
               codec => "plain"
             }
           }

filter {
  xml {
                source => "message"
                target => "xmldata"
                #store_xml => "false"
                xpath => ["/response/suggestion","suggestion"]
  }

  mutate {
    remove_field => ["message", "xmldata"]
  }

  split {
    field => "[suggestion]"
  }

  xml {
    source => "suggestion"
    xpath => ["suggestion/supporters_count/text()", "supporters_count"]
    xpath => ["suggestion/created_at/text()", "created_at"]
    xpath => ["suggestion/text/text()", "text"]
    force_content => true
    force_array => false
    remove_field => ["message"]
    store_xml => false
  }
  mutate {
    convert => {
      "supporters_count" => "integer"
      "created_at" => "string"
      "text" => "string"
    }
    replace => {
      "supporters_count" => "%{[supporters_count][0]}"
      "created_at" => "%{[created_at][0]}"
      "text" => "%{[text][0]}"
    }
  }
}

```

В результате [https://screenshots.firefox.com/lNz170RX6DjAI44F/talkkib.plesk.com](https://screenshots.firefox.com/lNz170RX6DjAI44F/talkkib.plesk.com)  
А в логе:

`[2018-03-13T06:25:17,353][WARN][logstash.filters.split] Only String and Array types are splittable. field:[suggestion] is of type = NilClass`

Почему-то по suggestion не сплитится...

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [March 13, 2018, 9:46pm UTC](https://discuss.elastic.co/t/grok-filter/121988/23 "2018-03-13T21:46:46Z")

</div>

Вы не пробовали поставить stdout вывод после первого xml фильтра, убрать все остальное и посмотерть, что выводиться? Если бы вы это сделали, то вы могли бы заметить, что у вас suggestion в первом фильтре не выбираются потому, что xpath не правильный.

```auto
filter {
  # First extract individual suggestions
  xml {
    source => "message"
    xpath => ["/response/suggestions/suggestion", "suggestion"]
    force_content => false
    force_array => true
    remove_field => ["message"]
    store_xml => false
  }
  # Split suggestions into records
  split {
   field => "suggestion"
 }
 # Now parse each xml in each record
 xml {
   source => "suggestion"
   xpath => ["/suggestion/supporters_count/text()", "supporters_count"]
   xpath => ["/suggestion/created_at/text()", "created_at"]
   xpath => ["/suggestion/text/text()", "text"]
   force_content => true
   force_array => false
   remove_field => ["suggestion"]
   store_xml => false
 }
 # The filter above may produce multiple entries for each xpath element
 # We need to adjust type and interested only in the first one
 mutate {
   convert => {
     "supporters_count" => "integer"
     "created_at" => "string"
     "text" => "string"
   }
   replace => {
     "supporters_count" => "%{[supporters_count][0]}"
     "created_at" => "%{[created_at][0]}"
     "text" => "%{[text][0]}"
   }
 }
}

```

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 14, 2018, 4:36am UTC](https://discuss.elastic.co/t/grok-filter/121988/24 "2018-03-14T04:36:50Z")

</div>

Попробовал с простейшим конфигом:

```
input {
   http_poller {
     urls => {
         test => {
                   # Supports all options supported by ruby's Manticore HTTP client
                   method => get
                   url => "https://plesk.uservoice.com/api/v1/forums/184549/suggestions?sort=newest"
                   headers => {
                      Accept => "application/json"
                      Authorization => "Bearer xxxxxxxx"
                   }

                 }
               }
               request_timeout => 60
               # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
               # schedule => { cron => "* * * * * UTC"}
               schedule => { every => "1h"}
               codec => "plain"
             }
}

filter {
  # First extract individual suggestions
  xml {
    source => "message"
    xpath => ["/response/suggestions/suggestion", "suggestion"]
    force_content => false
    force_array => true
    remove_field => ["message"]
    store_xml => false
  }
  # Split suggestions into records
  split {
   field => "suggestion"
 }
}

output {
  stdout { codec => rubydebug }
}

```

Все равно не парсится. Та же ошибка:

`[2018-03-14T04:20:28,514][WARN][logstash.filters.split] Only String and Array types are splittable. field:suggestion is of type = NilClass`

Уже и не знаю, видимо дело не в xpath, а в чем-то еще. Ну как еще его можно написать, если в выводе:

```
 curl -XGET "https://plesk.uservoice.com/api/v1/forums/184549/suggestions?sort=newest" -H "Authorization: Bearer xxxxxxxxx"

<?xml version="1.0" encoding="UTF-8"?>
<response>
<response_data>
  <page type="integer">1</page>
  <per_page type="integer">10</per_page>
  <total_records type="integer">1911</total_records>
  <filter>public</filter>
  <sort>newest</sort>
</response_data>
  <suggestions type="array">
    <suggestion>
<url>http://plesk.uservoice.com/forums/184549-feature-suggestions/suggestions/33620917-option-to-not-select-alias-domains-in-let-s-encryp</url>
<path>/forums/184549-feature-suggestions/suggestions/33620917-option-to-not-select-alias-domains-in-let-s-encryp</path>
<id type="integer">33620917</id>
<state>published</state>
<title>Option to not select alias domains in Let's Encrypt by default</title>
<text>By default, the Let's Encrypt extension pre-selects all alias domains of a domain to be included in the Let's Encrypt certificate. This can lead to some unwanted results, for example when one of the alias domains is removed from the DNS at a later stage (because it was only used for testing) which would result in Let's Encrypt being unable to renew the certificate.

It would be nice if there was an option that allows us to configure that alias domains should _not_ be selected by default in the Let's Encrypt extension.
</text>

```

Не видит он suggestion и все.

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 14, 2018, 6:05am UTC](https://discuss.elastic.co/t/grok-filter/121988/25 "2018-03-14T06:05:52Z")

</div>

Кажется, понял в чем проблема. Дело в том, что много раз внутри вывода еще встречается, помимо самого первого . Видимо поэтому xpath неверный получается.  
Попробовал

xpath =\> ["/suggestions/suggestion", "suggestion"]

не помогло. Как его правильно написать, чтобы игнорировались все эти response?

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [March 14, 2018, 11:55am UTC](https://discuss.elastic.co/t/grok-filter/121988/26 "2018-03-14T11:55:00Z")

</div>

А что выводиться, если убрать split?

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 14, 2018, 1:18pm UTC](https://discuss.elastic.co/t/grok-filter/121988/27 "2018-03-14T13:18:56Z")

</div>

Ошибки сплита нет, конечно. А в выводе что-то вроде

[https://gist.githubusercontent.com/IronButterfly/99356b3f12995aa4e4c0e3657a9b6e7b/raw/9606eeafee782e03d05dbe1f0670c59ca8fd0940/gistfile1.txt](https://gist.githubusercontent.com/IronButterfly/99356b3f12995aa4e4c0e3657a9b6e7b/raw/9606eeafee782e03d05dbe1f0670c59ca8fd0940/gistfile1.txt)

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [March 14, 2018, 2:25pm UTC](https://discuss.elastic.co/t/grok-filter/121988/28 "2018-03-14T14:25:50Z")

</div>

Ну так вы получаете информацию в json, а не в xml -

```auto
 "message": "{\"response_data\":{\"page\":1,\"per_page\":10,\"total_records\":1911,\"filter\":\"public\",\"sort\":\"newest\"},\"suggestions\":[{\"url\":\"http://plesk.uservoice.com/forums/184549-feature-suggestions/suggestions/33620917-option-to-not-select-alias-domains-in-let-s-

```

Естественно, он ничего не парсит и все оставляет в message. В таком случае, вам действительно надо использовать кодек json. Уберите xml фильтр, замените кодек в http\_pooler на json, и давайте посмотрим, что он будет выдавать.

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 14, 2018, 3:07pm UTC](https://discuss.elastic.co/t/grok-filter/121988/29 "2018-03-14T15:07:17Z")

</div>

Убрал все фильтры, кодек в http\_pooler поставил вместо plain в json и вот что в результате:

> <https://gist.github.com/IronButterfly/ff9d501ac46d5052572e16d203370e48>

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [March 15, 2018, 12:46am UTC](https://discuss.elastic.co/t/grok-filter/121988/30 "2018-03-15T00:46:25Z")

</div>

```auto
filter {
  # Split suggestions into records
  split {
    field => "[_source][suggestions]"
  }
  # Extract fields that we care about and delete the rest
  mutate {
    add_field => {"formatted_text" => "%{[_source][suggestions][formatted_text]}"}
    add_field => {"created_at" => "%{[_source][suggestions][created_at]}"}
    add_field => {"supporters_count" => "%{[_source][suggestions][supporters_count]}"}
    remove_field => "_source"
  }
}

```

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 15, 2018, 5:09am UTC](https://discuss.elastic.co/t/grok-filter/121988/31 "2018-03-15T05:09:08Z")

</div>

Все равно не сплитит.

`[2018-03-15T04:56:00,116][WARN][logstash.filters.split] Only String and Array types are splittable. field:[_source][suggestions] is of type = NilClass`

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [March 15, 2018, 11:45am UTC](https://discuss.elastic.co/t/grok-filter/121988/32 "2018-03-15T11:45:10Z")

</div>

Я пробовал с тем json, который вы мне прислали читая его из файла - все работает. Вам надо убедиться, что путь `_source/suggestions` существует в записи, которую вы получаете от http\_pooler.

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 16, 2018, 7:00am UTC](https://discuss.elastic.co/t/grok-filter/121988/33 "2018-03-16T07:00:57Z")

</div>

Уффф... Наконец-то разобрался 🙂 Стало сплитится и все как надо выводить с таким вот конфигом:

```
input {
   http_poller {
     urls => {
         test => {
                   method => get
                   url => "https://plesk.uservoice.com/api/v1/forums/184549/suggestions?sort=newest"
                   headers => {
                      Accept => "application/json"
                      Authorization => "Bearer xxxxxxxxx"
                   }
                   
                 }
               }
               request_timeout => 60
               schedule => { every => "1h"}
               codec => "json"
             }
}

filter {
  # Split suggestions into records
  split {
    field => "[suggestions]"
  }
  # Extract fields that we care about and delete the rest
  mutate {
    add_field => {"id" => "%{[suggestions][id]}"}
    add_field => {"title" => "%{[suggestions][title]}"}
    add_field => {"text" => "%{[suggestions][text]}"}
    add_field => {"created_at" => "%{[suggestions][created_at]}"}
    add_field => {"status" => "%{[suggestions][status][name]}"}
    add_field => {"category" => "%{[suggestions][category][name]}"}
    add_field => {"supporters_count" => "%{[suggestions][supporters_count]}"}
    remove_field => "suggestions"
    remove_field => "response_data"
  }
}

```

Но возникли еще вопросы. Поможете, если я Вам еще не надоел?

1. Вывод выглядит вот так:

проблема в том, что вместо "@timestamp" мне нужно использовать "created\_at". Но это поле идет в индекс как текст, а не как дата.

1. В соответствии с полем schedule выборка происходит каждый час. Хотелось бы, чтобы в тех записях в индеске ElasticSearch, которые уже существуют, обновлялись измененные поля, и добавлялись новые записи, которые появились за прошедший час.

2. Хотелось бы как-то из поля "text" вырезать все URLs и emails, и кроме того, переводить весь текст в lowercase.

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 16, 2018, 8:34am UTC](https://discuss.elastic.co/t/grok-filter/121988/34 "2018-03-16T08:34:22Z")

</div>

Разобрался с пунктом 1 и частично с пунктом 3 с помощью такого фильтра:

```
filter {
  # Split suggestions into records
  split {
    field => "[suggestions]"
  }
  # Extract fields that we care about and delete the rest
  mutate {
    add_field => {"id" => "%{[suggestions][id]}"}
    add_field => {"title" => "%{[suggestions][title]}"}
    add_field => {"text" => "%{[suggestions][text]}"}
    add_field => {"created_at" => "%{[suggestions][created_at]}"}
    add_field => {"status" => "%{[suggestions][status][name]}"}
    add_field => {"category" => "%{[suggestions][category][name]}"}
    add_field => {"supporters_count" => "%{[suggestions][supporters_count]}"}
    remove_field => "suggestions"
    remove_field => "response_data"
    #remove_field => "@timestamp"
  }
  mutate {
    lowercase => ["text"]
  }
  mutate {
    gsub => ["created_at", "/", "-"]
}

  date {
    match => ["created_at", "YYYY-MM-dd HH:mm:ss Z"]
    target => "created_at"
  }
}

```

Остался пункт 2 и вырезание урлов и емейлов.

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [March 16, 2018, 2:39pm UTC](https://discuss.elastic.co/t/grok-filter/121988/35 "2018-03-16T14:39:32Z")

</div>

> [@pavuk](#):
>
> Остался пункт 2

Вам надо просто индексировать эти записи с одним и тем же id в elasticsearch. Это можно достичь, указав поле с id в параметре [document\_id](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id) .

> [@pavuk](#):
>
> и вырезание урлов и емейлов.

Это можно достичь с помощью [gsub в mutate](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub).

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 16, 2018, 3:26pm UTC](https://discuss.elastic.co/t/grok-filter/121988/36 "2018-03-16T15:26:30Z")

</div>

Что-то не пойму, как правильно написать document\_id.  
Сделал:

`document_id => "id"`

Но, так только одна запись добавляется.  
Если так:

`document_id => "%{[suggestions][id]}"`

то ошибка.

Про gsub понятно. Осталось только правильные regexp-ы написать.

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [March 16, 2018, 8:22pm UTC](https://discuss.elastic.co/t/grok-filter/121988/37 "2018-03-16T20:22:35Z")

</div>

```auto
document_id => "%{id}"

```

---

<div class="post-metadata">

**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\
**Post date:** [March 19, 2018, 3:30am UTC](https://discuss.elastic.co/t/grok-filter/121988/38 "2018-03-19T03:30:44Z")

</div>

Огромное спасибо за помощь и науку @Igor_Motov ! В итоге, получил практически то, что хотел изначально.

Может быть кому-то будет интересно и полезно, как получать данные из UserVoice по API. Мой почти финальный конфиг получился таким:

```
input {
  http_poller {
    urls => {
      test => {
        method => get
        url => "https://plesk.uservoice.com/api/v1/forums/184549/suggestions?per_page=300?sort=newest"
        headers => {
          Accept => "application/json"
          Authorization => "Bearer xxxxxxxxxxxx"
                   }
                   
               }
             }
          request_timeout => 60
          schedule => { every => "1h"}
          codec => "json"
             }
}

filter {
  
  split {
    field => "[suggestions]"
  }
  
  mutate {
    add_field => {"id" => "%{[suggestions][id]}"}
    add_field => {"title" => "%{[suggestions][title]}"}
    add_field => {"text" => "%{[suggestions][text]}"}
    add_field => {"created_at" => "%{[suggestions][created_at]}"}
    add_field => {"status" => "%{[suggestions][status][name]}"}
    add_field => {"category" => "%{[suggestions][category][name]}"}
    add_field => {"supporters_count" => "%{[suggestions][supporters_count]}"}
    remove_field => "suggestions"
    remove_field => "response_data"
    remove_field => "@timestamp"
  }

  if [status] == "%{[suggestions][status][name]}" {
    mutate {
     replace => ["status", "No status"]
  }
  }

  if [text] == "%{[suggestions][text]}" {
    mutate {
     replace => ["text", "%{title}"]
  }
  }

  mutate {
    strip => ["text"]
  }
  
  mutate {
    lowercase => ["text"]
  }
  
  mutate {
    gsub => ["text", "http\S+", ""]
  }

  mutate {
    convert => { "supporters_count" => "integer" }
  }

  mutate {
    gsub => ["created_at", "/", "-"]
  }

  date {
    match => ["created_at", "YYYY-MM-dd HH:mm:ss Z"]
    target => "created_at"
  }

}

output {
  elasticsearch {
    hosts => ["http://xxxxxxxxx.com:9200/"]
    user => xxxxx
    password => xxxxx
    index => "uservoice"
    document_id => "%{id}"
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 16, 2018, 3:30am UTC](https://discuss.elastic.co/t/grok-filter/121988/39 "2018-04-16T03:30:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/grok-filter/121988.md?page=1)