# Применение csv фильтра после grok

**URL:** <https://discuss.elastic.co/t/csv-grok/216068>\
**Category:** Вопросы на русском языке\
**Created:** [January 22, 2020, 1:24pm UTC](https://discuss.elastic.co/t/csv-grok/216068 "2020-01-22T13:24:30Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![artem33region](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@artem33region](https://discuss.elastic.co/u/artem33region)\
**Post date:** [January 22, 2020, 1:24pm UTC](https://discuss.elastic.co/t/csv-grok/216068/1 "2020-01-22T13:24:30Z")

</div>

Здравствуйте!  
Есть необходимость парсить csv фильтром то, что не распарсил grok.  
Я сделал конфиг:

```
filter {

  if "ise" in [tags] and ([message] !~ "NetworkDeviceName=ASA1-VPN" and [message] !~ "NetworkDeviceName=ASA2-VPN") {
    drop { }
  }

  if "ise" in [tags] {
    grok {
      patterns_dir => ["/etc/logstash/paterns"]
      match => { "message" => "<181>%{WORD:month}\s*%{NUMBER:date} %{TIME:time} %{DATA:Server} %{DATA:CISE_logging_category} %{NUMBER:trash3} %{NUMBER:trash4} %{NUMBER:trash5} %{DATE_12:Logged_at_start} %{DATA:trash6} %{NUMBER:seq_num} %{NUMBER:cisco_message_code} %{WORD:facility} Radius-Accounting: RADIUS Accounting start request, ConfigVersionId=%{NUMBER:config_version_id}, Device IP Address=%{IP:device_ip}, RequestLatency=%{NUMBER:request_latency}, NetworkDeviceName=%{DATA:device_name_start}, User-Name=%{DATA:vpn_user_start}, %{GREEDYDATA:message2}" }
      match => { "message" => "<181>%{WORD:month}\s*%{NUMBER:date} %{TIME:time} %{DATA:Server} %{DATA:CISE_logging_category} %{NUMBER:trash3} %{NUMBER:trash4} %{NUMBER:trash5} %{DATE_12:Logged_at_stat} %{DATA:trash6} %{NUMBER:seq_num} %{NUMBER:cisco_message_code} %{WORD:facility} Radius-Accounting: RADIUS Accounting stop request, ConfigVersionId=%{NUMBER:config_version_id}, Device IP Address=%{IP:device_ip}, RequestLatency=%{NUMBER:request_latency}, NetworkDeviceName=%{DATA:device_name_stop}, User-Name=%{DATA:vpn_user_stop}, %{GREEDYDATA:message2}" }
      remove_field => ["message"]
    }

    mutate {
      gsub => ["message2", " ", ""]
    }
    csv {
      separator => "," 
      autodetect_column_names => "true"
    }
  }
}

```

Но почему то не работает, возможно ли сделать парсинг с помощью csv после grok?  
Спасибо!

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [January 22, 2020, 2:18pm UTC](https://discuss.elastic.co/t/csv-grok/216068/2 "2020-01-22T14:18:39Z")

</div>

Какое поле вы пытаетесь парсить с помощью фильтра `csv`?

---

<div class="post-metadata">

**Author:** ![artem33region](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@artem33region](https://discuss.elastic.co/u/artem33region)\
**Post date:** [January 22, 2020, 4:58pm UTC](https://discuss.elastic.co/t/csv-grok/216068/3 "2020-01-22T16:58:13Z")

</div>

message2

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [January 22, 2020, 5:14pm UTC](https://discuss.elastic.co/t/csv-grok/216068/4 "2020-01-22T17:14:46Z")

</div>

CVS работает с message [по умолчанию](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-source).

---

<div class="post-metadata">

**Author:** ![artem33region](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@artem33region](https://discuss.elastic.co/u/artem33region)\
**Post date:** [January 23, 2020, 6:45am UTC](https://discuss.elastic.co/t/csv-grok/216068/5 "2020-01-23T06:45:19Z")

</div>

То есть нужно вот так:

```
csv {
  source => "message2"
  separator => "," 
  autodetect_column_names => "true"
}
```

---

<div class="post-metadata">

**Author:** ![artem33region](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@artem33region](https://discuss.elastic.co/u/artem33region)\
**Post date:** [January 23, 2020, 6:55am UTC](https://discuss.elastic.co/t/csv-grok/216068/6 "2020-01-23T06:55:57Z")

</div>

Да, заработало, спасибо!

---

<div class="post-metadata">

**Author:** ![artem33region](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@artem33region](https://discuss.elastic.co/u/artem33region)\
**Post date:** [January 24, 2020, 9:08am UTC](https://discuss.elastic.co/t/csv-grok/216068/7 "2020-01-24T09:08:12Z")

</div>

Игорь, подскажите пожалуйста, что не так. Мне нужно из @timestamp вычесть другую отметку времени для этого написал код:

```
  if "ise" in [tags] and [Acct_Session_Time] {
    date {
      match => ["Acct_Session_Time", "UNIX"]
      target => "Duration_temp"
    }

    ruby {
      code => "
        event.set('Duration', event.get('Duration_temp').time.strftime('%H:%M:%S'))
        event['Session_start'] = (event('@timestamp') - event('Duration_temp'))
        "
    }

```

Фильтр date преобразует число в дату после 1970 года и записывает её в поле "Duration\_temp", это преобразование работает.

Фильтр ruby:  
Первый код вырезает из поля Duration\_temp часы, минуты и секунды, это тоже работает.  
Второй код вычитает из @timestamp, Duration\_temp по сути должно появиться поле Session\_start, но не появляется - не работает.

Подскажите, как мне получить результат вычитания @timestamp из Duration\_temp?

Спасибо!

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [January 24, 2020, 6:53pm UTC](https://discuss.elastic.co/t/csv-grok/216068/8 "2020-01-24T18:53:21Z")

</div>

У `event('@timestamp')` скобки не те, или `.get()` не хватает.

---

<div class="post-metadata">

**Author:** ![artem33region](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@artem33region](https://discuss.elastic.co/u/artem33region)\
**Post date:** [January 27, 2020, 4:42am UTC](https://discuss.elastic.co/t/csv-grok/216068/9 "2020-01-27T04:42:42Z")

</div>

И так и так пробовал, результат ноль..

```
    event.set('Session_start', event.get['@timestamp'] - event.get['Duration_temp'])
    event['Session_start'] = (event['@timestamp'] - event['Duration_temp'])
```

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [January 28, 2020, 9:32pm UTC](https://discuss.elastic.co/t/csv-grok/216068/10 "2020-01-28T21:32:44Z")

</div>

Не могу воспроизвести проблему. Пытался вот так:

**test.conf** :

```auto
input {
  stdin {

  }
}

filter {
  csv {
    separator => ","
    autodetect_column_names => "true"
  }

  date {
    match => ["Acct_Session_Time", "UNIX"]
    target => "Duration_temp"
  }

  ruby {
    code => "
      event.set('Duration', event.get('Duration_temp').time.strftime('%H:%M:%S'))
      event.set('Session_start', (event.get('@timestamp') - event.get('Duration_temp')))
    "
  }

  date {
    match => ["Session_start", "UNIX"]
    target => "Session_start_date"
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

```

**test.log** :

```auto
Acct_Session_Time
2000
3000

```

**Запускаю** :

```auto
 bin/logstash -f config/test.conf < test.log

```

**Получаю** :

```auto
{
              "Duration" => "00:33:20",
            "@timestamp" => 2020-01-28T21:31:03.981Z,
              "@version" => "1",
                  "host" => "grasshopper",
               "message" => "2000",
         "Session_start" => 1580245063.981,
     "Acct_Session_Time" => "2000",
    "Session_start_date" => 2020-01-28T20:57:43.981Z,
         "Duration_temp" => 1970-01-01T00:33:20.000Z
}
{
              "Duration" => "00:50:00",
            "@timestamp" => 2020-01-28T21:31:03.982Z,
              "@version" => "1",
                  "host" => "grasshopper",
               "message" => "3000",
         "Session_start" => 1580244063.982,
     "Acct_Session_Time" => "3000",
    "Session_start_date" => 2020-01-28T20:41:03.982Z,
         "Duration_temp" => 1970-01-01T00:50:00.000Z
}

```

Все, вроде, как надо.

---

<div class="post-metadata">

**Author:** ![artem33region](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@artem33region](https://discuss.elastic.co/u/artem33region)\
**Post date:** [January 29, 2020, 6:15am UTC](https://discuss.elastic.co/t/csv-grok/216068/11 "2020-01-29T06:15:32Z")

</div>

Игорь спасибо!  
Как всегда, Вы на высоте!)

---

<div class="post-metadata">

**Author:** ![artem33region](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@artem33region](https://discuss.elastic.co/u/artem33region)\
**Post date:** [January 29, 2020, 5:34pm UTC](https://discuss.elastic.co/t/csv-grok/216068/12 "2020-01-29T17:34:39Z")

</div>

Единственное что, у меня почему то еще минус 3 часа вычитает)  
Походу с часовым поясом, что-то..

```
@timestamp Jan 29, 2020 @ 20:34:37.197
Acct_Session_Time 5637
Duration 01:33:57
Duration_temp 1970-01-01T01:33:57.000Z
Session_start 1580313640.197
Session_start_date 2020-01-29T16:00:40.197Z
```

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [January 29, 2020, 6:19pm UTC](https://discuss.elastic.co/t/csv-grok/216068/13 "2020-01-29T18:19:57Z")

</div>

Как вы этот вывод получили?

---

<div class="post-metadata">

**Author:** ![artem33region](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@artem33region](https://discuss.elastic.co/u/artem33region)\
**Post date:** [January 30, 2020, 4:24am UTC](https://discuss.elastic.co/t/csv-grok/216068/14 "2020-01-30T04:24:05Z")

</div>

Добавил Ваш конфиг в свой и просто скопировал из кибаны.

---

<div class="post-metadata">

**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\
**Post date:** [January 30, 2020, 2:09pm UTC](https://discuss.elastic.co/t/csv-grok/216068/15 "2020-01-30T14:09:12Z")

</div>

Из какой части кибаны? Если из Discover, то там поля с типом date показываются в вашей зоне. Остильные поля, скорее всего текстовые, поэтому они в той зоне, в которой вы их обрабатывали. Все что оканчивается на `Z` - в UTC, если временной зоны в конце нет - то это ваше местное время.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 27, 2020, 2:09pm UTC](https://discuss.elastic.co/t/csv-grok/216068/16 "2020-02-27T14:09:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.