# How to let logstash splits event field values and assign it to @metadata field

**URL:** <https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833>\
**Category:** Logstash\
**Created:** [August 11, 2016, 3:29pm UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833 "2016-08-11T15:29:28Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![daiyue](https://avatars.discourse-cdn.com/v4/letter/d/bc8723/32.png) [@daiyue](https://discuss.elastic.co/u/daiyue)\
**Post date:** [August 11, 2016, 3:29pm UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/1 "2016-08-11T15:29:28Z")

</div>

Hi, I have a `logstash` event, which has the following field,

```
{
  "_index": "logstash-2016.08.09",
  "_type": "log",
  "_id": "AVZvz2ix",
  "_score": null,
  "_source": {
    "message": "function_name~execute||line_no~128||debug_message~id was not found",
    "@version": "1",
    "@timestamp": "2016-08-09T14:57:00.147Z",
    "beat": {
      "hostname": "coredev",
      "name": "coredev"
    },
    "count": 1,
    "fields": null,
    "input_type": "log",
    "offset": 22299196,
    "source": "/project_root/project_1/log/core.log",
    "type": "log",
    "host": "coredev",
    "tags": [
      "beats_input_codec_plain_applied"
    ]
  },
  "fields": {
    "@timestamp": [
      1470754620147
    ]
  },
  "sort": [
    1470754620147
  ]
}

```

I am wondering how to use `filter` (`kv` maybe?) to extract `core.log` from `"source": "/project_root/project_1/log/core.log"`, and put it in e.g. `[@metadata][log_type]`, and so later on, I can use `log_type` in `output` to create an unique `index`, composing of hostname + logtype + timestamp, e.g.

```
output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][_source][host]}-%{[@metadata][log_type]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
  stdout { codec => rubydebug }
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 12, 2016, 6:06am UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/2 "2016-08-12T06:06:38Z")

</div>

Use a grok filter.

```plaintext
grok {
  match => ["source", "/(?<[@metadata][log_type]>[^/]+)$"]
}

```

> ... to create an unique index, composing of hostname + logtype + timestamp ...

Why so many indexes? You are aware of the constant per-shard overhead?

---

<div class="post-metadata">

**Author:** ![daiyue](https://avatars.discourse-cdn.com/v4/letter/d/bc8723/32.png) [@daiyue](https://discuss.elastic.co/u/daiyue)\
**Post date:** [August 12, 2016, 11:20am UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/3 "2016-08-12T11:20:54Z")

</div>

Hi, yes, I know that there is a fixed overhead for using each index, and can be resource-intensive on aggregations. The logs harvested are coming from different modules of the whole system, so they do possess a different meanings in terms of the fields.

---

<div class="post-metadata">

**Author:** ![daiyue](https://avatars.discourse-cdn.com/v4/letter/d/bc8723/32.png) [@daiyue](https://discuss.elastic.co/u/daiyue)\
**Post date:** [August 12, 2016, 2:04pm UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/4 "2016-08-12T14:04:40Z")

</div>

Hi, I am wondering whats the syntax behind

`"/(?<[@metadata][log_type]>[^/]+)$"`

`grok` `match` seems to use `=>` between field and value?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 12, 2016, 2:07pm UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/5 "2016-08-12T14:07:12Z")

</div>

You can use

```
match => { "source" => "/(?<[@metadata][log_type]>[^/]+)$" }

```

if you prefer. They're equivalent.

---

<div class="post-metadata">

**Author:** ![daiyue](https://avatars.discourse-cdn.com/v4/letter/d/bc8723/32.png) [@daiyue](https://discuss.elastic.co/u/daiyue)\
**Post date:** [August 12, 2016, 2:09pm UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/6 "2016-08-12T14:09:31Z")

</div>

Is it a regex for `"/(?<[@metadata][log_type]>[^/]+)$"`? How does it work?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 12, 2016, 2:12pm UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/7 "2016-08-12T14:12:18Z")

</div>

`/(?<[@metadata][log_type]>[^/]+)$` is a regexp, yes. After a slash it captures one or more characters that are not slashes (`[^/]+`) up to the end of the line and stores them in the `[@metadata][log_type]` field.

---

<div class="post-metadata">

**Author:** ![daiyue](https://avatars.discourse-cdn.com/v4/letter/d/bc8723/32.png) [@daiyue](https://discuss.elastic.co/u/daiyue)\
**Post date:** [August 12, 2016, 2:25pm UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/8 "2016-08-12T14:25:30Z")

</div>

thx for the explanation, where can I find the reference to the syntax of the expression, since it looks a bit different from the regex. For the store value part `?<>` particularly.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 12, 2016, 2:29pm UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/9 "2016-08-12T14:29:58Z")

</div>

No, that's pretty standard stuff (for modern regexps at least).

[http://www.regular-expressions.info/named.html](http://www.regular-expressions.info/named.html)

---

<div class="post-metadata">

**Author:** ![daiyue](https://avatars.discourse-cdn.com/v4/letter/d/bc8723/32.png) [@daiyue](https://discuss.elastic.co/u/daiyue)\
**Post date:** [August 16, 2016, 10:25am UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/10 "2016-08-16T10:25:10Z")

</div>

Hi, I tried the script for extracting log type,

```
filter {

  grok {
    match => ["source", "/(?<[@metadata][log_type]>[^/]+)$"]
  }

  kv { value_split => "~"
       field_split => "||" }

  date {
    locale => "en"
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]
  }
}

output {
    elasticsearch {
      hosts => "172.17.0.2:9200"
      manage_template => false
      index => "%{host}-%{[@metadata][log_type]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][log_type]}"
    }
    stdout { codec => rubydebug }
}

```

it generates the following errors:

```
logstash_1 | {:timestamp=>"2016-08-16T10:09:41.352000+0000", :message=>"Pipeline aborted due to error", :exception=>#<RegexpError: invalid char in group name <[@metadata][log_type]>: /\/(?<[@metadata][log_type]>[^\/]+)$/m>, :backtrace=>["org/jruby/RubyRegexp.java:1434:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:127:in `compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473:in `start_pipeline'"], :level=>:error}
logstash_1 | {:timestamp=>"2016-08-16T10:09:44.354000+0000", :message=>"stopping pipeline", :id=>"main"}
dockerelk_logstash_1 exited with code 0

```

How to resolve the issue?

thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 11:14am UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/11 "2016-08-16T11:14:56Z")

</div>

Okay. It seems you can't have square brackets (or at signs?) in named capture destinations. Pick another field name. You can use a mutate filter to rename it to `[@metadata][log_type]` later on.

---

<div class="post-metadata">

**Author:** ![daiyue](https://avatars.discourse-cdn.com/v4/letter/d/bc8723/32.png) [@daiyue](https://discuss.elastic.co/u/daiyue)\
**Post date:** [August 16, 2016, 11:35am UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/12 "2016-08-16T11:35:39Z")

</div>

thanks for replying, what do you mean by 'pick another field name'?

cheers

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 11:42am UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/13 "2016-08-16T11:42:52Z")

</div>

Don't call the field `[@metadata][log_type]` in the grok expression. Call it something else.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:43am UTC](https://discuss.elastic.co/t/how-to-let-logstash-splits-event-field-values-and-assign-it-to-metadata-field/57833/14 "2017-07-06T04:43:21Z")

</div>