# Splitting logs to custom format

**URL:** <https://discuss.elastic.co/t/splitting-logs-to-custom-format/89824>\
**Category:** Logstash\
**Created:** [June 18, 2017, 5:53am UTC](https://discuss.elastic.co/t/splitting-logs-to-custom-format/89824 "2017-06-18T05:53:48Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [June 18, 2017, 5:53am UTC](https://discuss.elastic.co/t/splitting-logs-to-custom-format/89824/1 "2017-06-18T05:53:49Z")

</div>

Hi I have logs that in general looks like:

UID A\_TS B\_TS C\_TS D\_TS X Y Z

index: (UID = unique Id , TS = time Stamp , X etc = other values )

I want to create from those logs tables ("logs") that will be in the following format:

UID A\_TS X Y Z  
UID B\_TS X Y Z  
UID C\_TS X Y Z  
UID D\_TS X Y Z

Even Better to create:

UID A\_TS X Z  
UID B\_TS Y Z  
UID C\_TS X Y Z  
UID D\_TS X

How is this possible in Logstash?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 20, 2017, 5:56am UTC](https://discuss.elastic.co/t/splitting-logs-to-custom-format/89824/2 "2017-06-20T05:56:06Z")

</div>

Use a csv filter to parse the input. Store the timestamp values in the same field so that they form an array. Then use the split filter to split each input event into multiple events with the timestamp field being variable. Finally use a file output (if you indeed want to produce files) that uses a line codec with a custom format. Something like this might work:

```nohighlight
filter {
  csv {
    columns => ["UID", "A_TS", "B_TS", "C_TS", "X", "Y", "Z"]
    separator => " "
  }
  mutate {
    add_field => ["TS", "%{A_TS}"]
    add_field => ["TS", "%{B_TS}"]
    add_field => ["TS", "%{C_TS}"]
    remove_field => ["A_TS", "B_TS", "C_TS"]
  }
  split {
    field => "TS"
  }
}
output {
  file {
    ...
    codec => line {
      format => "%{UID} %{TS} %{X} %{Y} %{Z}"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [June 20, 2017, 8:17am UTC](https://discuss.elastic.co/t/splitting-logs-to-custom-format/89824/3 "2017-06-20T08:17:29Z")

</div>

Thanks!

Is it possible to send in this case also straight to ES? (I believe it is I just want to be sure before I start)  
or this is from some reason a special case (since it is CSV)?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 20, 2017, 8:33am UTC](https://discuss.elastic.co/t/splitting-logs-to-custom-format/89824/4 "2017-06-20T08:33:15Z")

</div>

Sure, you can send it to ES.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2017, 8:33am UTC](https://discuss.elastic.co/t/splitting-logs-to-custom-format/89824/5 "2017-07-18T08:33:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.