Splitting logs to custom format

Elastic Stack Logstash
1

Hi I have logs that in general looks like:

UID A_TS B_TS C_TS D_TS X Y Z

index: (UID = unique Id , TS = time Stamp , X etc = other values )

I want to create from those logs tables ("logs") that will be in the following format:

UID A_TS X Y Z
UID B_TS X Y Z
UID C_TS X Y Z
UID D_TS X Y Z

Even Better to create:

UID A_TS X Z
UID B_TS Y Z
UID C_TS X Y Z
UID D_TS X

How is this possible in Logstash?

2

Use a csv filter to parse the input. Store the timestamp values in the same field so that they form an array. Then use the split filter to split each input event into multiple events with the timestamp field being variable. Finally use a file output (if you indeed want to produce files) that uses a line codec with a custom format. Something like this might work:

filter {
  csv {
    columns => ["UID", "A_TS", "B_TS", "C_TS", "X", "Y", "Z"]
    separator => " "
  }
  mutate {
    add_field => ["TS", "%{A_TS}"]
    add_field => ["TS", "%{B_TS}"]
    add_field => ["TS", "%{C_TS}"]
    remove_field => ["A_TS", "B_TS", "C_TS"]
  }
  split {
    field => "TS"
  }
}
output {
  file {
    ...
    codec => line {
      format => "%{UID} %{TS} %{X} %{Y} %{Z}"
    }
  }
}
1 Like
3

Thanks!

Is it possible to send in this case also straight to ES? (I believe it is I just want to be sure before I start)
or this is from some reason a special case (since it is CSV)?

4

Sure, you can send it to ES.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.