Generating multiple events from a single message

Elastic Stack Logstash
1

Hi,

I have a csv file that looks like this: The first line is the header column

Item, Type, Category, Q1, Q2, Q3, Q4
Tools, License, Cost Saving, 15000, 0, 6000, 899

I am trying to consume this in Logstash and send the resulting events/documents to elasticsearch. The overall functionality works but what I want is to generate 4 events /documents from each row like this:

Item: Tools
Category: Cost Saving
Type: License
Quarter: Q1
Quarter1_Value : 15000

Item: Tools
Category: Cost Saving
Type: License
Quarter: Q2
Quarter2Value: 0

and so on for Q3 and Q4 too.

My csv filter currently looks like this:
filter {
csv {
columns => ["item","category","type","q1","q2", "q3", "q4"]
add_field => {
"quarter" => "q1"
"amount" => "%{q1}"
}
}
}

But what I am missing is a way to generate 4 events from one single row. Is this possible? And is there another filter that I have to use to accomplish this? I am new to Logstash and any help is greatly appreciated.

Thanks

2

It does not do what you want, but this might get you started...

filter {
  mutate { gsub => [ "message", ", ", "," ] }
  csv { autodetect_column_names => true }
  ruby { code => "event.set('Qs', [ event.get('Q1') , event.get('Q2') , event.get('Q3') , event.get('Q4') ] )" }
  split { field => "Qs" }
}

Actually you could add

  if [Q1] == [Qs] { mutate { add_field => { "Quarter" => "Q1" } } }
  if [Q2] == [Qs] { mutate { add_field => { "Quarter" => "Q2" } } }
  if [Q3] == [Qs] { mutate { add_field => { "Quarter" => "Q3" } } }
  if [Q4] == [Qs] { mutate { add_field => { "Quarter" => "Q4" } } }

to that and it would do pretty much what you want. But looking at a filter that ugly might make your eyes bleed :slight_smile: Probably also need to mutate / remove_field some stuff.

mutate { remove_field => [ "Q1", "Q2", "Q3", "Q4" ] }
3

Thank you so much. Will try the suggestion and get back to you.