Import CSV file into nested fields

Elastic Stack Logstash
1

I'm trying to import a CSV file into nested fields. Trying to follow the aggregate example #4, is coming up short for me. This partially works, but not really what I'm looking for. I'd just like the values grouped in some way, either by array index or ?. I didn't see a nested forum post, so creating a new one. Here is where I'm at so far. Just simply trying to group these by Id and then a nested structure / array for the rest.

CSV file:
1,123Name,2.03
1,456Name,2.03
2,123Name,2.03
2,789Name,2.03

filter {
  csv {
    autogenerate_column_names => false
    skip_header => true
    columns => ["Id","IdName","IdVersion"]
  }

  aggregate {
    task_id => "%{Id}"
    code => "
      map['Id'] ||= event.get('Id')
      map['ListofVals'] ||= []
      if !( map['ListofVals'].include? event.get('IdName') )
        map['ListofVals'] << {'IdName' => event.get('IdName')}
        map['ListofVals'] << {'IdVersion' => event.get('IdVersion')}
      end
      event.cancel()
    "
    push_previous_map_as_event => true
    timeout => 5
  }


          "Id" : "1",
          "ListofVals" : [
            {
              "IdName" : "123Name"
            },
            {
              "IdVersion" : "2.03"
            },
            {
              "IdName" : "456Name"
            },
            {
              "IdVersion" : "2.03"
            },
          ]
		  ....
          "Id" : "2",
          "ListofVals" : [
            {
              "IdName" : "123Name"
            },
            {
              "IdVersion" : "2.03"
            },
            {
              "IdName" : "789Name"
            },
            {
              "IdVersion" : "2.03"
            },
          ]
2

What don't you like about the result that you got?

3

I'm trying to get them to something like:

ListofVals [
{ IdName:123Name, IdVersion: 2.03 },
{ IdName: 456Name, IdVersion: 2.03}
]
Or
ListofVals [
[0]
IdName:123Name, IdVersion: 2.03
[1]
IdName: 456Name, IdVersion: 2.03
]
Something like that...a separation of grouped values that corresponds to what is in the CSV.

4

You can get the first using

    aggregate {
        task_id => "%{Id}"
        push_map_as_event_on_timeout => true
        code => '
            map["Id"] ||= event.get("Id")
            map["ListofVals"] ||= []
            if !( map["ListofVals"].include? event.get("IdName") )
                map["ListofVals"] << {"IdName" => event.get("IdName"), "IdVersion" => event.get("IdVersion")}
            end
            event.cancel()
        '
        timeout => 5
    }

I would use push_map_as_event_on_timeout instead of push_previous_map_as_event in case there are out-of-order lines in the file.

I do not understand what you mean by the second.

1 Like
5

Thank you! The second was just bad pseudo code...

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.