Mapping multiple values in single CSV row output

[karnamonkster (Karan)]

We have an ES index that has different register for Meters
Example:

Event1
meter: FT1234
register:FLOW RATE
value: somevalue1

Event2
meter:FT1234
register:REV RATE
value: somevalue2

Event3
meter:FT1234
register:VOLUME
value: somevalue3

Now i needs all these register into a single CSV output file which would be like
Headers: Meter, FLOWRATE,REVRATE,VOLUME
VALUES: FT1234,somevalue1,somevalue2,somevalue3

[Badger]

You can do that with a csv output.

[karnamonkster (Karan)]

Hi @Badger
That is what I intent to test, but I am not able to proceed with how to get two registers in same row under their respective headers.

[Badger]

Oh, I missed that. You can use an aggregate filter to combine all the values for a given meter id.

[karnamonkster (Karan)]

Hi @Badger,

Here is what my unsuccessful attempt looks like.

filter {

if [register] == "FLOW RATE" {
    aggregate {
            task_id => "%{meterid}"
            code => "map['flow_rate'] = event.get('value')"
        }
    }
if [register] == "FWD TOTALIZER MES" {
    aggregate {
            task_id => "%{meterid}"
            code => "map['fwd_mes'] = event.get('value')"
            
        }
    }
if [register] == "REV TOTALIZER MES" {
    aggregate {
            task_id => "%{meterid}"
            code => "map['rev_mes'] = event.get('value');
                    event.set('flow_rate', map['flow_rate']);
                    event.set('fwd_mes', map['fwd_mes']);
                    event.set('rev_mes', map['rev_mes']);
                        "
            end_of_task => true
            
        }
    }    

}
I am sure the logic not correct since i do not have any start and end event.
Its registers fetched in different times, and once i have these 3 i push it.

[karnamonkster (Karan)]

It works with the same configuration.
Thanks @Badger , But i am still not able to get a simple Header on my output CSV

[Badger]

You can tell the csv output to add a header but because every line is written independently that results in every other line being a header.

[karnamonkster (Karan)]

Hi @Badger ,
Is there any way we can keep a file with header only and append it with the data parsed in the columns?

[Badger]

Not that I know of.

[karnamonkster (Karan)]

Hey @Badger,

I was doing a CSV file output for say every hour using the format

path => "mycsv-%{fielddate}.csv"

Where fielddate is YYYYMMddhh format.
Creates the files for every hour as expected, but the header is only on the first file created and the rest are without header.
Any thing you could point out why is this behavior?