Removing whitespace from column in csv import

Elastic Stack Logstash
1

I am trying to ingest csv file into ES. The csv file contains undetermined column names in header. I would to eliminate white space from column names. Since I don't know the column names I can't manually supply column names.
e.g. "column name with space" should be renamed to "column_name_with_space". Can I achieve this in logstash?

Thanks

2

You could do it with gsub - https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub

3

@warkolm I don't I can use that solution as I will not know the column name in advance. The data in csv files can have different columns.

4

You might need some custom Ruby code to manipulate ad-hoc field names after splitting the original message with CSV.

E.g.

## CSV filter + other stuff
filter {
    ruby  {
        code => "
                hash = event.to_hash
                hash.each{|k,v| event.set(k.gsub(/ /,'_'),v) && event.remove(k) if k.include?(' ')  }
            "
    }
}

It's kinda hacky but gets the job done.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.