Creating multiple fields from single field

Elastic Stack Logstash
1

I'm using the csv input filter to ingest csv files into Elasticsearch. One of the fields, "CustomFields" contains key/value pairs like this:

CustomFields
"""Author=Jules J. Toner, S.J."";Binding=Paper;""Publication Date=2011"";""Pages=337 pages"";ISBN=9780912422428"

Instead of ingesting CustomFields as a single string field, I'd like to create a new top level field for each key/value pair in this field like this

I used mutate/split filter to get this field to look like this:

     "ProductCustomFields" => [
    [0] "\"Author=Jules J. Toner, S.J.\"",
    [1] "Binding=Paper",
    [2] "\"Publication Date=2011\"",
    [3] "\"Pages=337 pages\"",
    [4] "ISBN=9780912422428"
],

However, what I really want is this:

     "ProductCustomFields" => [
    [0] "\"Author=Jules J. Toner, S.J.\"",
    [1] "Binding=Paper",
    [2] "\"Publication Date=2011\"",
    [3] "\"Pages=337 pages\"",
    [4] "ISBN=9780912422428"
],
     "Author" => "Jules J. Toner, S.J.",
     "Binding" => "Paper",
     "Publication Date" => "2011",

...

Any suggestions?

2

Have you tried applying the kv filter, possibly after cleaning up the data using a mutate filter?

3

Thanks, Christian. I was looking at the ruby filter as well (found an example online). But it looks like your suggestion is much easier to implement.

Chris

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.