Multiple csv filter in logstash not displaying header in kibana

Elastic Stack Logstash
1

Hi all,

I'm just getting into using elastic services and as a proof of concept am trying to get csv data from two files that will have two different sets of headers to kibana. The data gets there just fine but I have not header information. If I take the configuration below and remove the else if portion so that I'm only getting one csv file (the get_pods.csv if it matters) then I can get the information in Kibana and I have header fields to work with. I've tried various things I've found online with no luck. Only thing I havent tried is putting the csv's in different directories. Can anyone tell me what they see that is wrong here? Also I'm assuming the problem is logstash conf because of the behavior above but possible something I need to adjust in kibana as well....

Thanks in advance

Logstash.conf:
input {
beats {
port => 5044
}
}

filter {
    if [path] == "/Users/me/CodeBase/healthcheck/dev_getPods.csv" {
        csv {
        columns => [
            "DATE",
            "ENV",
            "NAME",
            "READY",
            "STATUS",
            "RESTARTS",
            "AGE"
        ]
        separator => ","
        }
    } else if [path] == "/Users/me/CodeBase/healthcheck/dev_identity_properties.csv" {
        csv {
        columns => [
            "DATE",
            "PROPERTY",
            "VALUE",
            "LAST_CHANGE",
            "PROPERTY",
            "OLD_VALUE"
        ]
        separator => ","
        }
    }
}
output {
        elasticsearch {
                hosts => "elasticsearch:9200"
        }
}

As requested, I've added the rubydebug and some sample output. I've swapper out some of the identifying info for something generic just fyi.

Logstash.conf:

input {
    beats {
        port => 5044
    }
}

filter {
    if [path] == "/Users/me/CodeBase/healthcheck/dev_getPods.csv" {
        csv {
            columns => ["DATE","ENV","NAME","READY","STATUS","RESTARTS","AGE"]
            separator => ","
        }
    } else if [path] == "/Users/me/CodeBase/healthcheck/dev_identity_properties.csv" {
        csv {
            columns => ["DATE","PROPERTY","VALUE","LAST_CHANGE","PROPERTY","OLD_VALUE"]
            separator => ","
        }
      }
}

output {
        stdout { codec => rubydebug }
        elasticsearch {  hosts => "elasticsearch:9200"}
}

Sample output:

logstash_1       | {
logstash_1       |     "@timestamp" => 2019-05-16T21:26:22.437Z,
logstash_1       |          "input" => {
logstash_1       |         "type" => "log"
logstash_1       |     },
logstash_1       |           "tags" => [
logstash_1       |         [0] "beats_input_codec_plain_applied"
logstash_1       |     ],
logstash_1       |     "prospector" => {
logstash_1       |         "type" => "log"
logstash_1       |     },
logstash_1       |        "message" => "2019-05-16-15:26:13,dev,inventory-develop-659b9759f-gpppm,2/2,Running,0,4d",
logstash_1       |         "fields" => {
logstash_1       |         "collection" => "dev_getPods"
logstash_1       |     },
logstash_1       |            "log" => {
logstash_1       |         "file" => {
logstash_1       |             "path" => "/Users/me/CodeBase/healthcheck/dev_getPods.csv"
logstash_1       |         }
logstash_1       |     },
logstash_1       |           "host" => {
logstash_1       |                   "id" => "someId",
logstash_1       |                 "name" => "someName",
logstash_1       |         "architecture" => "x86_64",
logstash_1       |                   "os" => {
logstash_1       |                 "name" => "Mac OS X",
logstash_1       |             "platform" => "darwin",
logstash_1       |                "build" => "17G2208",
logstash_1       |               "family" => "darwin",
logstash_1       |              "version" => "10.13.6"
logstash_1       |         }
logstash_1       |     },
logstash_1       |         "source" => "/Users/me/CodeBase/healthcheck/dev_getPods.csv",
logstash_1       |           "beat" => {
logstash_1       |             "name" => "someName",
logstash_1       |         "hostname" => "someHostName",
logstash_1       |          "version" => "6.7.0"
logstash_1       |     },
logstash_1       |         "offset" => 4811,
logstash_1       |       "@version" => "1"
logstash_1       | }

The data does make it to Kibana just no header fields.

2

Anything I can add to the question to get some traction on this?

3

Can you run with

output { stdout { codec => rubydebug } }

and show us an example of an event that does not get parsed by the filter you posted?

4

Done. Made a few edits in an attempt to keep things clear. Let me know if anything sticks out. I've been trying a few different things but just no luck so far...

5

It has "/Users/me/CodeBase/healthcheck/dev_getPods.csv" in [log][file][path] and [source], but not [path].

6

Hmm.... I'm not quite sure I follow you on that. Could you explain a little bit further for me?

7

Your configuration says

if [path] == "/Users/me/CodeBase/healthcheck/dev_getPods.csv" {
    csv {

That is testing whether the [path] field contains that value. It does not, so the csv filter is not applied. Change it to

if [log][file][path] == "/Users/me/CodeBase/healthcheck/dev_getPods.csv" {
    csv {
8

Perfect works like a charm. I didn't see that in a lot of the configurations I've been looking through as I try to learn this stack but at least know I can hunt online specifically for documentation on how that piece works. Thanks for helping me solve this you're awesome!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.