CSV Filter Column

Elastic Stack Logstash
1

Hi,
Is it possible to expand column from a variable ?

EnvironmentFile=-/etc/logstash/env
SCALL='"timestamp","id","app","username"'

Filter
csv {
separator => "|"
columns => [ "${SCALL}"]
}

2

Well, it will do the substituion of $SCALL, but columns is still an array containing a single string, so it just sets the name of column 1 to '"timestamp","id","app","username"'.

3

Is there any way to split the strings ... this is what i am looking for

Instead of this approach

filter {
   if "a" in [message] {
           csv {
                     separator => "|"
                     columns => [ "a","b","c","d", .......,"z"]
                 }
  } else if "1" in [message]
           csv {
                     separator => "|"
                     columns => [ "1","2","3","4", .........,"99"]
                  }
   }
}

can i substitute the columns using a variable

EnvironmentFile=-/etc/logstash/env
ALPHA='"a","b","c",...........,"z"'
NUM='"1","2","3",..........,"99"'

    filter {
       if "a" in [message] {
               csv {
                         separator => "|"
                         columns => [ "${APLHA}"]
                     }
      } else if "1" in [message]
               csv {
                         separator => "|"
                         columns => [ "${NUM}"]
                      }
       }
    }

Well trying to avoid the repetitive column entries, since i have around 20 if conditions ...

4

you can define environment variables in logstash and use them in filter part. Check this for reference:

https://www.elastic.co/guide/en/logstash/current/environment-variables.html

5

Tried, won't work.

CSV filter  [ "${ALPHA}"]  takes it as the first column of an array 

    ---- Partial OUTPUT ---

    {
            _index" : "test",
            "_type" : "doc",
            "_id" : "1GLnAWIBad5e0Ecyh3B7",
            "_score" : 1.0,
            "_source" : {
    "a\","b\","c\","d\","e\": "2018-03-05 21:29:31 0500"
              "column2" : "a",
              "column3" : "b",
              "column4" : "c",
              "column5" : "d",
              "column6" : "e",
              "message" : "2018-03-05 21:29:31 -0500|a|b|c|d|e",
              "@timestamp" => 2018-03-07T18:06:11.049Z,
                "@version" => "1"
    }
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.