# Creating and using variables across the logstash config **URL:** \ **Category:** Logstash\ **Created:** [November 22, 2016, 8:16am UTC](https://discuss.elastic.co/t/creating-and-using-variables-across-the-logstash-config/66823 "2016-11-22T08:16:09Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![muhamadli302](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/32.png) [@muhamadli302](https://discuss.elastic.co/u/muhamadli302)\ **Post date:** [November 22, 2016, 8:16am UTC](https://discuss.elastic.co/t/creating-and-using-variables-across-the-logstash-config/66823/1 "2016-11-22T08:16:09Z")
Hi, I'm using LS 2.2, reading a CSV of events. I want it to be completely automatic, so in order to create a meaningful index name, I tried to "implant" the name I want to use, and then parse it and use it in the output. Here is my config file: ``` `file { beats { port => 5044 } } filter { csv { separator => "," columns => ["column1", "column2".....] } if [column1] == "index_name" { grok { match => {"column2" => ["%{DATA:index_name}"]}} } } # more filters that are not related... output { elasticsearch { index => "%{index_name}" hosts => [array of ips] } }` ``` What happened with this config is that I got an index with the name %{index\_name}, so clearly I did something wrong. Just to clarify, the info I implanted inside the csv is just a line that goes like this: index\_name,ed-info (where ed-info is the actual name I want to give the index...) Thank you for your help! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 22, 2016, 9:20pm UTC](https://discuss.elastic.co/t/creating-and-using-variables-across-the-logstash-config/66823/2 "2016-11-22T21:20:20Z")
The `index_name` field was apparently never set. Please show an example message (use a `stdout { codec => rubydebug }` output). ---
**Author:** ![muhamadli302](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/32.png) [@muhamadli302](https://discuss.elastic.co/u/muhamadli302)\ **Post date:** [November 27, 2016, 2:10pm UTC](https://discuss.elastic.co/t/creating-and-using-variables-across-the-logstash-config/66823/3 "2016-11-27T14:10:34Z")
``` { "@version" => "1", "@timestamp" => "2016-11-27T13:38:40.919Z", "beat" => { "hostname" => "CNC", "name" => "CNC" }, "count" => "1", "fields" => nil, "input_type" => "log", "offset" => 8738, "source" => "C:\\logs\\beats_test.csv", "type" => "log", "host" => "CNC", "tags" => [ [0] "_dataparsefailure" ], "extracted.ext" => nil } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 27, 2016, 9:06pm UTC](https://discuss.elastic.co/t/creating-and-using-variables-across-the-logstash-config/66823/4 "2016-11-27T21:06:55Z")
Indeed, there is no `index_name` field. Unless you tell us what an input event looks like (i.e. what data your filters are trying to process) and what other filters you have it's very hard to help. ---
**Author:** ![muhamadli302](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/32.png) [@muhamadli302](https://discuss.elastic.co/u/muhamadli302)\ **Post date:** [November 29, 2016, 3:22pm UTC](https://discuss.elastic.co/t/creating-and-using-variables-across-the-logstash-config/66823/5 "2016-11-29T15:22:25Z")
Hi, I'm using ELK-Forensics ([https://github.com/cvandeplas/ELK-forensics](https://github.com/cvandeplas/ELK-forensics)) with a few minor changes: ``` input { tcp { type => "l2tcsv" port => 18005 } } filter { if [type] == "l2tcsv" { csv { separator => "," quote_char => "ยช" # workaround: don't use a quote character as " gives issues if the field contains a " columns => ["date","time","timezone","macb","source","sourcetype","eventtype","user","host","short","desc","version","filename","inode","notes","format","extra"] } if [date] == "date" { drop {} # drop the first line that contains the column names } if [date] == "index_name" { grok { match => {"time" => ["%{DATA:index_name}"]}} } mutate { merge => ["date", "time"] } # merge and join need to be in separate mutates mutate { merge => ["date", "timezone"] } # merge and join need to be in separate mutates mutate { join => ["date", " "] } # merge and join need to be in separate mutates date { match => ["date", "MM/dd/YYYY HH:mm:ss z"] } # extract macb info if ("M" in [macb]) { mutate { add_tag => ["modified"] } } if ("A" in [macb]) { mutate { add_tag => ["accessed"] } } if ("C" in [macb]) { mutate { add_tag => ["changed"] } } if ("B" in [macb]) { mutate { add_tag => ["birth"] } } # Extract filenames if [source] == "FILE" { grok { break_on_match => false match => ["desc", "(:(?/.*?))?$", "extracted.path", "(?[^/]+?)?$", "extracted.filename", "((\.(?[^./]+))?)?$" ] } } if [source] == "META" { grok { break_on_match => false match => ["filename", "(:(?/.*?))?$", "extracted.path", "(?[^/]+?)?$", "extracted.filename", "((\.(?[^./]+))?)?$" ] } } # Extract urls if [source] == "WEBHIST" { grok { match => ["desc", "Location: (?.*?)[$]"] } } mutate { convert => ["inode", "integer", "version", "integer"] lowercase => ["extracted.ext"] remove_field => ["message", "short", "date", "time", "timezone"] } } } output { if [type] == "l2tcsv" { elasticsearch { index => "%{index_name}" hosts =>[array of ips] } } } ``` The data I'm processing is Psort CSV output ([https://github.com/log2timeline/plaso/wiki](https://github.com/log2timeline/plaso/wiki)), and I before I'm sending it to Logstash, I'm adding the line I described before at the beginning of the document. If there's any other information you need, please let me know. Thank you for your help! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 29, 2016, 8:42pm UTC](https://discuss.elastic.co/t/creating-and-using-variables-across-the-logstash-config/66823/6 "2016-11-29T20:42:18Z")
I asked for two things, an example input event and your filter configuration. You have so far only provided your filter configuration. What I can say so far is that the `index_name` field will only be populated if the first column of the CSV is "index\_name" _and_ the event type is 12tcsv. That doesn't appear to be the case. In the sample output event you provided earlier the type is "logs", but I also note that that event entered Logstash via a beats input, but there's no beats input in the configuration you posted. ---
**Author:** ![muhamadli302](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/32.png) [@muhamadli302](https://discuss.elastic.co/u/muhamadli302)\ **Post date:** [December 1, 2016, 9:03am UTC](https://discuss.elastic.co/t/creating-and-using-variables-across-the-logstash-config/66823/7 "2016-12-01T09:03:54Z")
I'm sorry, I didn't notice I posted the wrong version of the logstash config. I will make some order and transfer you the correct configs and input. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 29, 2016, 9:03am UTC](https://discuss.elastic.co/t/creating-and-using-variables-across-the-logstash-config/66823/8 "2016-12-29T09:03:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.