Declaring variables in output.conf and using those variable values in conditional statements

Hi,

I have following configuration in the output.conf

output {

	if (([column1] == "valc11" and [column2] == "valc12") or ([column3] == "valc13" and [column1] == "valc11") or ([column2] == "valc12" and [column3] == "valc13")) {
            file {
                    path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
                    codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
            }
    }
	
	if (([column1] == "valc21" and [column2] == "valc22") or ([column3] == "valc23" and [column1] == "valc21") or ([column2] == "valc22" and [column3] == "valc23")) {
            file {
                     path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
                    codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
            }
    }
	
	if (([column1] == "valc31" and [column2] == "valc32") or ([column3] == "valc33" and [column1] == "valc31") or ([column2] == "valc32" and [column3] == "valc33")) {
            file {
                    path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
                    codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
            }
    }

}

where the match is made on the above condition , i have 14 such conditions in the output.conf and when the condition is matched then the values are stored in the csv as per the codec.

The code for the condition and file are repetitive and i wanted to avoid this scenario by declaring the values of column1, column2 and column3 as comma seperated values (variables) and loop over as a condition and replace the value with the index in the if condition for the column values in output.conf.

Could you please help me with providing an example of how this can be achieved in the output.conf file and i avoid the code repetition.

Thanks in advance
Navin

I think you could solve this by introducing tags in your input section.
Eg.

input {
	if (([column1] == "valc11" and [column2] == "valc12") or ([column3] == "valc13" and [column1] == "valc11") or ([column2] == "valc12" and [column3] == "valc13")) {
            file {
					id => "file1_ingestion"
                    path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
                    codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
					add_tag => ["file1_ingestion"]
            }
    }
	
	if (([column1] == "valc21" and [column2] == "valc22") or ([column3] == "valc23" and [column1] == "valc21") or ([column2] == "valc22" and [column3] == "valc23")) {
            file {
			        id => "file2_ingestion"
                    path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
                    codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
					add_tag => ["file2_ingestion"]
            }
    }
	
	if (([column1] == "valc31" and [column2] == "valc32") or ([column3] == "valc33" and [column1] == "valc31") or ([column2] == "valc32" and [column3] == "valc33")) {
            file {
			        id => "file3_ingestion"
                    path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
                    codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
					add_tag => ["file3_ingestion"]
            }
    }
}

output {
	if "file1_ingestion" in [tags] {
		#ur stuff 
	}
	if "file2_ingestion" in [tags] {
		#ur stuff 
	}
	if "file3_ingestion" in [tags] {
		#ur stuff 
	}
}	

Is this what you want to accomplish?

Hi Charlie,

No i want to declare the variables as comma separated variables and iterate over them in output.conf file like we do in other programming languages.

i.e Eg : column1Vals = valc11, valc21, valc31 etc
column2Vals = valc21, valc22, valc32 etc
column3Vals = valc31, valc32, valc33 etc

Then instead of writing the if conditions multiple times just to check the match conditions of the values.

I would like to iterate over the index of values for all three columns and if the condition matches then output to a file

Eg : Run a loop for 'i' number of iterations, i.e., it should start from first element position to the last element position in the csv string.
and run a condition to check for i'th value

if (([column1] == val at ith place of column1Vals and [column2] == val at ith place of column2Vals) or ([column3] == val at ith place of column3Vals and [column1] == val at ith place of column1Vals) or ([column2] == val at ith place of column2Vals and [column3] == val at ith place of column3Vals))

something like this.
please let me know if this is possible to achieve and also what do i follow to achieve or kindly provide solution for this.

Thanks,
Navin

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.