if (([column1] == "valc11" and [column2] == "valc12") or ([column3] == "valc13" and [column1] == "valc11") or ([column2] == "valc12" and [column3] == "valc13")) {
file {
path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
}
}
if (([column1] == "valc21" and [column2] == "valc22") or ([column3] == "valc23" and [column1] == "valc21") or ([column2] == "valc22" and [column3] == "valc23")) {
file {
path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
}
}
if (([column1] == "valc31" and [column2] == "valc32") or ([column3] == "valc33" and [column1] == "valc31") or ([column2] == "valc32" and [column3] == "valc33")) {
file {
path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
}
}
}
where the match is made on the above condition , i have 14 such conditions in the output.conf and when the condition is matched then the values are stored in the csv as per the codec.
The code for the condition and file are repetitive and i wanted to avoid this scenario by declaring the values of column1, column2 and column3 as comma seperated values (variables) and loop over as a condition and replace the value with the index in the if condition for the column values in output.conf.
Could you please help me with providing an example of how this can be achieved in the output.conf file and i avoid the code repetition.
I think you could solve this by introducing tags in your input section.
Eg.
input {
if (([column1] == "valc11" and [column2] == "valc12") or ([column3] == "valc13" and [column1] == "valc11") or ([column2] == "valc12" and [column3] == "valc13")) {
file {
id => "file1_ingestion"
path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
add_tag => ["file1_ingestion"]
}
}
if (([column1] == "valc21" and [column2] == "valc22") or ([column3] == "valc23" and [column1] == "valc21") or ([column2] == "valc22" and [column3] == "valc23")) {
file {
id => "file2_ingestion"
path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
add_tag => ["file2_ingestion"]
}
}
if (([column1] == "valc31" and [column2] == "valc32") or ([column3] == "valc33" and [column1] == "valc31") or ([column2] == "valc32" and [column3] == "valc33")) {
file {
id => "file3_ingestion"
path => "/opt/logstashcsvs/logstash-%{+YYYY-MM-DD-HH-mm}.csv"
codec => line { format => "%{column1}|%{column2}|%{column3}|%{column4}|%{column5}|%{column6}|%{column7}|%{column8}|%{column9}|%{column10}"}
add_tag => ["file3_ingestion"]
}
}
}
output {
if "file1_ingestion" in [tags] {
#ur stuff
}
if "file2_ingestion" in [tags] {
#ur stuff
}
if "file3_ingestion" in [tags] {
#ur stuff
}
}
Then instead of writing the if conditions multiple times just to check the match conditions of the values.
I would like to iterate over the index of values for all three columns and if the condition matches then output to a file
Eg : Run a loop for 'i' number of iterations, i.e., it should start from first element position to the last element position in the csv string.
and run a condition to check for i'th value
if (([column1] == val at ith place of column1Vals and [column2] == val at ith place of column2Vals) or ([column3] == val at ith place of column3Vals and [column1] == val at ith place of column1Vals) or ([column2] == val at ith place of column2Vals and [column3] == val at ith place of column3Vals))
something like this.
please let me know if this is possible to achieve and also what do i follow to achieve or kindly provide solution for this.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.