I have a very big unstructured file. Firstly, I want to parse all lines that starts with ABC or CDE and store them as one document in Elasticsearch. One file should be one document in index, so @message should look like all ABC + CDE lines.
Secondly, I also have "ignore list", like lines starting with empty space, "Timing", "----" etc., and basicaly ALL that is left ater parse is complete, I want to store in new key value (data). Final result should look something like:
"_source" : {
"message" : "ABC V4.1.2 MODEL,CDE: 0 uri: xxxx.xxx"
"data" : "ALL LINES THAT ARE NOT IN IGNORE LIST"
}
This is my logstash conf file for the first part, but for some reason it is not processing anything because index is empty, and I can't see any error in logs, probably because I'm not saving it properly.
input {
file {
path => "/etc/logstash/files/*"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
if ([message] !~ "^ABC"){
drop{}
}
else if ([message] !~ "^CDE"){
drop{}
}
}
output{
elasticsearch {
hosts => ["XXX"]
index => "index1"
}
}
When I add 1 ef expression it works, but when I add second one , it doesn't process anything. Any help? Thank you