Hi,
after doing the split filter, so my message was an array ok, so %{[message][0]} %{[message][1]} ... that gives me their value fine. now I have an array of length 30 for example and I start the condition if {}
example logstash.conf:
I mean, can I create a counter for example to replace %{[message][29]} with x+1 and I declare int x=0 or something like that because it's a bad idea to keep doing it manually %{[message][0]} %{[message][1]} %{[message][29]} %{[message][40]} %{[message][50]} ......
i want to do a loop from %{[message][0]} to number_of_elements i did this
filter {
mutate { split => {"message" => "|"} }
ruby {
code => "event.set('number_of_elements', event.get('message').length)"
}
ruby {
code => '
for i in number_of_elements
k = "message#{i}"
s = event.get(" ... ??????????")
if s
event.set(k, s)
end
end
'
}
}
I know that my configuration was so bad but i just want to add field just make it somthing like this
I have files that contain these logs so admin knows every field name so he will give me all possible cases so in this situation I have so many if{} statement for example in one log sometimes contains 40 pipeline "|" sometimes 50 ... so it's a bad idea to keep doing it manually %{[message][0]} %{[message][1]} %{[message][29]} %{[message][40]} %{[message][50]} ......
So i'm asking if there is any solution for this for example create a counter x can replace %{[message][0]} to %{[message][x]} %{[message][x+1]} %{[message][x+1]} and it should increment x i mean x=x+1.
I mean if it can be like this
You probably can do that with a ruby code and the ruby filter, but it is still not clear from where are you going to get the field names.
For example:
add_field => { "DateTime" => "%{[message][x]}" }
The field named DateTime is hardcoded, the same for the others fields.
The example you gave you are dynamically setting the value, but not the field name, how will you get the correct field name?
Also, does your log messages have any order? For example, if you have a message with 5 fields, and another one with 10 fields, are the fields always in the same order, like this:
If you have something like that, or can change your source file to be something like that, you can easily parse your messages using the csv filter, you would just need to set all the possible columns in the order they appear and the filter will populate them when they exist.
These are transaction logs so the developer who made the application "Gateway" writes the logs in file with the variables that made so here are the fields names comes from. So the developer who made the application who gave me the fields names.
my log messages are not in the same order
I already tried the ruby filter but didn't know how to do it, I get the message size with
but i didn't know how to initialize a variable and make it ++ i mean for example x=x+1 in a loop or ... something like that maybe can you please correct me or if there to something else
ruby {
code => '
for i in number_of_elements
k = "message#{i}"
s = event.get(" ... ??????????")
if s
event.set(k, s)
end
end
'
}
Then I think you could iterate in that array using each_with_index.
fieldName_in_ruby_code.each_with_index do |val, index|
ruby code
end
But again, where will the field names come from? How will you know that the value of message[2] in one message is different from the value of message[2] and should be in another field?
Your examples only shows iterations over the value, not the field name.
but didn't get the field DateTimeException caught while applying mutate filter {:exception=>"Invalid FieldReference: [message][%{x"}
how can i enter the value of x under %{[message][%{x}]} if you have any idea please ?
Thanks!!
It won't work this way, you need to do everything inside one ruby filter and you can't use the variables from your ruby code outside it, in the mutate filter.
The loop will only exist in inside the ruby code, so you will need to use event.set if you want to add fields.
But as I said, your field names are all static, it makes no sense to have a loop on the values if you are you going to statically set the field names, you need to provide more context about the name of the fields.
If you splitted your message into an array, you can access it using the index in any mutate filter.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.