No, the base conditional. Because I don't understand what this is validating:
if [xxx]{}
Is it validating if the field exists, if the fields is empty. What is it validating?
It is a Rubism, a missing field returns a nil and Ruby interprets that as false
In the config if [xx] is the same as if event.get("[xx]") in Ruby - if event.get("[xx]") is not nil or false then the branch is taken.
You must take the false value into account.
Most folks are getting a log line or similar of known(ish) content and so, after grokking etc., can use this Ruby side effect to test for missing fields.
In your case the documents are coming from ES, are fully formed (no chopping up into fields needed) and are of unknown content to you.
I suggest that you spend some time in Kibana analysing the documents in the index you are querying and find some "signature" that is common to all docs of-interest or common to all docs to exclude.
Then query by this signature to limit the docs retrieved - bang, no conditional required.
I think I understood. Tomorrow I'll try with this condition to see if I have different results. And I'll try as well to use Kibana to try and find the signature you proposed.
As soon as I have news, I'll let you know.
Thank you
1 Like
Hello,
I've just tested your solution but no file was created 
I've tried as well to keep looking on the logs file every field, but it's like searching a needle in a haystack.
Wait a minute, all I have said before relates to the LS behaviour of newer versions 5+
I guess you can't use a newer version of Logstash because ES is old.
Maybe your best bet is to post process the file with some python or similar - copy the correct lines to a new file.
Yap, I'm using Logstash 2.1.1 to retrieve data from Elasticsearch 1.4.4.
That was indeed my last resort. Make a script to do the post processing of the file. 
Thanks everyone for your suggestions and your time 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.