Put array elements into new fields with logstash filter

Elastic Stack Logstash
1

Hello,

I have an issue finding a solution for my problem :

My filebeat sends IIS logs to logstash, with an URIPATH field named iis.uri, it looks like :
/aaaaa/bbbbb/cccccc/dddddd/eeeeee

I copy this field i wanna keep into a new field named iis.uriSplit in a mutate-copy then i use mutate-split to get an array from this field, so i got :
iis.uri : /aaaaa/bbbbb/cccccc/dddddd/eeeeee
iis.uriSplit : (empty field here) , aaaaa , bbbbb , cccccc , dddddd , eeeeee

And now i woud like to get iis.uriSplit.1 = aaaaa , iis.uriSplit.2 = bbbbb etc.. to use them as alone fields for term aggregation in visualisations, and that's where is my problem.

I tried to use ruby filter to do so :

ruby {
 code => '
  urisplit = event.get("[iis][uriSplit]")
  urisplit.each_with_index do |hash, index|
                event.set("iis.uri.#{index}", hash)
  end
'
}
OR 
code => '
        event.get("[iis][uriSplit]").each_with_index do |hash, index|
          event.set("[iis][uri][#{index}]", hash)
        end
      '

But it tells me i have an ERROR at ' event.get(" '
Do you have a better knowledge about ruby and could you share it with me ?

Thanks for your time,

Best regards,

Louis Vince.

2

What exactly is the error?

3

Hi again,

Sorry i'm late to answer, i had to work on another subject.

I had an error on event.get cause my split separator was "/" and not "\" so i was escaping ", my bad.

So now i have my initial error :

[2020-10-07T09:51:05,710][ERROR][logstash.filters.ruby    ][main][cfe6ecf8cf1e91b0b7f95d7bc492842543ce44907b4797946370e95ae0f1abe4] Ruby exception occurred: undefined method `each_with_index' for nil:NilClass

I would like to get 5 fields maximum from this iis.uriSplit array, but my iis.uri begin with a "/" so i get an empty field i would like to delete, so i tried to slice my array but same error, undefined method ...
Last point, if my uri is only "/aaaaaa/bbbbbb" or "/aaaaaa/bbbbbb/cc/dd/ee/ff/gg" i would like to get 5 fields from a to e in each case. Can i put a simple condition on index or there will be a scope problem ?

Thanks for your time,

Louis Vince.

4

If you are getting that error then event.get("[iis][uriSplit]") is returning nil, which means the field [iis][uriSplit] does not exists. Is it possible that the field has a period in the name and is actually called [iis.uriSplit]?

5

You were right, but now i got :

[2020-10-08T15:47:30,275][WARN ][logstash.outputs.elasticsearch][main][3c05c5b1356a71bcf7bbbc8e41ba3f0c15e3c3f02d261fb1bfef89ef0679ed77] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.8.1-test-iis-2020.10.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x463e546e>], :response=>{"index"=>{"_index"=>"filebeat-7.8.1-test-iis-2020.10.08", "_type"=>"_doc", "_id"=>"tr93CHUB-QmTWdJue61y", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [iis.uri] of different type, current_type [text], merged_type [ObjectMapper]"}}}}

So i tried to had return [event] after my end but it doesn't change the warn. My index is empty.

Then i change event.set("[iis.uri.#{index}]", hash) with the return [event] and get :

[2020-10-08T16:00:33,913][WARN ][logstash.outputs.elasticsearch][main][d1fe8cf683c32a081e7311b29bcfd01d3d7cb39f123ffea9648ea861c3635ecd] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.8.1-test-iis-2020.10.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x778b043b>], :response=>{"index"=>{"_index"=>"filebeat-7.8.1-test-iis-2020.10.08", "_type"=>"_doc", "_id"=>"B7-DCHUB-QmTWdJucq-3", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Can't merge a non object mapping [iis.uri] with an object mapping [iis.uri]"}}}}

So i tried to take off return [event] and i get :

[2020-10-08T16:03:40,678][WARN ][logstash.outputs.elasticsearch][main][d1fe8cf683c32a081e7311b29bcfd01d3d7cb39f123ffea9648ea861c3635ecd] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.8.1-test-iis-2020.10.08", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x22509c0f>], :response=>{"index"=>{"_index"=>"filebeat-7.8.1-test-iis-2020.10.08", "_type"=>"_doc", "_id"=>"VL-GCHUB-QmTWdJuTK8y", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [iis.uri] of different type, current_type [text], merged_type [ObjectMapper]"}}}}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.