How to conditionaly set type of event


(Marin Vila) #1

I have more than one type of document per file.
I want to set them apart with different types.


This topic lead me to writing :

replace => { "type" => "%{log_type}"}

"log_type" was added in grok and separates names few events.

What I get is:

a warning

[2017-12-01T09:52:28,089][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.12.01", :_type=>"My_log_type", :_routing=>nil}, #LogStash::Event:0x5acfcebf], :response=>{"index"=>{"_index"=>"logstash-2017.12.01", "_type"=>"My_log_type", "_id"=>"z7Z9EWABqNc_5GGY1bF0", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [logstash-2017.12.01] as the final mapping would have more than 1 type: [%{log_type}, My_log_type]"}}}}

And in elastic my event has _type and type "%{log_type}" as an actual value.


(Christian Dahlqvist) #2

Elasticsearch 6.0 allows only one type per index, so setting different types will no longer work.


(Marin Vila) #3

Should I then set different indexes from this file. Or just have one file generate one type with all fields that are in all types of events?


(Christian Dahlqvist) #4

That depends on y our data. If you do not have any mapping conflicts, you can store it all in one index. If you have mapping conflicts you may need to split data into multiple indices.


(Marin Vila) #5

Thanks!

I will work with different indexes just for separation sake.


(Christian Dahlqvist) #6

If you go down that route, make sure you do not end up with a large number of very small indices/shards, as this can be very inefficient.


(Marin Vila) #7

I will keep that in mind.
It probably wont go far.

In my case indices will be weekly with 3 week retention and 8 types at most.
I will review this in few weeks.

Thanks for your guidance.


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.