How to conditionaly set type of event

[rinmalavi (Marin Vila)]

I have more than one type of document per file.
I want to set them apart with different types.

How to conditionally set the type?

I understand thanks. I set type following your example and I end up in ES with a field type (in _source) and a field _type in the meta data. They both have the same value, the one I used when setting typein logstash. Should I deduce from this that the field called type in logstash is treated in a special way by the elasticsearch output plugin - by assuming that its value should also be used to populate _type before sending to ES ?

This topic lead me to writing :

replace => { "type" => "%{log_type}"}

"log_type" was added in grok and separates names few events.

What I get is:

a warning

[2017-12-01T09:52:28,089][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.12.01", :_type=>"My_log_type", :_routing=>nil}, #LogStash::Event:0x5acfcebf], :response=>{"index"=>{"_index"=>"logstash-2017.12.01", "_type"=>"My_log_type", "_id"=>"z7Z9EWABqNc_5GGY1bF0", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [logstash-2017.12.01] as the final mapping would have more than 1 type: [%{log_type}, My_log_type]"}}}}

And in elastic my event has _type and type "%{log_type}" as an actual value.

[Christian_Dahlqvist]

Elasticsearch 6.0 allows only one type per index, so setting different types will no longer work.

[rinmalavi (Marin Vila)]

Should I then set different indexes from this file. Or just have one file generate one type with all fields that are in all types of events?

[Christian_Dahlqvist]

That depends on y our data. If you do not have any mapping conflicts, you can store it all in one index. If you have mapping conflicts you may need to split data into multiple indices.

[rinmalavi (Marin Vila)]

Thanks!

I will work with different indexes just for separation sake.

[Christian_Dahlqvist]

If you go down that route, make sure you do not end up with a large number of very small indices/shards, as this can be very inefficient.

[rinmalavi (Marin Vila)]

I will keep that in mind.
It probably wont go far.

In my case indices will be weekly with 3 week retention and 8 types at most.
I will review this in few weeks.

Thanks for your guidance.