How to conditionaly set type of event

Elastic Stack Logstash
1

I have more than one type of document per file.
I want to set them apart with different types.

This topic lead me to writing :

replace => { "type" => "%{log_type}"}

"log_type" was added in grok and separates names few events.

What I get is:

a warning

[2017-12-01T09:52:28,089][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.12.01", :_type=>"My_log_type", :_routing=>nil}, #LogStash::Event:0x5acfcebf], :response=>{"index"=>{"_index"=>"logstash-2017.12.01", "_type"=>"My_log_type", "_id"=>"z7Z9EWABqNc_5GGY1bF0", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [logstash-2017.12.01] as the final mapping would have more than 1 type: [%{log_type}, My_log_type]"}}}}

And in elastic my event has _type and type "%{log_type}" as an actual value.

2

Elasticsearch 6.0 allows only one type per index, so setting different types will no longer work.

3

Should I then set different indexes from this file. Or just have one file generate one type with all fields that are in all types of events?

4

That depends on y our data. If you do not have any mapping conflicts, you can store it all in one index. If you have mapping conflicts you may need to split data into multiple indices.

5

Thanks!

I will work with different indexes just for separation sake.

6

If you go down that route, make sure you do not end up with a large number of very small indices/shards, as this can be very inefficient.

7

I will keep that in mind.
It probably wont go far.

In my case indices will be weekly with 3 week retention and 8 types at most.
I will review this in few weeks.

Thanks for your guidance.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.