How to config Logstash to set custom type

Elastic Stack Logstash
1

Hi guys! I know that this maybe a duplicated topic but I guess that the old topic is just outdated since I couldn't make it works.

In my project I use FileBeat to ship JSON logs to Logstash. I want to have different type of logs eg. client side errors, server side errors ... so I think using the Elasticsearch's type could be a good idea.
I couldn't find any concrete document about the Logstash's config file but it seems like the trick is having a type file in the JSON object and tell Logstash to use that value as the type. I tried to follow this but it didn't work. All the documents were set to log type.

I also tried this but there is no different

input {
  beats {
    port => 5044
    codec => "json"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    document_type => "%{type}"
  }
}

Hope to hear from you!

2

You need to define %{type} somewhere, is it in your json?

1 Like
3

Thank! but how to define it? my JSON will be something like this:

{"type": "server-error", "message":"random-error", ...}

But of course it didn't work!

4

Seems like that should've worked. If you temporarily replace the elasticsearch output with a stdout { codec => rubydebug } output you can see exactly what the events look like. I'd be interesting to see your Filebeat configuration too.

1 Like
5

Thank @magnusbaeck!

My FileBeat's config is just the default one except the log directory.

After debugging I find out that I just cannot use the type field in the JSON object. Maybe that name is reserved or overwritten at some point?
So changing the Logstash config to document_type => "%{MY_FIELD}" (where MY_FIELD is NOT equal type) and having the corresponding field in my JSON object will make it work.

Recently I realized that beside the _type field which I think Elasticsearch set for every document, Logstash will have another type field (without the underscore). document_type setting will affect the _type field.