Forcing Types in Elasticsearch

[lisadeng]

I'm using the http_poller plugin right now to monitor websites.

I noticed that when I query a website that refuses a connection, it doesn't give me a status code field in the error log.
Right now I'm trying to visualize all the downed websites by status code, so it's important that even websites that refuse connections should provide some form of a status code.

I decided that in the filter, I'd do this:
if ![http_poller_metadata][code] {
mutate {
add_field => {"[http_poller_metadata][code]" => 500}
}
}

This seems to work well, since on Kibana, everything looks fine, (yay I got the nested fields to update nicely!), and it even shows the type as a long. The default has always been a long, and everything was working fine.

However, if I open up the JSON:
The original field:
http_poller_metadata.code: 200

Mine:
http_poller_metadata.code: "500"

This causes some indexing errors.
I was reading this article: https://www.elastic.co/blog/little-logstash-lessons-part-using-grok-mutate-type-data
But it more or less just explains the problem and now how to solve it.

Any tips? I can't figure out how to force types in Elasticsearch...

[magnusbaeck (Magnus Bäck)]

Two options (not mutually exclusive):

• Use a separate mutate filter to convert [http_poller_metadata][code] into an integer (use the convert option).
• Use an Elasticsearch index template to force the field as an integer.

[lisadeng]

Hey Magnus, I tried both, and it didn't work.

By that I mean, I put an index template and specified that the status code would be of type long.
First, I also simplified the log fields in the filter{} so the index template putting would be easier for me.
I took out the http_poller nested fields and just renamed them, so what used to be http_poller_metadata.code is now just status_code.

curl -XPUT http://localhost:9200/_template/keepalive -d '
{
"template": "keepalive-*",
"mappings":{
"website_healthcheck": {
"properties": {
"@host": {"type":"text"},
"@timestamp": {"type":"date"},
"@version": {"type":"text"},
"http_error":{"type":"text"},
"message":{"type":"text"},
"response_message":{"type":"text"},
"runtime":{"type":"long"},
"status_code":{"type":"long"},
"tags":{"type":"text"},
"times_retried":{"type":"long"},
"url":{"type":"text"}
}
}
}
}
'

Kibana recognizes that status code is number, but the JSON STILL has quotes
around the status code,
so it's still status_code:"500".

Currently, I'm no longer getting a 'uninvert' index error. However, over time I noticed that I get errors on 'shards failing', which I suspect has something to do with the fact that I'm running aggregations on a field that is supposedly a number, but shows up as a string in the JSON.

It's difficult to replicate this error it seems because I think it might only appear once I have many logs in an index.

[magnusbaeck (Magnus Bäck)]

Kibana recognizes that status code is number, but the JSON STILL has quotes
around the status code,
so it's still status_code:"500".

Yes, that's expected. Configuring the mapping of a field in an index template just affects how the field is mapped. The original documents are left untouched. What do your filters look like?

[lisadeng]

Hey Magnus, I think I made a mistake. Perhaps I accidentally looked at an older log (my logs are coming in every 30 seconds), but using the convert option ended up working.

filter {
if [http_poller_metadata] or [http_request_failure] {
mutate {
add_field => {"@host" => "%{http_poller_metadata[name]}"}
}
# Classify requests that can't connect or have an unexpected response code
if [http_request_failure] or [http_poller_metadata][code] !=200 {
mutate {
add_tag => "bad_request"
}
}

If no existing status code, add one for classification purposes

For example, request failures do not usually have error codes

if ![http_poller_metadata][code]{
mutate {
add_field => {
"[http_poller_metadata][code]" => 500
}
}
mutate {
convert => {"[http_poller_metadata][code]" => "integer"}
}
}
}

This actually ended up working for me when I tried it again this morning.
So, it now the number shows up in the JSON without the quotes.
I guess when I did it earlier I looked at the wrong logs, or was really confused...
Thanks again for your help!