How to Convert response-time from string to number in Kibana 5.1

Elastic Stack Kibana
#1

Hello,

Log data sending response-time in string format. So it is not showing in aggregations unless it's in number format. How to convert string to number in Painless script Kibana or any other alternative options? Please advise.

Thanks,
Chaitanya.

1 Like
#2

Kibana can with Scripted fields but that is not really converting the field , it is creating a new dynamic field

You need to actually send it to Elastic as a Int and Set the Mapping of your index so the field is a int not string

for example if you groking the field "%{NUMER:responseTime:int} adding the int will make it an integer (you can use float if it is a decimal) or you can do it with the "MUTATE" filter to convert the field from string to whatever

for mappings, consult the Mapping documentation - Good to define your data
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

you will need to re-index your data to change the mapping (or delete your data and send it in again)
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

finally , and probably the most CPU intensive way
https://www.elastic.co/guide/en/kibana/current/scripted-fields.html

but I would recommend not doing that it will be slower and higher cpu.

I would Just make sure your data gets sent in to ELK in the right data format and have your mappings set appropriately.

#3

Hi Ed,

Thank you !!

Your options would work but our data is coming from AWS CloudWatch logs and I don't have control on Elasticsearch. Can I make string to int in painless script in Kibana? If so can you provide me an example.

Thanks,
Chaitanya

#4

Thinking about it a little bit more I think I over thought your issue.

You will have to change the Mappings of your index. If you mapping data says "response-time" is a int , Elastic will try to convert the "string" to int. and if your mapping says it is a "String" it will convert int's to strings. So even if you write a painless script it will remain the same type

Changing the mapping will help you on all future data, old history will have to be re-indexed to pickup the new changes but then you don't have to write a script to run all the time you get new data

If you provide me your index's current mapping I can help you make the change
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html

#5

Hi Ed, Thank you for your time and advise.

In index, getting response-time as string. Please advise how to mapping it to int.
{
"cwl-2017.03.21": {
"mappings": {
"retail-rtmstack-dev-logsgroup": {
"properties": {
"@id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@log_group": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@log_stream": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@owner": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@timestamp": {
"type": "date"
},
"response-time": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},

Thanks,
Chaitanya

#6

just change the type for your field from text to "Integer" or if it is a float to float
https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html

Btw, now you have to make this file a template , its bacically just adding the "template matching" and any additional information you need when the index is created

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

  "template" : "<indexname>*",
  "mappings" : {

and then load your mapping back in to the index
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html

#7

Hi Ed,

Thank you for the details. I gone through the links but I am missing something. If you don't mind, can you provide step by step with the index data.

Thanks,
Chaitanya.

#8

I am not able to write a full tutorial on this for you, you may want to try googling are reading the rest of the docs.
It is pretty easy to do but it is a wide subject

I did a quick google and saw this

or maybe this too

#9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.