Updating mapping

Elastic Stack Elasticsearch
#1

I'm sorry for yet another noob asking about this, but this topic is confusing me and i didn't find a good solution yet :frowning:

I have been logging my public FTP server for a while now without problems, but recently added some nginx data to the same index (logstash-*).

Since I first tested in a separate database without any problems, i now added all data to the live index.

Problem:
There is a conflict with one of the fields. 'field:bytes' is stored as 'type:long' and in some cases as 'type:integer'

Expected result:
'type:integer' for all fields.

I guess i can do this with a PUT command, but i cannot seem to fully understand the right thing, and now i am scared of messing up my existing data. Could someone help me in the right direction ?

curl -X GET 'http://localhost:9200/logstash*/_mapping/nginx/field/bytes?pretty=true'
result:
"logstash-2015.10.30" : {
"mappings" : {
"nginx" : {
"bytes" : {
"full_name" : "bytes",
"mapping" : {
"bytes" : {
"type" : "long"
}
}
}
}
}
}

#2

I guess you are using ES 2.x. Since ES 2.0, you cannot have the same field name with different mappings across different types in the same index. In you case, the mapping type of bytes field in your FTP type is different from in nginx type.

So, you can either put nginx data into a different index or add a prefix to you bytes field, e.g., nginx-bytes. You won't be able to change to mapping type of existing data in your index.

#3

You need to add a template of that mapping, so that any new indices created use the same mapping with the field set. Then split your logs into different indices, mixing types like you have causes problems like you have.

#4

Thanks for the replies guy's.

Yeah this explains why it worked in the test-setup but not in live.

However when i had separate indices i had problems with geo-ip showing up as type:double, as i use geo-ip both on the ftp and on the nginx. preventing it from working correctly.

  • I would prefer separate indices for performance reasons, but what to do with the geo-ip ?
  • Could i somehow delete all type:nginx data out of my current index ?
#5

Like I said, you need to adapt the existing template for the new index.
Otherwise, just call one index logstash-nginx- and one logstash-ftp- then the same template will aply to both.

#6