How can I change fields type on elasticsearch without stop my services?

[Terence_Chuen]

In last two days,I search in google and looking for information in www.elastic.co,but I still feeling puzzled.Here is my actual situation:

I learn and set up ELK services at last month.I import my logs to elasticsearch via logstash.

But I got a problem when I make visualization chart,I need sum a field to do that,but I got error:

this field type is string,need number to sum! 

A number is required to calculate the sum,this makes sense,so I try change the field type from string to number(long).

And I maked that,here is my solution:

1. post a template to elasticsearch and create new index

2. set logstash output to new index and restart it

3. reindex old index to new one

4. delete old index

Here is my question:

Can I skip the second step?

I don't want to restart logstash,because it receive data in real time and I don't want it miss some data.

If I dont't restart logstash,I can't change output to new index.

It makes me confused.

[Troy_Axthelm]

Depending on your version of logstash, when you restart logstash it will flush everything in the pipe before stopping. This means you would not lose data assuming the sender to logstash will not drop data that is not successfully received by logstash.

You could modify the existing index template and the next time and index is created using that template the change would take place. This would allow you to make the change without having to do a restart of logstash.

[Terence_Chuen]

Thank you for your prompt reply.

I useing version 5.1.

My some equipment only support sent log in real time via udp port 514,I can't use such as filebeat to do that.

When I restart logstash,it have few sec reject connection at startup.

And I only have one index,if I config daily index,it would easier to change field type.

I think,can I use Alias deal with this problem?

I have old index: A
I create new index: B
and I create alias A(this alias name same as old index A) and connect A and B

then reindex from A to B,during this time logstash output to alias A.

I have doubts:
1.can I create a alias name same as exist index?
2.the data which from logstash output to alias A will write to index A and B?

[Troy_Axthelm]

Bummer about using UDP (especially in an environment where you cannot afford to lose any logs).

Does your current output from logstash to elasticsearch have a section like:

elasticsearch {
  ...
  index => <indexname>-%{+YYYY.MM.dd}
}

Or are you just using a static index:

elasticsearch {
  ...
  index => <indexname>
}

Based on the above, I am thinking you have a single static index.

You can try aliasing a new index with the same name as your current index and it should fail on your post if there is actually an issue. If there is not an issue and the aliasing works, I have no idea what the behavior would be (whether it would go to both indices, only the new one , or only the old one). In the event the aliasing fails, I cannot think of a method to do this without a restart of logstash. I will ponder it more and get back to you if I think of something.

If you do decided to try this, I would setup a testing logstash/elasticsearch instance and try the full transition before doing anything in the production environment. Sorry I am not of better assistance on your question.

Edit: If you try it, please let us know what happens.

Update: Curiosity got to me so I tried it. You cannot have an alias name be the same as an index name. This makes sense and it fails nicely:

{"error":{"root_cause":[{"type":"invalid_alias_name_exception","reason":"Invalid alias name [], an index exists with the same name as the alias","index":""}],"type":"invalid_alias_name_exception","reason":"Invalid alias name [], an index exists with the same name as the alias","index":""},"status":400`}