Adding .raw fields to existing data

Elastic Stack Logstash
1

Hi all,

I created a cluster where I host my logs. I have around 8 index patterns and ~40 million documents currently. I noticed that my fields are being analyzed and broken down (so foo-bar is broken down to "foo" and "bar" in visualizations for example). So I wanted to solve it by adding .raw fields, like in the default logstash-* pattern. I added to /_template endpoint some templates which I copied from logstash-* default template along with the necessary changes. Now to move forward, I want to make all my documents add the .raw fields (to all the strings) as well as future documents (currently my redis holds everything and I really want to get this over with so I don't lose data. I tried to reindex by stopping my logstash service and running manually with a configuration file but it's really slow and logstash crashes. Can anyone help? (the real ip was replaced with 1.1.1.1)

input
{
elasticsearch
{
hosts => ["1.1.1.1"]
index => "*"
size => 2000
scroll => "5m"
docinfo => true
}
}

output {
elasticsearch {
hosts => ["1.1.1.1"]
index => "r.%{[@metadata][_index]}"
document_type => "%{[@metadata][_type]}"
document_id => "%{[@metadata][_id]}"
}

2

Which part do you want help on exactly?

That config looks fine.

3

First, thanks for your reply! First questions is - is this the correct way? For example, is it not possible to just tell an index to reindex itself? for example something like: curl -XPUT 'localhost:9200/index/_reindex'

Secondly, if I use this method, how can I speed it up? It's very slow. And thirdly, if I use this method and rename my indices, I then have to play them back to rename them again? Or can I just rename and index? Or should I use aliases?

I hope it's clearer now.

Thanks again!

4

In 2.3, yes Reindex API | Elasticsearch Guide [2.3] | Elastic

Add more resources to ES :slight_smile:

Use aliases, you cannot rename indices.