Duplicate fields with different names

Elastic Stack Beats
1

I was working on my filebeat indexes in a local install of Elastic stack 7.17. While trying to sovle a problem, I added some aliases to these indexes, and then regretted it, so deleted the aliases.

At some point the data fields changed, and duplicated with *.keyword. So url.path became url.path.keyword. But not for all of my data. Some filebeat indexes remained unchanged. The issue is that now my historic data shows up as either url.path or url.path.keyword, but not both.

Is there a way to reclaim the original field (url.path, source.ip, agent.hostname), instead of having fractured data ur.path.keyword, url.path, agent.hostname, agent.hostname.keyword, source.ip, source.ip.keyword?

2

From Elastic Search to Elasticsearch

3

It sounds like you want to create a new index with the mappings you want, and then reindex into it.

In the near term, it's a pain, but you could probably do a boolean query to pull the contents of both fields.

4

Thanks Kathleen. I think you're right. What would be the best strategy here?

The index template (coming from filebeat) would need to be adjusted, as it's defining the indices, right? I suppose I have two options:

  1. delete the index template and regenerate the template with filebeat setup -e
  2. a better way? Can I modify the mappings of an index template without deleting it? How do new indices it creates pick up the changes? do I need to run a special command after updating it?
5

From Elasticsearch to Beats

6

Added filebeat

7

Hi there @FF_E4 - You can modify an index template but it will only be good for new indices - indices created using the template before it was modified will not pick up the changes.

I'm re-classifying this issue to the Filebeats area, hopefully we can get someone from that team to weigh in if there are better built-in options. Thanks!

1 Like