# Reindex data with new field patterns?

**URL:** <https://discuss.elastic.co/t/reindex-data-with-new-field-patterns/65439>\
**Category:** Elasticsearch\
**Created:** [November 9, 2016, 6:40am UTC](https://discuss.elastic.co/t/reindex-data-with-new-field-patterns/65439 "2016-11-09T06:40:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rogerwang](https://avatars.discourse-cdn.com/v4/letter/r/ea5d25/32.png) [@rogerwang](https://discuss.elastic.co/u/rogerwang)\
**Post date:** [November 9, 2016, 6:40am UTC](https://discuss.elastic.co/t/reindex-data-with-new-field-patterns/65439/1 "2016-11-09T06:40:33Z")

</div>

Hello,  
I have been gone trough most documents about how to reindex Elasticsearch data. However, I am still not quite satified with the result I am getting.

Question: sometimes we would like to change the logstash grok patterns and create new fields in index. I noticed that we could add new fields by change mapping. But how to add field with pattern? so that it could be generated match like logstash based on the data.

Since I could not find the way to add fields with pattern in mapping, I end up using logstash elasticsearch plugin to reindex the indices. It kind of works, but just wondering if there is any better way to do it.

2nd Question: some docs shows deleted after reindex with logstash elasticsearch plugin. why?

# curl 'localhost:9200/\_cat/indices?v' | grep syslog

health status index pri rep docs.count docs.deleted store.size pri.store.size  
yellow open syslog-2016.11.08 5 1 2852206 858040 1.2gb 1.2gb  
yellow open syslog-v2-2016.07.15 5 1 1 0 9kb 9kb  
yellow open new-syslog 5 1 1230000 90556 569.3mb 569.3mb

the new-syslog is the new index which is generated by logstash elasticsearch plugin and 90556 docs were deleted.

below is the reindexing conf file for your information:  
input {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "syslog-2016.11.08"  
size =\> 1000  
scroll =\> "5m"  
docinfo =\> true  
scan =\> true }  
}  
filter {  
mutate {  
#since all fields will be generated again, old fields are not required  
remove\_field =\> ["datastore\_latency\_to", "protocol", "@timestamp", "src\_int", "pid", "message\_system\_info", "syslog\_facility", "dst\_int", "message\_thread\_id", "message\_service\_info", "port", "syslog\_facility\_code", "syslog\_message", "tags", "ASA\_type", "device\_status", "syslog\_timestamp", "program", "@source\_host", "vmware\_warning\_msg", "message-body", "type", "message\_service", "syslog\_severity\_code", "datastore\_latency\_from", "host", "syslog\_program", "@message", "syslog\_hostname", "syslog\_severity", "syslog\_level", "message\_opID", "device\_naa", "acl", "action", "syslog\_pri", "@version"]}  
if [message] =~ "%ASA-" {  
grok {  
match =\> ["message", "%{CISCO\_TAGGED\_SYSLOG} %{GREEDYDATA:cisco\_message}"]  
add\_tag =\> "cisco-fw"  
}  
syslog\_pri { }  
grok {  
match =\> [  
"cisco\_message", "%{CISCOFW106001}",  
"cisco\_message", "%{CISCOFW106006\_106007\_106010}",  
"cisco\_message", "%{CISCOFW106014}",  
"cisco\_message", "%{CISCOFW106015}",  
"cisco\_message", "%{CISCOFW106021}",  
"cisco\_message", "%{CISCOFW106023}",  
"cisco\_message", "%{CISCOFW106100}",  
"cisco\_message", "%{CISCOFW110002}",  
"cisco\_message", "%{CISCOFW302010}",  
"cisco\_message", "%{CISCOFW302013\_302014\_302015\_302016}",  
"cisco\_message", "%{CISCOFW302020\_302021}",  
"cisco\_message", "%{CISCOFW305011}",  
"cisco\_message", "%{CISCOFW313001\_313004\_313008}",  
"cisco\_message", "%{CISCOFW313005}",  
"cisco\_message", "%{CISCOFW402117}",  
"cisco\_message", "%{CISCOFW402119}",  
"cisco\_message", "%{CISCOFW419001}",  
"cisco\_message", "%{CISCOFW419002}",  
"cisco\_message", "%{CISCOFW500004}",  
"cisco\_message", "%{CISCOFW602303\_602304}",  
"cisco\_message", "%{CISCOFW710001\_710002\_710003\_710005\_710006}",  
"cisco\_message", "%{CISCOFW713172}",  
"cisco\_message", "%{CISCOFW733100}",  
"cisco\_message", "%{GREEDYDATA:cisco\_message}"  
]  
}

```
  }

```

}  
output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "new-syslog"  
document\_type =\> "%{[@metadata][\_type]}"  
document\_id =\> "%{[@metadata][\_id]}"  
}  
}

Thanks and regards,  
Roger

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 9, 2016, 6:53am UTC](https://discuss.elastic.co/t/reindex-data-with-new-field-patterns/65439/2 "2016-11-09T06:53:30Z")

</div>

> [@rogerwang](#):
>
> But how to add field with pattern? so that it could be generated match like logstash based on the data.

Use a template - [Dynamic templates | Elasticsearch Guide [5.0] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/5.0/dynamic-templates.html)

> [@rogerwang](#):
>
> 2nd Question: some docs shows deleted after reindex with logstash elasticsearch plugin. why?

Maybe some existing ones were updated?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 7, 2016, 6:53am UTC](https://discuss.elastic.co/t/reindex-data-with-new-field-patterns/65439/3 "2016-12-07T06:53:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.