# Reparse logs in elastic search **URL:** \ **Category:** Elasticsearch\ **Created:** [October 7, 2015, 12:17pm UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770 "2015-10-07T12:17:36Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![dlb1001](https://avatars.discourse-cdn.com/v4/letter/d/ce7236/32.png) [@dlb1001](https://discuss.elastic.co/u/dlb1001)\ **Post date:** [October 7, 2015, 12:17pm UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/1 "2015-10-07T12:17:36Z")
Hello, Is there a way to re parse log entries in elastic search? The reason I ask is I've been inputing all of our old logs into elasticsearch. I thought my groks parsing handled the majority of entries, however out of the 3 years of logs, I ran into a few months where the log entries differed slightly, just enough to mess up my grok filters. Is there a way to search for the grok parse failures in elastic search and reparse the information for the ones that messed up? Thank you ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 7, 2015, 12:25pm UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/2 "2015-10-07T12:25:14Z")
If you have saved the original log message as a field it would be easy to recreate the original logs that resulted in grok problem and push them to Logstash once again and afterwards delete the `_grokparsefailure` messages. There's nothing turn-key solution for this though. If you don't have the full original message then perhaps you can stitch it together using various fields in the message or write a custom Logstash configuration to fix the extracted logs. ---
**Author:** ![dlb1001](https://avatars.discourse-cdn.com/v4/letter/d/ce7236/32.png) [@dlb1001](https://discuss.elastic.co/u/dlb1001)\ **Post date:** [October 7, 2015, 12:46pm UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/3 "2015-10-07T12:46:48Z")
Now that I have a search which shows all entries with grok parse failures, how would I go about deleting only those entries from elasticsearch? Also there seems to be two fields for the orginal message, \_source and message which one would be better to use for outputting to a file. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 7, 2015, 1:00pm UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/4 "2015-10-07T13:00:31Z")
> Now that I have a search which shows all entries with grok parse failures, how would I go about deleting only those entries from elasticsearch? Use the [delete by query API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html). > Also there seems to be two fields for the orginal message, \_source and message which one would be better to use for outputting to a file. `_source` is the whole document, i.e. all fields. Depending on how your filters are (and were) set up you might get away with dumping just the `message` field, but that's impossible for me to say. It should be easy for you since you're familiar with your logging format and your Logstash filters. ---
**Author:** ![dlb1001](https://avatars.discourse-cdn.com/v4/letter/d/ce7236/32.png) [@dlb1001](https://discuss.elastic.co/u/dlb1001)\ **Post date:** [October 8, 2015, 11:04am UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/5 "2015-10-08T11:04:50Z")
Hello, Based on what you said I've been experimenting with the api. I'm still having a few issues though. I think this command will return messages from all indexes of type apache which have tags of \_grokparsefailure. Also it will print it out in a nice format with the pretty command. However I keep getting an exception when I use it. Also I'm not quite sure how to get it to only print out the message field? I saw there is a way to have it print out the source message with the \_source\_include and \_source\_exclude commands. But I'm still not sure about just returning the message. ``` curl -XGET 'localhost:9200/_all/apache/_search?pretty' -d ' { "query": { “term” : { “tags” : “_grokparsefailure” } } }’ ``` Thank you ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 8, 2015, 11:51am UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/6 "2015-10-08T11:51:35Z")
Please always post exact error messages instead of saying "I'm getting an exception". If this is the exact command you're using the problem is that some of the double quotes are proper, typographically correct, quotation marks rather than the ones normally used in programming. Compare `"query"` and `“term”`. Use [`fields`](https://www.elastic.co/guide/en/elasticsearch/reference/1.4/search-request-fields.html) to select which document fields to include in the response (although you can always extract the message from the full source document in `_source`). ``` { "fields": ["message"], "query": { ... } } ``` ---
**Author:** ![dlb1001](https://avatars.discourse-cdn.com/v4/letter/d/ce7236/32.png) [@dlb1001](https://discuss.elastic.co/u/dlb1001)\ **Post date:** [October 8, 2015, 12:18pm UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/7 "2015-10-08T12:18:16Z")
Sorry I forgot the error on my last message. So based on your last input I modified what I was using, got rid of the incorrect quotes and added the field field. All of my indexes start with logstash-[date] as below, and they are of type "apache". when I think I have the command correct I'll be switching the specific index to \_all and also changing the size to a much larger number. Anyway this is what I"m using: curl -XGET '[http://localhost:9200/logstash-2015.09.26/apache/\_search](http://localhost:9200/logstash-2015.09.26/apache/_search)' -d '{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "\_grokparsefailure" } } }' And this is the error I receive. {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][0]: SearchParseException[[logstash-2015.09.26][0]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "\_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][0]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }{[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][1]: SearchParseException[[logstash-2015.09.26][1]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "\_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][1]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }{[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][2]: SearchParseException[[logstash-2015.09.26][2]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "\_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][2]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }{[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][3]: SearchParseException[[logstash-2015.09.26][3]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "\_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][3]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }{[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][4]: SearchParseException[[logstash-2015.09.26][4]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "\_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][4]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }]","status":400} ---
**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\ **Post date:** [October 8, 2015, 12:25pm UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/8 "2015-10-08T12:25:35Z")
Note that in @magnusbaeck 's post above the option is `fields` rather than `field`. ---
**Author:** ![dlb1001](https://avatars.discourse-cdn.com/v4/letter/d/ce7236/32.png) [@dlb1001](https://discuss.elastic.co/u/dlb1001)\ **Post date:** [October 8, 2015, 12:34pm UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/9 "2015-10-08T12:34:59Z")
that was it, I totally missed it. Thank you both for all the help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:45pm UTC](https://discuss.elastic.co/t/reparse-logs-in-elastic-search/31770/10 "2017-07-05T23:45:54Z")