Reparse logs in elastic search

Elastic Stack Elasticsearch
1

Hello,

Is there a way to re parse log entries in elastic search? The reason I ask is I've been inputing all of our old logs into elasticsearch. I thought my groks parsing handled the majority of entries, however out of the 3 years of logs, I ran into a few months where the log entries differed slightly, just enough to mess up my grok filters. Is there a way to search for the grok parse failures in elastic search and reparse the information for the ones that messed up?

Thank you

2

If you have saved the original log message as a field it would be easy to recreate the original logs that resulted in grok problem and push them to Logstash once again and afterwards delete the _grokparsefailure messages. There's nothing turn-key solution for this though.

If you don't have the full original message then perhaps you can stitch it together using various fields in the message or write a custom Logstash configuration to fix the extracted logs.

3

Now that I have a search which shows all entries with grok parse failures, how would I go about deleting only those entries from elasticsearch? Also there seems to be two fields for the orginal message, _source and message which one would be better to use for outputting to a file.

4

Now that I have a search which shows all entries with grok parse failures, how would I go about deleting only those entries from elasticsearch?

Use the delete by query API.

Also there seems to be two fields for the orginal message, _source and message which one would be better to use for outputting to a file.

_source is the whole document, i.e. all fields. Depending on how your filters are (and were) set up you might get away with dumping just the message field, but that's impossible for me to say. It should be easy for you since you're familiar with your logging format and your Logstash filters.

5

Hello,

Based on what you said I've been experimenting with the api. I'm still having a few issues though. I think this command will return messages from all indexes of type apache which have tags of _grokparsefailure. Also it will print it out in a nice format with the pretty command. However I keep getting an exception when I use it. Also I'm not quite sure how to get it to only print out the message field? I saw there is a way to have it print out the source message with the _source_include and _source_exclude commands. But I'm still not sure about just returning the message.

curl -XGET 'localhost:9200/_all/apache/_search?pretty' -d ' {  "query": { “term” : { “tags” : “_grokparsefailure” }  } }’

Thank you

6

Please always post exact error messages instead of saying "I'm getting an exception".

If this is the exact command you're using the problem is that some of the double quotes are proper, typographically correct, quotation marks rather than the ones normally used in programming. Compare "query" and “term”.

Use fields to select which document fields to include in the response (although you can always extract the message from the full source document in _source).

{
  "fields": ["message"],
  "query": {
    ...
  }
}
7

Sorry I forgot the error on my last message. So based on your last input I modified what I was using, got rid of the incorrect quotes and added the field field. All of my indexes start with logstash-[date] as below, and they are of type "apache". when I think I have the command correct I'll be switching the specific index to _all and also changing the size to a much larger number. Anyway this is what I"m using:

curl -XGET 'http://localhost:9200/logstash-2015.09.26/apache/_search' -d '{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "_grokparsefailure" } } }'

And this is the error I receive.

{"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][0]: SearchParseException[[logstash-2015.09.26][0]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][0]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }{[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][1]: SearchParseException[[logstash-2015.09.26][1]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][1]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }{[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][2]: SearchParseException[[logstash-2015.09.26][2]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][2]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }{[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][3]: SearchParseException[[logstash-2015.09.26][3]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][3]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }{[Fhe98oygSiqlao6f34R6gQ][logstash-2015.09.26][4]: SearchParseException[[logstash-2015.09.26][4]: from[-1],size[1]: Parse Failure [Failed to parse source [{ "size" : 1, "field" : ["message"], "query" : { "term" : { "tags" : "_grokparsefailure" } } }]]]; nested: SearchParseException[[logstash-2015.09.26][4]: from[-1],size[1]: Parse Failure [No parser for element [field]]]; }]","status":400}

8

Note that in @magnusbaeck 's post above the option is fields rather than field.

9

that was it, I totally missed it. Thank you both for all the help.