Search time field extraction

Elastic Stack Elasticsearch
1

I am a newbie to ES/Kibana/logstash and still getting my feet wet.

I could be missing something obvious but I haven't come across a way so far
to extract a field at search time i.e., without setting up a
grok{}/grep{}/similar filter in logstash. The grok{} filter is great for
an event format that we know the structure but it isn't always practical to
setup filters for all interested fields ahead of the time.

As an example, one of the events I am working on has the following format:

Trigger: LOGON Start Time: 2014-02-24T04:42:28 End Time:

2014-02-24T04:42:31 Duration: 2918ms.

I wanted to graph on the Duration value i.e., 2918 and would like to
extract it. I setup a grok{} filter and was able to extract it but given
that the extraction is index-time, only future events gets the field
indexed.

I am wondering if someone knows a way/have a suggestion on how to:

  1. extract a field of interest at search time
  2. refresh an index when a new grok{} filter is added and a new field
    extracted

Appreciate your help!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/27edad95-a6f3-4729-ba05-6171a1ed6920%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Technically, you can probably do this with a little scripting and the
script_fields functionality:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-script-fields.html

However, Kibana does not expose this at the moment so for now, you'll need
to produce the exact field values at index time. And yes it will not apply
to already indexed data as you discovered - you will need to delete those
old indexes and then re-run the logs again using your new LS config file if
you want the values from the old logs.

Btw, incase you've not seen this, there is a nifty grok debugger here
(makes grokking and testing a lot quicker):

http://grokdebug.herokuapp.com/

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8f8ba448-6330-4b4f-9410-14454845b6b7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.