# Kibana: extract fields, ad-hoc? **URL:** \ **Category:** Elasticsearch\ **Created:** [December 8, 2013, 3:58pm UTC](https://discuss.elastic.co/t/kibana-extract-fields-ad-hoc/14763 "2013-12-08T15:58:26Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Brice\_Ruth](https://avatars.discourse-cdn.com/v4/letter/b/bc79bd/32.png) [@Brice\_Ruth](https://discuss.elastic.co/u/Brice_Ruth)\ **Post date:** [December 8, 2013, 3:58pm UTC](https://discuss.elastic.co/t/kibana-extract-fields-ad-hoc/14763/1 "2013-12-08T15:58:26Z")
Good morning. Newbie Kibana/elasticsearch/fluentd user here. Long-time Splunk power-user. One of the things I find myself doing all the time in Splunk is working with ultimately unstructured data and, as I need to, extracting a field from search events matching my current search so I can subsequently chart that field or do something interesting with it. For example: invalid\_session | rex "ID: (?\.\*)" | stats dc(msg\_id) by host So, here I'm using the "rex" command to ad-hoc extract a "msg\_id" field which I can subsequently use in another command (here, counting the distinct msg\_id by host). Is there any way to do this with Kibana? This is hugely valuable when day-in/day-out you're dealing with unstructured data of one sort or another (too many to setup field extractions for!). Thanks in advance! Brice Ruth -- * * * _Notice_: The information contained in this message or any attached document is confidential and intended only for individuals to whom it is addressed. If you got this message in error, please inform me immediately using one of the methods above. In some cases, I may ask you to return the documents at my expense. In general, please simply destroy the information at once. Any unauthorized use, distribution, or copying of this information is prohibited. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/49e1df0a-1707-42be-aeac-f63c90ce52dd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/49e1df0a-1707-42be-aeac-f63c90ce52dd%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 10, 2013, 12:27pm UTC](https://discuss.elastic.co/t/kibana-extract-fields-ad-hoc/14763/2 "2013-12-10T12:27:47Z")
Hello Brice, I think that, for now, the best way to go is to use something like Logstash to extract fields from unstructured data at index time. I'm not aware of a way to do that at query time. Best regards, Radu On Sun, Dec 8, 2013 at 5:58 PM, Brice Ruth [bruth@compcc.com](mailto:bruth@compcc.com) wrote: > Good morning. > > Newbie Kibana/elasticsearch/fluentd user here. Long-time Splunk > power-user. One of the things I find myself doing all the time in Splunk is > working with ultimately unstructured data and, as I need to, extracting a > field from search events matching my current search so I can subsequently > chart that field or do something interesting with it. For example: > > invalid\_session | rex "ID: (?\.\*)" | stats dc(msg\_id) by host > > So, here I'm using the "rex" command to ad-hoc extract a "msg\_id" field > which I can subsequently use in another command (here, counting the > distinct msg\_id by host). > > Is there any way to do this with Kibana? This is hugely valuable when > day-in/day-out you're dealing with unstructured data of one sort or another > (too many to setup field extractions for!). > > Thanks in advance! > Brice Ruth > > * * * > > _Notice_: The information contained in this message or any attached > document is confidential and intended only for individuals to whom it is > addressed. If you got this message in error, please inform me immediately > using one of the methods above. In some cases, I may ask you to return the > documents at my expense. In general, please simply destroy the information > at once. Any unauthorized use, distribution, or copying of this information > is prohibited. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/49e1df0a-1707-42be-aeac-f63c90ce52dd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/49e1df0a-1707-42be-aeac-f63c90ce52dd%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_1v7JM\_tpjeQ7iK1ZLf-rsZNB5nz28kzcg8xjDbX-ZFiA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_1v7JM_tpjeQ7iK1ZLf-rsZNB5nz28kzcg8xjDbX-ZFiA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Brice\_Ruth\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brice_ruth_2/32/1919_2.png) [@Brice\_Ruth\_2](https://discuss.elastic.co/u/Brice_Ruth_2)\ **Post date:** [December 10, 2013, 1:10pm UTC](https://discuss.elastic.co/t/kibana-extract-fields-ad-hoc/14763/3 "2013-12-10T13:10:30Z")
Thanks for the response! I think this is an important feature to get on the roadmap, architecturally. It's massively useful when processing unstructured data. Thanks! On Dec 10, 2013 6:27 AM, "Radu Gheorghe" [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) wrote: > Hello Brice, > > I think that, for now, the best way to go is to use something like > Logstash to extract fields from unstructured data at index time. I'm not > aware of a way to do that at query time. > > Best regards, > Radu > > On Sun, Dec 8, 2013 at 5:58 PM, Brice Ruth [bruth@compcc.com](mailto:bruth@compcc.com) wrote: > > > Good morning. > > > > Newbie Kibana/elasticsearch/fluentd user here. Long-time Splunk > > power-user. One of the things I find myself doing all the time in Splunk is > > working with ultimately unstructured data and, as I need to, extracting a > > field from search events matching my current search so I can subsequently > > chart that field or do something interesting with it. For example: > > > > invalid\_session | rex "ID: (?\.\*)" | stats dc(msg\_id) by host > > > > So, here I'm using the "rex" command to ad-hoc extract a "msg\_id" field > > which I can subsequently use in another command (here, counting the > > distinct msg\_id by host). > > > > Is there any way to do this with Kibana? This is hugely valuable when > > day-in/day-out you're dealing with unstructured data of one sort or another > > (too many to setup field extractions for!). > > > > Thanks in advance! > > Brice Ruth > > > > * * * > > > > _Notice_: The information contained in this message or any attached > > document is confidential and intended only for individuals to whom it is > > addressed. If you got this message in error, please inform me immediately > > using one of the methods above. In some cases, I may ask you to return the > > documents at my expense. In general, please simply destroy the information > > at once. Any unauthorized use, distribution, or copying of this information > > is prohibited. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/49e1df0a-1707-42be-aeac-f63c90ce52dd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/49e1df0a-1707-42be-aeac-f63c90ce52dd%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Performance Monitoring \* Log Analytics \* Search Analytics > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/drvwBQhOJzo/unsubscribe](https://groups.google.com/d/topic/elasticsearch/drvwBQhOJzo/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_1v7JM\_tpjeQ7iK1ZLf-rsZNB5nz28kzcg8xjDbX-ZFiA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_1v7JM_tpjeQ7iK1ZLf-rsZNB5nz28kzcg8xjDbX-ZFiA%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- * * * _Notice_: The information contained in this message or any attached document is confidential and intended only for individuals to whom it is addressed. If you got this message in error, please inform me immediately using one of the methods above. In some cases, I may ask you to return the documents at my expense. In general, please simply destroy the information at once. Any unauthorized use, distribution, or copying of this information is prohibited. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAMCMPw8NoTOs5r78%3DO-ac18-N%2B0YEA3bZbyESOHxpVr9NGbDhQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAMCMPw8NoTOs5r78%3DO-ac18-N%2B0YEA3bZbyESOHxpVr9NGbDhQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Per\_Erik\_Persson](https://avatars.discourse-cdn.com/v4/letter/p/919ad9/32.png) [@Per\_Erik\_Persson](https://discuss.elastic.co/u/Per_Erik_Persson)\ **Post date:** [December 10, 2013, 9:55pm UTC](https://discuss.elastic.co/t/kibana-extract-fields-ad-hoc/14763/4 "2013-12-10T21:55:25Z")
Would it be possible to write some type of plugin to extract the relevant fields afterwards? To write filters for logstash that catch every possible emailpattern beforehand is not that easy. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b79908af-cdd6-433a-8761-54198834fed4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b79908af-cdd6-433a-8761-54198834fed4%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:02am UTC](https://discuss.elastic.co/t/kibana-extract-fields-ad-hoc/14763/5 "2017-07-06T02:02:06Z")