Kibana: extract fields, ad-hoc?

Good morning.

Newbie Kibana/elasticsearch/fluentd user here. Long-time Splunk power-user.
One of the things I find myself doing all the time in Splunk is working
with ultimately unstructured data and, as I need to, extracting a field
from search events matching my current search so I can subsequently chart
that field or do something interesting with it. For example:

invalid_session | rex "ID: (?<msg_id>.*)" | stats dc(msg_id) by host

So, here I'm using the "rex" command to ad-hoc extract a "msg_id" field
which I can subsequently use in another command (here, counting the
distinct msg_id by host).

Is there any way to do this with Kibana? This is hugely valuable when
day-in/day-out you're dealing with unstructured data of one sort or another
(too many to setup field extractions for!).

Thanks in advance!
Brice Ruth

--


Notice: The information contained in this message or any attached
document is confidential and intended only for individuals to whom it is
addressed. If you got this message in error, please inform me immediately
using one of the methods above. In some cases, I may ask you to return the
documents at my expense. In general, please simply destroy the information
at once. Any unauthorized use, distribution, or copying of this information
is prohibited.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/49e1df0a-1707-42be-aeac-f63c90ce52dd%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hello Brice,

I think that, for now, the best way to go is to use something like Logstash
to extract fields from unstructured data at index time. I'm not aware of a
way to do that at query time.

Best regards,
Radu

On Sun, Dec 8, 2013 at 5:58 PM, Brice Ruth bruth@compcc.com wrote:

Good morning.

Newbie Kibana/elasticsearch/fluentd user here. Long-time Splunk
power-user. One of the things I find myself doing all the time in Splunk is
working with ultimately unstructured data and, as I need to, extracting a
field from search events matching my current search so I can subsequently
chart that field or do something interesting with it. For example:

invalid_session | rex "ID: (?<msg_id>.*)" | stats dc(msg_id) by host

So, here I'm using the "rex" command to ad-hoc extract a "msg_id" field
which I can subsequently use in another command (here, counting the
distinct msg_id by host).

Is there any way to do this with Kibana? This is hugely valuable when
day-in/day-out you're dealing with unstructured data of one sort or another
(too many to setup field extractions for!).

Thanks in advance!
Brice Ruth


Notice: The information contained in this message or any attached
document is confidential and intended only for individuals to whom it is
addressed. If you got this message in error, please inform me immediately
using one of the methods above. In some cases, I may ask you to return the
documents at my expense. In general, please simply destroy the information
at once. Any unauthorized use, distribution, or copying of this information
is prohibited.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/49e1df0a-1707-42be-aeac-f63c90ce52dd%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_1v7JM_tpjeQ7iK1ZLf-rsZNB5nz28kzcg8xjDbX-ZFiA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for the response! I think this is an important feature to get on the
roadmap, architecturally. It's massively useful when processing
unstructured data.

Thanks!
On Dec 10, 2013 6:27 AM, "Radu Gheorghe" radu.gheorghe@sematext.com wrote:

Hello Brice,

I think that, for now, the best way to go is to use something like
Logstash to extract fields from unstructured data at index time. I'm not
aware of a way to do that at query time.

Best regards,
Radu

On Sun, Dec 8, 2013 at 5:58 PM, Brice Ruth bruth@compcc.com wrote:

Good morning.

Newbie Kibana/elasticsearch/fluentd user here. Long-time Splunk
power-user. One of the things I find myself doing all the time in Splunk is
working with ultimately unstructured data and, as I need to, extracting a
field from search events matching my current search so I can subsequently
chart that field or do something interesting with it. For example:

invalid_session | rex "ID: (?<msg_id>.*)" | stats dc(msg_id) by host

So, here I'm using the "rex" command to ad-hoc extract a "msg_id" field
which I can subsequently use in another command (here, counting the
distinct msg_id by host).

Is there any way to do this with Kibana? This is hugely valuable when
day-in/day-out you're dealing with unstructured data of one sort or another
(too many to setup field extractions for!).

Thanks in advance!
Brice Ruth


Notice: The information contained in this message or any attached
document is confidential and intended only for individuals to whom it is
addressed. If you got this message in error, please inform me immediately
using one of the methods above. In some cases, I may ask you to return the
documents at my expense. In general, please simply destroy the information
at once. Any unauthorized use, distribution, or copying of this information
is prohibited.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/49e1df0a-1707-42be-aeac-f63c90ce52dd%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/drvwBQhOJzo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_1v7JM_tpjeQ7iK1ZLf-rsZNB5nz28kzcg8xjDbX-ZFiA%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--


Notice: The information contained in this message or any attached
document is confidential and intended only for individuals to whom it is
addressed. If you got this message in error, please inform me immediately
using one of the methods above. In some cases, I may ask you to return the
documents at my expense. In general, please simply destroy the information
at once. Any unauthorized use, distribution, or copying of this information
is prohibited.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMCMPw8NoTOs5r78%3DO-ac18-N%2B0YEA3bZbyESOHxpVr9NGbDhQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Would it be possible to write some type of plugin to extract the relevant
fields afterwards?
To write filters for logstash that catch every possible emailpattern
beforehand is not that easy.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b79908af-cdd6-433a-8761-54198834fed4%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

1 Like