Newbie Kibana/elasticsearch/fluentd user here. Long-time Splunk power-user.
One of the things I find myself doing all the time in Splunk is working
with ultimately unstructured data and, as I need to, extracting a field
from search events matching my current search so I can subsequently chart
that field or do something interesting with it. For example:
invalid_session | rex "ID: (?<msg_id>.*)" | stats dc(msg_id) by host
So, here I'm using the "rex" command to ad-hoc extract a "msg_id" field
which I can subsequently use in another command (here, counting the
distinct msg_id by host).
Is there any way to do this with Kibana? This is hugely valuable when
day-in/day-out you're dealing with unstructured data of one sort or another
(too many to setup field extractions for!).
Thanks in advance!
Brice Ruth
--
Notice: The information contained in this message or any attached
document is confidential and intended only for individuals to whom it is
addressed. If you got this message in error, please inform me immediately
using one of the methods above. In some cases, I may ask you to return the
documents at my expense. In general, please simply destroy the information
at once. Any unauthorized use, distribution, or copying of this information
is prohibited.
I think that, for now, the best way to go is to use something like Logstash
to extract fields from unstructured data at index time. I'm not aware of a
way to do that at query time.
Best regards,
Radu
On Sun, Dec 8, 2013 at 5:58 PM, Brice Ruth bruth@compcc.com wrote:
Good morning.
Newbie Kibana/elasticsearch/fluentd user here. Long-time Splunk
power-user. One of the things I find myself doing all the time in Splunk is
working with ultimately unstructured data and, as I need to, extracting a
field from search events matching my current search so I can subsequently
chart that field or do something interesting with it. For example:
invalid_session | rex "ID: (?<msg_id>.*)" | stats dc(msg_id) by host
So, here I'm using the "rex" command to ad-hoc extract a "msg_id" field
which I can subsequently use in another command (here, counting the
distinct msg_id by host).
Is there any way to do this with Kibana? This is hugely valuable when
day-in/day-out you're dealing with unstructured data of one sort or another
(too many to setup field extractions for!).
Thanks in advance!
Brice Ruth
Notice: The information contained in this message or any attached
document is confidential and intended only for individuals to whom it is
addressed. If you got this message in error, please inform me immediately
using one of the methods above. In some cases, I may ask you to return the
documents at my expense. In general, please simply destroy the information
at once. Any unauthorized use, distribution, or copying of this information
is prohibited.
Thanks for the response! I think this is an important feature to get on the
roadmap, architecturally. It's massively useful when processing
unstructured data.
I think that, for now, the best way to go is to use something like
Logstash to extract fields from unstructured data at index time. I'm not
aware of a way to do that at query time.
Best regards,
Radu
On Sun, Dec 8, 2013 at 5:58 PM, Brice Ruth bruth@compcc.com wrote:
Good morning.
Newbie Kibana/elasticsearch/fluentd user here. Long-time Splunk
power-user. One of the things I find myself doing all the time in Splunk is
working with ultimately unstructured data and, as I need to, extracting a
field from search events matching my current search so I can subsequently
chart that field or do something interesting with it. For example:
invalid_session | rex "ID: (?<msg_id>.*)" | stats dc(msg_id) by host
So, here I'm using the "rex" command to ad-hoc extract a "msg_id" field
which I can subsequently use in another command (here, counting the
distinct msg_id by host).
Is there any way to do this with Kibana? This is hugely valuable when
day-in/day-out you're dealing with unstructured data of one sort or another
(too many to setup field extractions for!).
Thanks in advance!
Brice Ruth
Notice: The information contained in this message or any attached
document is confidential and intended only for individuals to whom it is
addressed. If you got this message in error, please inform me immediately
using one of the methods above. In some cases, I may ask you to return the
documents at my expense. In general, please simply destroy the information
at once. Any unauthorized use, distribution, or copying of this information
is prohibited.
Notice: The information contained in this message or any attached
document is confidential and intended only for individuals to whom it is
addressed. If you got this message in error, please inform me immediately
using one of the methods above. In some cases, I may ask you to return the
documents at my expense. In general, please simply destroy the information
at once. Any unauthorized use, distribution, or copying of this information
is prohibited.
Would it be possible to write some type of plugin to extract the relevant
fields afterwards?
To write filters for logstash that catch every possible emailpattern
beforehand is not that easy.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.