Dynamic fields from key=value formatted messages?


(Matt Wise) #1

Hey we'd like to set up a default format for all of our logging systems...
perhaps looking like this:

"key1=value1;key2=value2;key3=value3...."

With this pattern, we'd allow developers to define any key/value pairs they
want to log, and separate them with a common separator.

If we did this, what do we need to do in ElasticSearch to parse the
@message field and automatically parse these key=value pairs into
searchable fields?

Any thoughts?

--Matt

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Yong Wang) #2

why not take json as the format of your log? it’s better for elasticsearch to handle this.

Sent from Surface

From: Matt
Sent: Sunday, November 10, 2013 9:32 AM
To: elasticsearch@googlegroups.com

Hey we'd like to set up a default format for all of our logging systems... perhaps looking like this:

"key1=value1;key2=value2;key3=value3...."

With this pattern, we'd allow developers to define any key/value pairs they want to log, and separate them with a common separator.

If we did this, what do we need to do in ElasticSearch to parse the @message field and automatically parse these key=value pairs into searchable fields?

Any thoughts?

--Matt

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Matt Wise) #3

The log lines are coming through Syslog, then going through Flume, then
being pushed into ElasticSearch. We have the ability to format the "msg"
part of the log line a bit, but it would be very hard to do JSON. This is
why we want to do something like a key=value system.

Matt Wise
Sr. Systems Architect

On Sun, Nov 10, 2013 at 9:20 PM, cnwangyong@gmail.com wrote:

why not take json as the format of your log? it’s better for
elasticsearch to handle this.

Sent from Surface

From: Matt matt@nextdoor.com
Sent: Sunday, November 10, 2013 9:32 AM
To: elasticsearch@googlegroups.com

Hey we'd like to set up a default format for all of our logging systems...
perhaps looking like this:

"key1=value1;key2=value2;key3=value3...."

With this pattern, we'd allow developers to define any key/value pairs
they want to log, and separate them with a common separator.

If we did this, what do we need to do in ElasticSearch to parse the
@message field and automatically parse these key=value pairs into
searchable fields?

Any thoughts?

--Matt

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qdXhgRNVocw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #4