# Dynamic fields from key=value formatted messages? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 10, 2013, 1:32am UTC](https://discuss.elastic.co/t/dynamic-fields-from-key-value-formatted-messages/14336 "2013-11-10T01:32:13Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Matt\_Wise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_wise/32/701_2.png) [@Matt\_Wise](https://discuss.elastic.co/u/Matt_Wise)\ **Post date:** [November 10, 2013, 1:32am UTC](https://discuss.elastic.co/t/dynamic-fields-from-key-value-formatted-messages/14336/1 "2013-11-10T01:32:13Z")
Hey we'd like to set up a default format for all of our logging systems... perhaps looking like this: "key1=value1;key2=value2;key3=value3...." With this pattern, we'd allow developers to define any key/value pairs they want to log, and separate them with a common separator. If we did this, what do we need to do in ElasticSearch to parse the @message field and automatically parse these key=value pairs into searchable fields? Any thoughts? --Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![limac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/limac/32/3981_2.png) [@limac](https://discuss.elastic.co/u/limac)\ **Post date:** [November 11, 2013, 5:20am UTC](https://discuss.elastic.co/t/dynamic-fields-from-key-value-formatted-messages/14336/2 "2013-11-11T05:20:36Z")
why not take json as the format of your log? it’s better for elasticsearch to handle this. Sent from Surface From: Matt Sent: Sunday, November 10, 2013 9:32 AM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Hey we'd like to set up a default format for all of our logging systems... perhaps looking like this: "key1=value1;key2=value2;key3=value3...." With this pattern, we'd allow developers to define any key/value pairs they want to log, and separate them with a common separator. If we did this, what do we need to do in ElasticSearch to parse the @message field and automatically parse these key=value pairs into searchable fields? Any thoughts? --Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Matt\_Wise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_wise/32/701_2.png) [@Matt\_Wise](https://discuss.elastic.co/u/Matt_Wise)\ **Post date:** [November 11, 2013, 6:03pm UTC](https://discuss.elastic.co/t/dynamic-fields-from-key-value-formatted-messages/14336/3 "2013-11-11T18:03:11Z")
The log lines are coming through Syslog, then going through Flume, then being pushed into Elasticsearch. We have the ability to format the "msg" part of the log line a bit, but it would be very hard to do JSON. This is why we want to do something like a key=value system. Matt Wise Sr. Systems Architect > **[Join Nextdoor, an app for neighborhoods where you can get local tips, buy and...](https://nextdoor.com)** > > Nextdoor is the neighborhood hub for trusted connections and the exchange of helpful information, goods, and services. On Sun, Nov 10, 2013 at 9:20 PM, [cnwangyong@gmail.com](mailto:cnwangyong@gmail.com) wrote: > why not take json as the format of your log? it’s better for > elasticsearch to handle this. > > Sent from Surface > > _From:_ Matt [matt@nextdoor.com](mailto:matt@nextdoor.com) > _Sent:_ Sunday, November 10, 2013 9:32 AM > _To:_ [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > > Hey we'd like to set up a default format for all of our logging systems... > perhaps looking like this: > > "key1=value1;key2=value2;key3=value3...." > > With this pattern, we'd allow developers to define any key/value pairs > they want to log, and separate them with a common separator. > > If we did this, what do we need to do in Elasticsearch to parse the > @message field and automatically parse these key=value pairs into > searchable fields? > > Any thoughts? > > --Matt > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/qdXhgRNVocw/unsubscribe](https://groups.google.com/d/topic/elasticsearch/qdXhgRNVocw/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:07am UTC](https://discuss.elastic.co/t/dynamic-fields-from-key-value-formatted-messages/14336/4 "2017-07-06T02:07:49Z")