# Storing syslog data **URL:** \ **Category:** Elasticsearch\ **Created:** [December 12, 2013, 1:19am UTC](https://discuss.elastic.co/t/storing-syslog-data/14828 "2013-12-12T01:19:28Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Aaron\_Anderson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron_anderson/32/1910_2.png) [@Aaron\_Anderson](https://discuss.elastic.co/u/Aaron_Anderson)\ **Post date:** [December 12, 2013, 1:19am UTC](https://discuss.elastic.co/t/storing-syslog-data/14828/1 "2013-12-12T01:19:28Z")
Not knowing much of anything about ES, I want to use it behind graylog2 as a syslog server. This is what my Firewall syslog entries look like... { "\_index": "logstash-2013.12.11", "\_type": "logs", "\_id": "7PtMS0SUT\_yY4mbZyTs6OQ", "\_score": null, "\_source": { "message": "\<134\>Dec 11 17:28:09 swpeak10 id=firewall sn=0017C5B334C0 time="2013-12-11 22:28:09 UTC" bid=1 fw=69.2.197.33 pri=6 c=512 m=602 msg="DNS packet allowed" n=0 src=10.1.0.5:53:X0-V81:corpdc2.d.com dst=10.10.x.1:2737:X0-V81: proto=udp/2737 \n", "@timestamp": "2013-12-11T22:28:08.818Z", "@version": "1", "type": "syslog", "host": "10.10.230.31", "tags": [ "\_grokparsefailure" ], "priority": 13, "severity": 5, "facility": 1, "facility\_label": "user-level", "severity\_label": "Notice" }, "sort": [ 1386800888818 ] } Should I be parsing the records before they enter ES for better search?, ie, should i parse out src= and dst= into their own fields if i want to be able to search those later? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/007e0c2e-84c6-409e-8cf3-bd0444ecea8b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/007e0c2e-84c6-409e-8cf3-bd0444ecea8b%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Aaron\_Anderson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron_anderson/32/1910_2.png) [@Aaron\_Anderson](https://discuss.elastic.co/u/Aaron_Anderson)\ **Post date:** [December 12, 2013, 1:24am UTC](https://discuss.elastic.co/t/storing-syslog-data/14828/2 "2013-12-12T01:24:29Z")
Note: This is from Logstash/Kibana. I'm also testing out Graylog2 and got things confused. Long day. On Wednesday, December 11, 2013 8:19:28 PM UTC-5, Aaron Anderson wrote: > Not knowing much of anything about ES, I want to use it behind graylog2 as > a syslog server. > > This is what my Firewall syslog entries look like... > > { > "\_index": "logstash-2013.12.11", > "\_type": "logs", > "\_id": "7PtMS0SUT\_yY4mbZyTs6OQ", > "\_score": null, > "\_source": { > "message": "\<134\>Dec 11 17:28:09 swpeak10 id=firewall sn=0017C5B334C0 time="2013-12-11 22:28:09 UTC" bid=1 fw=69.2.197.33 pri=6 c=512 m=602 msg="DNS packet allowed" n=0 src=10.1.0.5:53:X0-V81:corpdc2.d.com dst=10.10.x.1:2737:X0-V81: proto=udp/2737 \n", > "@timestamp": "2013-12-11T22:28:08.818Z", > "@version": "1", > "type": "syslog", > "host": "10.10.230.31", > "tags": [ > "\_grokparsefailure" > ], > "priority": 13, > "severity": 5, > "facility": 1, > "facility\_label": "user-level", > "severity\_label": "Notice" > }, > "sort": [ > 1386800888818 > ] > } > > Should I be parsing the records before they enter ES for better > search?, ie, should i parse out src= and dst= into their own fields if i > want to be able to search those later? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/73bc1573-61a6-42db-bd4c-c746581517b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/73bc1573-61a6-42db-bd4c-c746581517b3%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Aaron\_Anderson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron_anderson/32/1910_2.png) [@Aaron\_Anderson](https://discuss.elastic.co/u/Aaron_Anderson)\ **Post date:** [December 12, 2013, 3:31am UTC](https://discuss.elastic.co/t/storing-syslog-data/14828/3 "2013-12-12T03:31:09Z")
By adding a filter: filter { kv { } } I end up with this: { "\_index": "logstash-2013.12.12", "\_type": "logs", "\_id": "-O7OAplKTvqT5t7fe1BmrQ", "\_score": null, "\_source": { "message": "\<134\>Dec 11 22:23:01 swpeak10 id=firewall sn=0017C5B334C0 time="2013-12-12 03:23:01 UTC" bid=1 fw=69.x.x.33 pri=6 c=262144 m=98 msg="Connection Opened" n=0 src=10.10.240.1:4844:X0-V81: dst=10.1.0.5:53:X0-V81:corpdc2.x.com proto=udp/dns \n", "@timestamp": "2013-12-12T03:23:00.879Z", "@version": "1", "type": "syslog", "host": "10.10.230.31", "tags": [ "\_grokparsefailure" ], "priority": 13, "severity": 5, "facility": 1, "facility\_label": "user-level", "severity\_label": "Notice", "id": "firewall", "sn": "0017C5B334C0", "time": "2013-12-12 03:23:01 UTC", "bid": "1", "fw": "69.x.x.33", "pri": "6", "c": "262144", "m": "98", "msg": "Connection Opened", "n": "0", "src": "10.10.240.1:4844:X0-V81:", "dst": "10.1.0.5:53:X0-V81:corpdc2.x.com", "proto": "udp/dns" }, "sort": [ 1386818580879 ] } I still want to tidy up the "src" and "dst" fields. Any comments/info on doing that would be appreciated a great deal. On Wednesday, December 11, 2013 8:19:28 PM UTC-5, Aaron Anderson wrote: > Not knowing much of anything about ES, I want to use it behind graylog2 as > a syslog server. > > This is what my Firewall syslog entries look like... > > { > "\_index": "logstash-2013.12.11", > "\_type": "logs", > "\_id": "7PtMS0SUT\_yY4mbZyTs6OQ", > "\_score": null, > "\_source": { > "message": "\<134\>Dec 11 17:28:09 swpeak10 id=firewall sn=0017C5B334C0 time="2013-12-11 22:28:09 UTC" bid=1 fw=69.2.197.33 pri=6 c=512 m=602 msg="DNS packet allowed" n=0 src=10.1.0.5:53:X0-V81:corpdc2.d.com dst=10.10.x.1:2737:X0-V81: proto=udp/2737 \n", > "@timestamp": "2013-12-11T22:28:08.818Z", > "@version": "1", > "type": "syslog", > "host": "10.10.230.31", > "tags": [ > "\_grokparsefailure" > ], > "priority": 13, > "severity": 5, > "facility": 1, > "facility\_label": "user-level", > "severity\_label": "Notice" > }, > "sort": [ > 1386800888818 > ] > } > > Should I be parsing the records before they enter ES for better > search?, ie, should i parse out src= and dst= into their own fields if i > want to be able to search those later? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1af43d46-b165-4ae4-909b-594851261f70%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1af43d46-b165-4ae4-909b-594851261f70%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 13, 2013, 1:44pm UTC](https://discuss.elastic.co/t/storing-syslog-data/14828/4 "2013-12-13T13:44:06Z")
Hi Aaron, Yes, you can parse those with something like Logstash's grok filter[http://logstash.net/docs/1.3.1/filters/grok](http://logstash.net/docs/1.3.1/filters/grok). You'd probably want to put IPs in their own fields of type IP[http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ip-type.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ip-type.html) . On Thu, Dec 12, 2013 at 3:19 AM, Aaron Anderson [anderson.aaron@gmail.com](mailto:anderson.aaron@gmail.com)wrote: > Not knowing much of anything about ES, I want to use it behind graylog2 as > a syslog server. > > This is what my Firewall syslog entries look like... > > { > "\_index": "logstash-2013.12.11", > "\_type": "logs", > "\_id": "7PtMS0SUT\_yY4mbZyTs6OQ", > "\_score": null, > "\_source": { > "message": "\<134\>Dec 11 17:28:09 swpeak10 id=firewall sn=0017C5B334C0 time="2013-12-11 22:28:09 UTC" bid=1 fw=69.2.197.33 pri=6 c=512 m=602 msg="DNS packet allowed" n=0 src=10.1.0.5:53:X0-V81:corpdc2.d.com dst=10.10.x.1:2737:X0-V81: proto=udp/2737 \n", > "@timestamp": "2013-12-11T22:28:08.818Z", > "@version": "1", > "type": "syslog", > "host": "10.10.230.31", > "tags": [ > "\_grokparsefailure" > ], > "priority": 13, > "severity": 5, > "facility": 1, > "facility\_label": "user-level", > "severity\_label": "Notice" > }, > "sort": [ > 1386800888818 > ] > } > > Should I be parsing the records before they enter ES for better > search?, ie, should i parse out src= and dst= into their own fields if i > want to be able to search those later? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/007e0c2e-84c6-409e-8cf3-bd0444ecea8b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/007e0c2e-84c6-409e-8cf3-bd0444ecea8b%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3FF8KCGO9NkgkcgoAkhXB4aRj-OJX3Y0%3D2QzRkM9w83w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3FF8KCGO9NkgkcgoAkhXB4aRj-OJX3Y0%3D2QzRkM9w83w%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:01am UTC](https://discuss.elastic.co/t/storing-syslog-data/14828/5 "2017-07-06T02:01:26Z")