Storing syslog data


(Aaron Anderson) #1

Not knowing much of anything about ES, I want to use it behind graylog2 as
a syslog server.

This is what my Firewall syslog entries look like...

{
"_index": "logstash-2013.12.11",
"_type": "logs",
"_id": "7PtMS0SUT_yY4mbZyTs6OQ",
"_score": null,
"_source": {
"message": "<134>Dec 11 17:28:09 swpeak10 id=firewall sn=0017C5B334C0 time="2013-12-11 22:28:09 UTC" bid=1 fw=69.2.197.33 pri=6 c=512 m=602 msg="DNS packet allowed" n=0 src=10.1.0.5:53:X0-V81:corpdc2.d.com dst=10.10.x.1:2737:X0-V81: proto=udp/2737 \n",
"@timestamp": "2013-12-11T22:28:08.818Z",
"@version": "1",
"type": "syslog",
"host": "10.10.230.31",
"tags": [
"_grokparsefailure"
],
"priority": 13,
"severity": 5,
"facility": 1,
"facility_label": "user-level",
"severity_label": "Notice"
},
"sort": [
1386800888818
]
}

Should I be parsing the records before they enter ES for better search?,
ie, should i parse out src= and dst= into their own fields if i want to be
able to search those later?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/007e0c2e-84c6-409e-8cf3-bd0444ecea8b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Aaron Anderson) #2

Note: This is from Logstash/Kibana. I'm also testing out Graylog2 and got
things confused. Long day.

On Wednesday, December 11, 2013 8:19:28 PM UTC-5, Aaron Anderson wrote:

Not knowing much of anything about ES, I want to use it behind graylog2 as
a syslog server.

This is what my Firewall syslog entries look like...

{
"_index": "logstash-2013.12.11",
"_type": "logs",
"_id": "7PtMS0SUT_yY4mbZyTs6OQ",
"_score": null,
"_source": {
"message": "<134>Dec 11 17:28:09 swpeak10 id=firewall sn=0017C5B334C0 time="2013-12-11 22:28:09 UTC" bid=1 fw=69.2.197.33 pri=6 c=512 m=602 msg="DNS packet allowed" n=0 src=10.1.0.5:53:X0-V81:corpdc2.d.com dst=10.10.x.1:2737:X0-V81: proto=udp/2737 \n",
"@timestamp": "2013-12-11T22:28:08.818Z",
"@version": "1",
"type": "syslog",
"host": "10.10.230.31",
"tags": [
"_grokparsefailure"
],
"priority": 13,
"severity": 5,
"facility": 1,
"facility_label": "user-level",
"severity_label": "Notice"
},
"sort": [
1386800888818
]
}

Should I be parsing the records before they enter ES for better
search?, ie, should i parse out src= and dst= into their own fields if i
want to be able to search those later?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/73bc1573-61a6-42db-bd4c-c746581517b3%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Aaron Anderson) #3

By adding a filter:

filter {
kv { }
}

I end up with this:
{
"_index": "logstash-2013.12.12",
"_type": "logs",
"_id": "-O7OAplKTvqT5t7fe1BmrQ",
"_score": null,
"_source": {
"message": "<134>Dec 11 22:23:01 swpeak10 id=firewall sn=0017C5B334C0
time="2013-12-12 03:23:01 UTC" bid=1 fw=69.x.x.33 pri=6 c=262144 m=98
msg="Connection Opened" n=0 src=10.10.240.1:4844:X0-V81:
dst=10.1.0.5:53:X0-V81:corpdc2.x.com proto=udp/dns \n",
"@timestamp": "2013-12-12T03:23:00.879Z",
"@version": "1",
"type": "syslog",
"host": "10.10.230.31",
"tags": [
"_grokparsefailure"
],
"priority": 13,
"severity": 5,
"facility": 1,
"facility_label": "user-level",
"severity_label": "Notice",
"id": "firewall",
"sn": "0017C5B334C0",
"time": "2013-12-12 03:23:01 UTC",
"bid": "1",
"fw": "69.x.x.33",
"pri": "6",
"c": "262144",
"m": "98",
"msg": "Connection Opened",
"n": "0",
"src": "10.10.240.1:4844:X0-V81:",
"dst": "10.1.0.5:53:X0-V81:corpdc2.x.com",
"proto": "udp/dns"
},
"sort": [
1386818580879
]
}

I still want to tidy up the "src" and "dst" fields. Any comments/info on
doing that would be appreciated a great deal.

On Wednesday, December 11, 2013 8:19:28 PM UTC-5, Aaron Anderson wrote:

Not knowing much of anything about ES, I want to use it behind graylog2 as
a syslog server.

This is what my Firewall syslog entries look like...

{
"_index": "logstash-2013.12.11",
"_type": "logs",
"_id": "7PtMS0SUT_yY4mbZyTs6OQ",
"_score": null,
"_source": {
"message": "<134>Dec 11 17:28:09 swpeak10 id=firewall sn=0017C5B334C0 time="2013-12-11 22:28:09 UTC" bid=1 fw=69.2.197.33 pri=6 c=512 m=602 msg="DNS packet allowed" n=0 src=10.1.0.5:53:X0-V81:corpdc2.d.com dst=10.10.x.1:2737:X0-V81: proto=udp/2737 \n",
"@timestamp": "2013-12-11T22:28:08.818Z",
"@version": "1",
"type": "syslog",
"host": "10.10.230.31",
"tags": [
"_grokparsefailure"
],
"priority": 13,
"severity": 5,
"facility": 1,
"facility_label": "user-level",
"severity_label": "Notice"
},
"sort": [
1386800888818
]
}

Should I be parsing the records before they enter ES for better
search?, ie, should i parse out src= and dst= into their own fields if i
want to be able to search those later?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1af43d46-b165-4ae4-909b-594851261f70%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Radu Gheorghe) #4

Hi Aaron,

Yes, you can parse those with something like Logstash's grok
filterhttp://logstash.net/docs/1.3.1/filters/grok.
You'd probably want to put IPs in their own fields of type
IPhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ip-type.html
.

On Thu, Dec 12, 2013 at 3:19 AM, Aaron Anderson anderson.aaron@gmail.comwrote:

Not knowing much of anything about ES, I want to use it behind graylog2 as
a syslog server.

This is what my Firewall syslog entries look like...

{
"_index": "logstash-2013.12.11",
"_type": "logs",
"_id": "7PtMS0SUT_yY4mbZyTs6OQ",
"_score": null,
"_source": {
"message": "<134>Dec 11 17:28:09 swpeak10 id=firewall sn=0017C5B334C0 time="2013-12-11 22:28:09 UTC" bid=1 fw=69.2.197.33 pri=6 c=512 m=602 msg="DNS packet allowed" n=0 src=10.1.0.5:53:X0-V81:corpdc2.d.com dst=10.10.x.1:2737:X0-V81: proto=udp/2737 \n",
"@timestamp": "2013-12-11T22:28:08.818Z",
"@version": "1",
"type": "syslog",
"host": "10.10.230.31",
"tags": [
"_grokparsefailure"
],
"priority": 13,
"severity": 5,
"facility": 1,
"facility_label": "user-level",
"severity_label": "Notice"
},
"sort": [
1386800888818
]
}

Should I be parsing the records before they enter ES for better
search?, ie, should i parse out src= and dst= into their own fields if i
want to be able to search those later?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/007e0c2e-84c6-409e-8cf3-bd0444ecea8b%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3FF8KCGO9NkgkcgoAkhXB4aRj-OJX3Y0%3D2QzRkM9w83w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #5