Here is how I am setup
.
Not sure if it clears things up.
I am shipping everything to my ELK server.
Lets go back to basics for a quick moment. What should a logstash.conf look like, if:
- Syslog (514, etc) is pointed to the ELK stack
- Syslog is outputing to the ELK stack for visualizations
Assumptions:
- No filters
- All network devices are pointing to TCP/UDP 514
Lets check to see if you have documents in the indices - try GET /logstash*/_search - what does it return?
Do you have a proxy in front of kibana?
Thanks,
Matt
No proxy, just a URL Rewrite. Here is the context of the statement:
{
"took" : 70,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "NkuASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.293Z",
"tags" : [
"_grokparsefailure"
],
"message" : """<189>date=2021-01-28 time=11:37:47 devname="FW1201135-Primary" devid="FG3H0E5819904320" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1611862667441816423 tz="-0800" srcip=10.200.1.45 srcport=64909 srcintf="ssl.root" srcintfrole="undefined" dstip=10.10.2.12 dstport=389 dstintf="vlan234" dstintfrole="lan" srcuuid="65aca018-0b02-51ea-41b0-2ce579685006" dstuuid="8ec351d2-188a-51ea-166f-5aa96f50a45d" poluuid="3b7da292-0fc1-51ea-66fe-def64f6b37fc" sessionid=618154101 proto=6 action="server-rst" user="jpichaparker" group="SSLVPN-Users" authserver="ad13srv-LDAP" policyid=14 policytype="policy" service="Internet-Locator-Service" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=5 sentbyte=3690 rcvdbyte=75223 sentpkt=35 rcvdpkt=60 appcat="unscanned" masterdstmac="74:8e:f8:70:d7:00" dstmac="74:8e:f8:70:d7:00" dstserver=0""",
"host" : "10.234.0.2"
}
},
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "N0uASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.310Z",
"tags" : [
"_grokparsefailure"
],
"message" : "<190>date=2021-01-28 time=11:37:47 devname=\"FW1201135-Primary\" devid=\"FG3H0E5819904320\" logid=\"1500054000\" type=\"utm\" subtype=\"dns\" eventtype=\"dns-query\" level=\"information\" vd=\"root\" eventtime=1611862667459335705 tz=\"-0800\" policyid=3 sessionid=618154408 srcip=10.10.2.12 srcport=52249 srcintf=\"vlan234\" srcintfrole=\"lan\" dstip=8.8.8.8 dstport=53 dstintf=\"CTL-WAN\" dstintfrole=\"wan\" proto=17 profile=\"default\" srcmac=\"74:8e:f8:70:d7:00\" xid=59835 qname=\"bn3pcor001-com.be.1drv.com\" qtype=\"A\" qtypeval=1 qclass=\"IN\"",
"host" : "10.234.0.2"
}
},
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "OEuASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.408Z",
"tags" : [
"_grokparsefailure"
],
"message" : "<190>date=2021-01-28 time=11:37:47 devname=\"FW1201135-Primary\" devid=\"FG3H0E5819904320\" logid=\"1059028704\" type=\"utm\" subtype=\"app-ctrl\" eventtype=\"signature\" level=\"information\" vd=\"root\" eventtime=1611862667545601859 tz=\"-0800\" appid=16009 srcip=10.15.10.22 dstip=52.185.211.133 srcport=56898 dstport=443 srcintf=\"vlan234\" srcintfrole=\"lan\" dstintf=\"CTL-WAN\" dstintfrole=\"wan\" proto=6 service=\"SSL\" direction=\"outgoing\" policyid=3 sessionid=618154400 applist=\"default\" action=\"pass\" appcat=\"Update\" app=\"MS.Windows.Update\" hostname=\"settings-win.data.microsoft.com\" incidentserialno=942818416 url=\"/\" msg=\"Update: MS.Windows.Update,\" scertcname=\"settings-win.data.microsoft.com\"",
"host" : "10.234.0.2"
}
},
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "OUuASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.447Z",
"tags" : [
"_grokparsefailure"
],
"message" : """<189>date=2021-01-28 time=11:37:47 devname="FW1201135-Primary" devid="FG3H0E5819904320" logid="0000000020" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1611862667582555909 tz="-0800" srcip=10.200.1.2 srcport=57065 srcintf="ssl.root" srcintfrole="undefined" dstip=10.10.8.100 dstport=3389 dstintf="vlan234" dstintfrole="lan" srcuuid="65aca018-0b02-51ea-41b0-2ce579685006" dstuuid="8ec351d2-188a-51ea-166f-5aa96f50a45d" poluuid="5eb2fcdc-5a2e-51ea-7cf5-f5ff351bbafe" sessionid=617912136 proto=6 action="accept" user="cmsmith" group="IT Management" authserver="ad13srv-LDAP" policyid=53 policytype="policy" service="RDP" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=4213 sentbyte=5214490 rcvdbyte=2123478 sentpkt=62478 rcvdpkt=39463 appcat="unscanned" sentdelta=217907 rcvddelta=76604 masterdstmac="74:8e:f8:70:d7:00" dstmac="74:8e:f8:70:d7:00" dstserver=0""",
"host" : "10.234.0.2"
}
},
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "OkuASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.457Z",
"tags" : [
"_grokparsefailure"
],
"message" : "<189>date=2021-01-28 time=11:37:47 devname=\"FW1201135-Primary\" devid=\"FG3H0E5819904320\" logid=\"1501054802\" type=\"utm\" subtype=\"dns\" eventtype=\"dns-response\" level=\"notice\" vd=\"root\" eventtime=1611862667591410999 tz=\"-0800\" policyid=27 sessionid=618154418 srcip=172.16.0.10 srcport=31689 srcintf=\"vlan234\" srcintfrole=\"lan\" dstip=205.251.196.100 dstport=53 dstintf=\"port5\" dstintfrole=\"wan\" proto=17 profile=\"default\" srcmac=\"74:8e:f8:70:d7:00\" xid=51188 qname=\"setup.fe.apple-dns.net\" qtype=\"A\" qtypeval=1 qclass=\"IN\" ipaddr=\"17.248.242.6, 17.248.242.13, 17.248.242.49, 17.248.242.45, 17.248.242.7, 17.248.242.47, 17.248.242.15, 17.248.242.44\" msg=\"Domain is monitored\" action=\"pass\" cat=52 catdesc=\"Information Technology\"",
"host" : "10.234.0.2"
}
},
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "O0uASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.470Z",
"tags" : [
"_grokparsefailure"
],
"message" : """<189>date=2021-01-28 time=11:37:47 devname="FW1201135-Primary" devid="FG3H0E5819904320" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1611862667611819056 tz="-0800" srcip=10.200.1.39 srcport=49400 srcintf="ssl.root" srcintfrole="undefined" dstip=10.10.2.12 dstport=389 dstintf="vlan234" dstintfrole="lan" srcuuid="65aca018-0b02-51ea-41b0-2ce579685006" dstuuid="8ec351d2-188a-51ea-166f-5aa96f50a45d" poluuid="3b7da292-0fc1-51ea-66fe-def64f6b37fc" sessionid=618144865 proto=17 action="accept" user="rharmsen" group="SSLVPN-EduUsers" authserver="ad13srv-LDAP" policyid=14 policytype="policy" service="LDAP_UDP" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=180 sentbyte=226 rcvdbyte=176 sentpkt=1 rcvdpkt=1 appcat="unscanned" masterdstmac="74:8e:f8:70:d7:00" dstmac="74:8e:f8:70:d7:00" dstserver=0""",
"host" : "10.234.0.2"
}
},
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "PEuASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.488Z",
"tags" : [
"_grokparsefailure"
],
"message" : """<189>date=2021-01-28 time=11:37:47 devname="FW1201135-Primary" devid="FG3H0E5819904320" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1611862667621826300 tz="-0800" srcip=10.200.1.39 srcport=54224 srcintf="ssl.root" srcintfrole="undefined" dstip=10.10.2.12 dstport=53 dstintf="vlan234" dstintfrole="lan" srcuuid="65aca018-0b02-51ea-41b0-2ce579685006" dstuuid="8ec351d2-188a-51ea-166f-5aa96f50a45d" poluuid="3b7da292-0fc1-51ea-66fe-def64f6b37fc" sessionid=618144864 proto=17 action="accept" user="rharmsen" group="SSLVPN-EduUsers" authserver="ad13srv-LDAP" policyid=14 policytype="policy" service="DNS" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=180 sentbyte=76 rcvdbyte=128 sentpkt=1 rcvdpkt=1 appcat="unscanned" masterdstmac="74:8e:f8:70:d7:00" dstmac="74:8e:f8:70:d7:00" dstserver=0""",
"host" : "10.234.0.2"
}
},
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "PUuASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.488Z",
"tags" : [
"_grokparsefailure"
],
"message" : """<189>date=2021-01-28 time=11:37:47 devname="FW1201135-Primary" devid="FG3H0E5819904320" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1611862667621827611 tz="-0800" srcip=10.200.1.20 srcport=54833 srcintf="ssl.root" srcintfrole="undefined" dstip=10.10.2.12 dstport=53 dstintf="vlan234" dstintfrole="lan" srcuuid="65aca018-0b02-51ea-41b0-2ce579685006" dstuuid="8ec351d2-188a-51ea-166f-5aa96f50a45d" poluuid="3b7da292-0fc1-51ea-66fe-def64f6b37fc" sessionid=618144870 proto=17 action="accept" user="lhooker" group="SSLVPN-Users" authserver="ad13srv-LDAP" policyid=14 policytype="policy" service="DNS" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=180 sentbyte=67 rcvdbyte=83 sentpkt=1 rcvdpkt=1 appcat="unscanned" masterdstmac="74:8e:f8:70:d7:00" dstmac="74:8e:f8:70:d7:00" dstserver=0""",
"host" : "10.234.0.2"
}
},
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "PkuASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.579Z",
"tags" : [
"_grokparsefailure"
],
"message" : "<189>date=2021-01-28 time=11:37:47 devname=\"FW1201135-Primary\" devid=\"FG3H0E5819904320\" logid=\"1501054802\" type=\"utm\" subtype=\"dns\" eventtype=\"dns-response\" level=\"notice\" vd=\"root\" eventtime=1611862667719241209 tz=\"-0800\" policyid=27 sessionid=618154422 srcip=172.16.0.10 srcport=25457 srcintf=\"vlan234\" srcintfrole=\"lan\" dstip=205.251.196.100 dstport=53 dstintf=\"port5\" dstintfrole=\"wan\" proto=17 profile=\"default\" srcmac=\"74:8e:f8:70:d7:00\" xid=9617 qname=\"gateway.fe.apple-dns.net\" qtype=\"A\" qtypeval=1 qclass=\"IN\" ipaddr=\"17.248.242.36, 17.248.242.48, 17.248.242.39, 17.248.242.10, 17.248.242.49, 17.248.242.4, 17.248.242.41, 17.248.242.17\" msg=\"Domain is monitored\" action=\"pass\" cat=52 catdesc=\"Information Technology\"",
"host" : "10.234.0.2"
}
},
{
"_index" : "logstash-2021.01.28-000001",
"_type" : "_doc",
"_id" : "P0uASncB3jQUJAgIdGvo",
"_score" : 1.0,
"_source" : {
"@version" : "1",
"type" : "syslog",
"@timestamp" : "2021-01-28T19:37:47.612Z",
"tags" : [
"_grokparsefailure"
],
"message" : "<189>date=2021-01-28 time=11:37:47 devname=\"FW1201135-Primary\" devid=\"FG3H0E5819904320\" logid=\"0317013312\" type=\"utm\" subtype=\"webfilter\" eventtype=\"ftgd_allow\" level=\"notice\" vd=\"root\" eventtime=1611862667747066026 tz=\"-0800\" policyid=27 sessionid=618154423 srcip=172.16.10.18 srcport=58935 srcintf=\"vlan234\" srcintfrole=\"lan\" dstip=17.248.242.43 dstport=443 dstintf=\"port5\" dstintfrole=\"wan\" proto=6 service=\"HTTPS\" hostname=\"gateway.icloud.com\" profile=\"wifi-default\" action=\"passthrough\" reqtype=\"direct\" url=\"https://gateway.icloud.com/\" sentbyte=517 rcvdbyte=0 direction=\"outgoing\" msg=\"URL belongs to an allowed category in policy\" method=\"domain\" cat=24 catdesc=\"File Sharing and Storage\"",
"host" : "10.234.0.2"
}
}
]
}
}
bump
We need to figure out why this endpoint is returning a 404 error - /internal/index-pattern-management/resolve_index/*
When I do a google search for the error I information about IIS and ASP.NET. How might these be related?
How is the url being rewritten?
1 Like
It should just be a simple URL Rewrite module within IIS 10.0. However, you gave me an idea.
I disabled the URL rewrite rule, restarted the website and I was able to not only see the logs* in the index pattern, but was able to create them.
BIG, and I mean, BIG QUESTION - Why?
The URL rewrite is simply: https://127.0.0.1:5601
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.