New fields

Elastic Stack Logstash
1

i have forwarded sys log messages to my ELK.
i have used grok to filter this logs and abstract the fields, but it didn't work.
i am sure with my grok.
when i forward the php logs directly to the ELK, with another grok filter, it works !!!
any idea, why syslog message with grok, to splitt fields is not working?

bellow a samble of the syslog message

<134>Jul 25 10:51:29 cisco tmm1[13548]: Rule /Common/elastic_log <HTTP_RESPONSE>: Client:x.x.x.x -- VS:y.y.y.y -- URL:www.aaaa.com/index.aspx -- username: -- Node:a.a.a.a -- RES:600

and the syslog grok filter is

match => { "message" => "<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco tmm1[%{BASE10NUM:tmm1_code}]: Rule /Common/elastic_log <HTTP_RESPONSE>: Client:%{IPV4:client_ip} -- VS:%{IPV4:server_ip} -- URL:%{DATA:uri} -- username:%{DATA:username} -- Node:%{IPV4:site_ip} -- RES:%{BASE10NUM:response}" }
}

2

It's a regexp, so you need to escape characters that have meaning in regexps, such as square brackets.

<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco tmm1\[%{BASE10NUM:tmm1_code}\]: Rule /Common/elastic_log <HTTP_RESPONSE>: Client:%{IPV4:client_ip} -- VS:%{IPV4:server_ip} -- URL:%{DATA:uri} -- username:%{DATA:username} -- Node:%{IPV4:site_ip} -- RES:%{BASE10NUM:response}
3

thanks a lot badger, it works great

4

please, i have updated my grok to
grok {
match => { "message" => "<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco %{DATA:tmm1_code}: Rule <HTTP_RESPONSE>: Client:%{IPV4:client_ipv4} -- Server:%{IPV4:virtual_server_ip} -- URL:%{DATA:request_uri} -- username:%{DATA:username} -- Node:%{IPV4:node_ipv4} -- RES:%{BASE10NUM:server_response_code} -- country_area:%{DATA:country}" }

it works fine, and it add all fields except "username" and "country" !!!!
all other fields appeared fine, like "elastics_code" , "request_uri", ................
any idea please?

5

In your example message from your first post, neither of those fields exist, so I am not surprised they do not match. Please show both the message and the grok pattern indented by 4 spaces, so that they look like this

<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco tmm1\[%{BASE10NUM:tmm1_code}\]: Rule /Common/elastic_log <HTTP_RESPONSE>: Client:%{IPV4:client_ip} -- VS:%{IPV4:server_ip} -- URL:%{DATA:uri} -- username:%{DATA:username} -- Node:%{IPV4:site_ip} -- RES:%{BASE10NUM:response}

rather than this

<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco tmm1[%{BASE10NUM:tmm1_code}]: Rule /Common/elastic_log <HTTP_RESPONSE>: Client:%{IPV4:client_ip} -- VS:%{IPV4:server_ip} -- URL:%{DATA:uri} -- username:%{DATA:username} -- Node:%{IPV4:site_ip} -- RES:%{BASE10NUM:response}

6

yes, i have updated the message :slight_smile:

example of the messsage is:

<134>Jul 29 11:06:09 cisco cisco[123]: Rule <HTTP_RESPONSE>: Client:196.153.4.34 -- Server:172.20.38.109 -- URL:cndc.aast.edu/images/s3.gif -- username: -- Node:172.20.39.75 -- RES:304 -- country_area:US

and the grok is

grok {
match => { "message" => "<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco %{DATA:tmm1_code}: Rule <HTTP_RESPONSE>: Client:%{IPV4:client_ipv4} -- Server:%{IPV4:virtual_server_ip} -- URL:%{DATA:request_uri} -- username:%{DATA:username} -- Node:%{IPV4:node_ipv4} -- RES:%{BASE10NUM:server_response_code} -- country_area:%{DATA:country}" }

again, all fields works fine, except these fields "username: and "country"

7

Please indent the message and the filter so that escaped characters are shown.

10 
<134>Jul 29 11:06:09 cisco cisco[123]: Rule <HTTP_RESPONSE>: Client:196.153.4.34 -- Server:172.20.38.109 -- URL:cndc.aast.edu/images/s3.gif -- username: -- Node:172.20.39.75 -- RES:304 -- country_area:US     


grok { match => { "message" => "<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco %{DATA:tmm1_code}: Rule <HTTP_RESPONSE>: Client:%{IPV4:client_ipv4} -- Server:%{IPV4:virtual_server_ip} -- URL:%{DATA:request_uri} -- username:%{DATA:username} -- Node:%{IPV4:node_ipv4} -- RES:%{BASE10NUM:server_response_code} -- country_area:%{DATA:country}"  }
11

You don't get a username field because there is no username field in the event.

I do not understand why DATA does not match US. You can use GREEDYDATA to pick up the US and the following whitespace. Alternatively use country_area:%{NOTSPACE:country}

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.