Grok formatting

[Paulo_Neto]

I am trying to format the text according to the example below:

%{TIMESTAMP_ISO8601:time} %{WORD:method} %{URIPATH:uri_requested} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:client_ip} %{NOTSPACE:http_version} %{NOTSPACE:user_agent} %{URI:referrer_url} %{IPORHOST:host} %{NUMBER:http_status_code} %{NUMBER:protocol_substatus_code} %{NUMBER:win32_status} %{NUMBER:bytes_sent} %{NUMBER:bytes_received} %{NUMBER:time_taken}

However, I am not successful, I keep getting the error message: "provided Grok expressions do not match field value"

Here we can see a sample log from IIS:

"2022-08-17 01:29:09 GET /root/ 80 email@example.com IP_EXAMPLE HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/104.0.0.0+Safari/537.36 http://example.site.com.br/ 200 0 0 10 20 431"

How can I include the correct Grok formatting in ELK Cloud?

[Badger]

@Paulo_Neto

%{IPORHOST:host}

There is nothing in your IIS log to match that. You should remove it. Also, hostnames cannot contain underscores, so IP_EXAMPLE does not match %{IPORHOST}

[Paulo_Neto]

Hello Badger, thank you for your reply.
The IP_EXAMPLE field, was supposed to be an example only and was used to mask the real data because it is sensitive.
Thanks for the comment about the unused field, I hadn't paid attention to that, I've already removed the field, everything else is ok?

How can I add this grok pattern?
I just did the tests with grok debugger

[Badger]

@Paulo_Neto

The IP_EXAMPLE field, was supposed to be an example only and was used to mask the real data because it is sensitive.

I understand you may not want to reveal private IP addresses, but you can always mask it with something like 127.0.0.1 which is a valid address. You would configure a grok filter using

grok { match => { "message" => "%{TIMESTAMP_ISO8601:time} %{WORD:method} %{URIPATH:uri_requested} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:client_ip} %{NOTSPACE:http_version} %{NOTSPACE:user_agent} %{URI:referrer_url} %{NUMBER:http_status_code} %{NUMBER:protocol_substatus_code} %{NUMBER:win32_status} %{NUMBER:bytes_sent} %{NUMBER:bytes_received} %{NUMBER:time_taken}" } }

[Paulo_Neto]

I see, thanks for the suggestion, I will use it that way.

Where can I include this?

From ELK Cloud I see that it is possible to include new patterns in ingest pipelines, as you can see below.

[Badger]

Since you posted in the logstash forum I assumed you were running logstash. If you want to use an ingestion processor then you should enquire in the elasticsearch forum.

[Paulo_Neto]

No, I am doing a test with the ELK cloud, if all goes well I will check how best to deploy it.

Thanks for the warning, I have changed the category of the topic.