Data Set
192.168.0.0 - - [03/Feb/2022:11:14:38 +0000] \"/I/went/to/your/house.html HTTP/1.1\" 200 109 \"https://https://grokdebug.herokuapp.com//\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36 Edg/97.0.1072.76\" \"-\"\"TLSv1.3\" \"TLS_AES_256_GCM_SHA384\" \"CN=DOE.JOHN.123456789,OU=SOMEWHERE,OU=PK,OU=USDA,O=Freestyle,C=US\"
Using the Grok Debugger in Kibana Console works with the below pattern.
(%{NGINX_HOST} )?\"?(?:%{NGINX_ADDRESS_LIST:nginx.access.remote_ip_list}|%{NOTSPACE:source.address}) - (-|%{DATA:user.name}) \[%{HTTPDATE:nginx.access.time}\] \\"%{DATA:nginx.access.info}\\" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \\"(-|%{DATA:http.request.referrer})\\" \\"(-|%{DATA:user_agent.original})\\" \\"-\\"\\"(-|%{DATA:ssl.protocol})\\" \\"(-|%{DATA:ssl.cipher})\\" \\"(-|%{DATA:ssl.client})\\"
The Custom Patterns are default for the Nginx access pipeline
NGINX_HOST (?:%{IP:destination.ip}|%{NGINX_NOTSEPARATOR:destination.domain})(:%{NUMBER:destination.port})?
NGINX_NOTSEPARATOR "[^\t ,:]+"
NGINX_ADDRESS_LIST (?:%{IP}|%{WORD})("?,?\s*(?:%{IP}|%{WORD}))*
When updating the ingest pipeline for the Filebeat Nginx Module (filebeat-7.16.2-nginx-access-pipeline) that matches the grok pattern from above, it does errors out with Invalid JSON string, which I think means, something is not escaped properly. I tested that theory by removing some of the backslashes and it worked. However, when I do that it no longer works in the Kibana console and subsequently does not work with the data. I have banged my head trying to figure this.
I am going straight from Filebeat to Elasticsearch.