Hi Folks,
Can someone help me parsing the below logs using ingest pipeline?
//Ankita
Hello @Ankita_Pachauri
Please share the logs in text format using </> instead of screenshot.
For the message fields what are the target fields you are looking for? you need to share this as well.
Thanks!!
Hi.
Please find the log.
"2025/08/22 16:27:18 [info]
822064#822064: *47171 recv() failed (104:
Connection reset by peer) while reading client request fline, client: 127.0.0.1, server: localhost, request:
1 I.16)000310001110002110000 0001110000110001
@\u8003\u0003A@D0@@Bp@w@R7@f@DPp]
#lt1u8008p1u8814101u9001 velga11（oN］
@2\u001e\u001c00\u001e/\u80100po
fs@\u0014\u001c9\u00001™\u0013\u0002\u0013\u00
83\u9013\u90016,606+6/050
(+#$^\00 00 00000 0000 00011100001100
1110000131000011000011000011000011m0m
4)00003)00001000111000211002021--
//Ankita
I would also like to know how to apply logic to parse the logs using if condition.
For example. If log.file.path is XYZ use this grok filter if abc then other filter.
//Ankita
Hello @Ankita_Pachauri
About the question related to log file path you can use below method :
This will parse the records as per the condition set at Grok processor :
Thanks!!
Hi Tortoise,
I tried the above solution but suddenly I saw that the remaining logs stopped. I am sharing the artifacts.
Please find my filters below:
PUT _ingest/pipeline/Nginx
{
"processors": [
{
"set": {
"field": "app_name",
"value": "nb"
}
},
{
"rename": {
"field": "log.file.path",
"target_field": "logpath"
}
},
{
"grok": {
"field": "message",
"patterns": ["%{timestamp:timestamp}%{SPACE}\\[%{LOGLEVEL:nginx_error_level}\\]%{SPACE}%{POSINT:nginx_error_pid}#%{NUMBER:nginx_error_tid}\\:%{SPACE}\\*%{NUMBER:nginx_error_connection_id}%{SPACE}%{DATA:nginx_error_message},%{SPACE}client:%{SPACE}%{IP:nginx_access_remote_ip},%{SPACE}server:%{SPACE}%{DATA:server},%{SPACE}request:%{SPACE}%{DATA:request},%{SPACE}upstream:%{SPACE}%{DATA:upstream},%{SPACE}host:%{SPACE}%{GREEDYDATA:host_url}"],
"pattern_definitions": {
"timestamp": "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}"
},
"if": "ctx.logpath != null && ctx.logpath.endsWith('error.log')&& ctx.message.contains('upstream')"
}},
{
"grok": {
"field": "message",
"patterns": ["%{timestamp:timestamp}%{SPACE}\\[%{LOGLEVEL:nginx_error_level}\\]%{SPACE}%{POSINT:nginx_error_pid}#%{NUMBER:nginx_error_tid}\\:%{SPACE}\\*%{NUMBER:nginx_error_connection_id}%{SPACE}%{DATA:nginx_error_message},%{SPACE}client:%{SPACE}%{IP:nginx_access_remote_ip},%{SPACE}server:%{SPACE}%{DATA:server},%{SPACE}request:%{SPACE}%{DATA:request},%{SPACE}host:%{SPACE}%{GREEDYDATA:host_url},%{SPACE}referrer:%{SPACE}%{GREEDYDATA:referrer}"],
"pattern_definitions": {
"timestamp": "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}"
},
"if": "ctx.logpath != null && ctx.logpath.endsWith('error.log')&& ctx.message.contains('referrer')"
}
},
{
"grok": {
"field": "message",
"patterns": ["%{timestamp:timestamp}%{SPACE}\\[%{LOGLEVEL:nginx_error_level}\\]%{SPACE}%{POSINT:nginx_error_pid}#%{NUMBER:nginx_error_tid}\\:%{SPACE}\\*%{NUMBER:nginx_error_connection_id}%{SPACE}%{DATA:nginx_error_message},%{SPACE}client:%{SPACE}%{IP:nginx_access_remote_ip},%{SPACE}server:%{SPACE}%{GREEDYDATA:server}"],
"pattern_definitions": {
"timestamp": "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}"
},
"if": "ctx.logpath != null && ctx.logpath.endsWith('error.log')"
}},
{
"grok": {
"field": "message",
"patterns": ["%{timestamp:timestamp}%{SPACE}\\[%{LOGLEVEL:nginx_error_level}\\]%{SPACE}%{POSINT:nginx_error_pid}#%{NUMBER:nginx_error_tid}\\:%{SPACE}\\*%{NUMBER:nginx_error_connection_id}%{SPACE}%{DATA:nginx_error_message},%{SPACE}client:%{SPACE}%{IP:nginx_access_remote_ip},%{SPACE}server:%{SPACE}%{DATA:server},%{SPACE}request:%{SPACE}%{DATA:request},%{SPACE}host:%{SPACE}%{GREEDYDATA:host_url}"],
"pattern_definitions": {
"timestamp": "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}"
},
"if": "ctx.logpath != null && ctx.logpath.endsWith('error.log')"
}},
{
"grok": {
"field": "message",
"patterns": ["%{timestamp:timestamp}%{SPACE}\\[%{LOGLEVEL:nginx_error_level}\\]%{SPACE}%{POSINT:nginx_error_pid}#%{NUMBER:nginx_error_tid}\\:%{SPACE}\\*%{NUMBER:nginx_error_connection_id}%{SPACE}%{DATA:nginx_error_message},%{SPACE}client:%{SPACE}%{IP:nginx_access_remote_ip},%{SPACE}server:%{SPACE}%{DATA:server},%{SPACE}request:%{SPACE}%{GREEDYDATA:request}"],
"pattern_definitions": {
"timestamp": "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}"
},
"if": "ctx.logpath != null && ctx.logpath.endsWith('error.log')"
}},
{
"grok": {
"field": "message",
"patterns": ["%{timestamp:timestamp}%{SPACE}\\[%{LOGLEVEL:nginx_error_level}\\]%{SPACE}%{POSINT:nginx_error_pid}#%{NUMBER:nginx_error_tid}\\:%{SPACE}\\*%{NUMBER:nginx_error_connection_id}%{SPACE}client%{SPACE}%{IP:nginx_access_remote_ip}%{SPACE}%{GREEDYDATA:nginx_error_message}"],
"pattern_definitions": {
"timestamp": "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}"
},
"if": "ctx.logpath != null && ctx.logpath.endsWith('error.log')"
}},
{
"remove": {
"field": "timestamp"
}
}
]
}
I am also sharing the kind of logs I am getting :
2025/08/22 20:59:59 [info] 803167#803167: *58485 client closed connection while SSL handshaking, client: 10.144.9.236, server: 0.0.0.0:443"
2025/08/22 20:59:55 [info] 168197#168197: *3955272 client 10.199.10.109 closed keepalive connection
2025/08/22 20:59:49 [info] 337777#337777: *2797419 SSL_do_handshake() failed (SSL: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:SSL alert number 48) while SSL handshaking, client: 10.196.170.140, server: 0.0.0.0:443
2025/08/22 20:55:35 [info] 337781#337781: *2795542 client closed connection while waiting for request, client: 127.0.0.1, server: 0.0.0.0:443
2025/08/22 20:55:35 [info] 337782#337782: *2795544 recv() failed (104: Connection reset by peer) while reading client request line, client: 127.0.0.1, server: localhost, request: "�fS�>:���Ȫ�d� � 1�P"�̭�7QO�Q�k �Q�Bд-��D a��� I�(�wRf�4��@˓"�,�0�+�/̨̩�$�(�#�'��kg�� "
2025/08/22 19:01:33 [error] 819896#819896: nb-n.com could not be resolved (110: Operation timed out)
2025/08/22 03:44:01 [emerg] 168195#168195: open() "/var/log/nginx/access.log" failed (13: Permission denied)
2025/08/22 03:44:01 [notice] 337782#337782: reopening logs
2025/08/22 00:06:08 [warn] 337777#337777: usage report: host not found resolving endpoint "nginx-mgmt.local"
2025/08/21 22:06:08 [notice] 1628688#1628688: signal 29 (SIGIO) received
2025/08/21 22:06:08 [notice] 1628688#1628688: worker process 335317 exited with code 0
2025/08/21 22:06:08 [notice] 335317#335317: exit
2025/08/23 02:04:16 [info] 44778#44778: *2923718 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while connecting to upstream, client: 10.199.10.109, server: n.com, request: "GET /retail-app/ HTTP/1.1", upstream: "http.com/", host: "n.com"
2025/08/22 23:08:26 [info] 168196#168196: *4007399 epoll_wait() reported that client prematurely closed connection, client: 10.199.11.23, server: n.com, request: "GET /retail-app/ HTTP/1.1", host: "now.com", referrer: "https://now.h.com/auth/realms/retail/login-actions/reset-credentials?execution=f8d99ccd-147b-4fe6-89d3-cb24cbb6340e&client_id=bb-web-client&tab_id=d97GW2Mkgto"
2025/08/22 22:25:55 [info] 44783#44783: *2853809 epoll_wait() reported that client prematurely closed connection, client: 10.199.8.92, server: now.com, request: "GET /retail-app/ HTTP/1.1", host: "now.com"
2025/08/22 20:13:31 [info] 168196#168196: *3933699 recv() failed (104: Connection reset by peer), client: 10.199.48.219, server: api-now.com, request: "GET /"
Help me write a generic sort of filter for the above logs. Because once I executed the pipeline. The only logs I can see was error.log and no other log was coming. I deleted the pipeline configuration and again logs started flowing for all.
//Ankita
I don't think that is possible. Especially if you are saying all these different types of sources are coming in from one input.
You are going to need to
- write a grok for each of the patterns you want to parse
2a) Then you will need to either all the groks in a single grok processor and they will be executed in order and the first on that matches will be used
2b) Or create grok processors and then use an if statement to execute say if you had simple tags or a simple way to create if statements.
It looks like you already started 2b) Process
That is probably an error in your if statements...
I think @Tortoise was giving you an example, not expecting it to work for everything.
There is reall nothing special / magic here it's just "Roll up your sleeves work" 
I would perhaps use an LLM to help you write the grok statements for all the different types... I doubt we volunteers have time to do that all for you.
1 Like
Ok thanks @stephenb for your response