Filebeat Apache Log fields (remote_ip, geo etc)

[radu990]

Hello,

Sorry for bothering. I'm a new user for ELK. I have managed to enable Apache module for Filebeat and I'm seeing the Apache logs coming in Discover Section in Kibana. However the remote_ip and other fields from Apache are being containted in the message field. How can I get the remote_ip and other fields mapped correctly? Mainly remote_ip, User, URL (GET /mail) ...

Thank you in advance!

==============================
Example of a Log in Discover in Kibana

@timestamp
Feb 9, 2021 @ 16:46:36.153

_id
Xhx5h3cB1OZ-J5vvq_rA

_index
filebeat-7.10.2-2021.02.09-000001

_score

_type
_doc

agent.ephemeral_id
c75f9d91-47af-49eb-8797-79920a1308a7

agent.hostname
Server

agent.id
77a89e44-7cec-4d01-8b8f-8195dd8c3679

agent.name
Server

agent.type
filebeat

agent.version
7.10.2

ecs.version
1.5.0

error.message
Provided Grok expressions do not match field value: [85.85.85.85 webmail.company.com "CN=User/O=Company/C=CZ" [14/Dec/2020:05:58:18 +0100] "GET /mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries&PresetFields=DBQuotaInfo;1,FolderName;($Inbox),UnreadCountInfo;1,SearchSort;DateD,s_UsingHttps;1,noPI;1&TZType=UTC&Start=1&Count=23&resortdescending=6 HTTP/1.1" 200 2054 "https://webmail.company.com/mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=l_ScriptFrame&l=en&gz&CR&MX&TSF=20170318T181650,92Z&TSX=20180206T185427,18Z&EFF=%2FiNotes%2FForms9_x&charset=UTF-8&charset=UTF-8&KIC&ua=safari&pt" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" 31

event.dataset
apache.access

event.ingested
Feb 9, 2021 @ 16:47:13.932

event.module
apache

fileset.name
access

host.architecture
x86_64

host.hostname
Server

host.id
fec37629-61fa-466c-b662-9537045df62a

host.ip
fe80::c573:3ba4:33e1:4ede, 192.168.1.13, fe80::5efe:c0a8:10d, fe80:7f:fffe

host.mac
00:15:5d:01:19:1c, 00:00:00:00:00:00:00:e0, 00:00:00:00:00:00:00:e0

host.name
Server

host.os.build
7601.24546

host.os.family
windows

host.os.kernel
6.1.7601.24545 (win7sp1_ldr_escrow.200102-1707)

host.os.name
Windows Server 2008 R2 Enterprise

host.os.platform
windows

host.os.version
6.1

input.type
log

log.file.path
D:\Lotus\Domino\data\weblogs\access12142020.log

log.offset
8,054,853

message
85.85.85.85 webmail.company.com "CN=User/O=Company/C=CZ" [14/Dec/2020:05:58:18 +0100] "GET /mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries&PresetFields=DBQuotaInfo;1,FolderName;($Inbox),UnreadCountInfo;1,SearchSort;DateD,s_UsingHttps;1,noPI;1&TZType=UTC&Start=1&Count=23&resortdescending=6 HTTP/1.1" 200 2054 "https://webmail.company.com/mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=l_ScriptFrame&l=en&gz&CR&MX&TSF=20170318T181650,92Z&TSX=20180206T185427,18Z&EFF=%2FiNotes%2FForms9_x&charset=UTF-8&charset=UTF-8&KIC&ua=safari&pt" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" 31

service.type
apache

suricata.eve.timestamp
Feb 9, 2021 @ 16:46:36.153

[radu990]

Looks Like I need a grok pattern

Can anybody please help with the grok pattern for below example of logs?

85.85.85.85 webmail.company.com "CN=First Last/O=Company/C=CZ" [14/Dec/2020:05:58:18 +0100] "GET /mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries&PresetFields=DBQuotaInfo;1,FolderName;($Inbox),UnreadCountInfo;1,SearchSort;DateD,s_UsingHttps;1,noPI;1&TZType=UTC&Start=1&Count=23&resortdescending=6 HTTP/1.1" 200 2054 "https://webmail.company.com/mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=l_ScriptFrame&l=en&gz&CR&MX&TSF=20170318T181650,92Z&TSX=20180206T185427,18Z&EFF=%2FiNotes%2FForms9_x&charset=UTF-8&charset=UTF-8&KIC&ua=safari&pt" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" 125 INOTES_LOGIN_ID=First%20Last; Shimmer=SI_TLM:20210209T072811%2C40Z&ST_Counter:3&LAO:mail&SAB:1&CS_TLM:20210209T072831%2C15Z&V_TLM:20210210T080147%2C82Z&DMS:5&ui:X&MOTLM:20210129T113159%2C00Z&DBQS:1503571%2C%207168000%2C%206963200%2C%200%2C%201503571&SPRKL:1&KOSCZ:GTB&FISD:1; INOTES_LOGIN_ID=First%20Last; DWAShared=0; DWAMode=0; INOTES_LOGIN_ID=First%20Last; DWAShared=0; DWAMode=0; LtpaToken2=FpoGJJz33bYLI+CtWy6OlIgoTJouNGEiduvxvQbcN8HRI7K6LThCsb1Dl8CzN72Zi05RGOUmQRMiOQcTk1norKHi6SbkEGI6GlXzjSIweBRSc8c+XPyAwA44PKPbu3WzrPfR0+uoC0sgTPvochvQ/VfPL/sSaqUFoRswRwyI+UeaOwTs/DvKiWLCpiKrVkFk3SmDjrxPBHb/WiL5nDkpp8Dsjjxnlo4vpx7BdOoVNai1jybvHkW28KXxkb21o8SSpmU7ZFdHyZFjDWCYuuCVOx7asV/q4a3lWdxlPfWdPcUguHML+xDmsrMPm6fTUSKeKIKdQEPr6VDmitBi7Z5URIlkRrUyslkTcc28y6fQir3Y20Hc9TmOvwaBlG/ehnpv; LtpaToken=0x4JJ4oWKojdqoz08Ng+MRUkkJq2vYGLGN9lp8HL8FxbD+xnivE7qzCzf92Q6x5OAPOBFRNgxd3Qg225zLwnJFWO0lGeIweH8VDgyWOMImNe6E9z9HBnQAN43vQ2uwtpv3X5E5DN0oLIPKLxAkqsHUDJqJ0SE6NZ6UnfLoR82JyjZVC/s6QEov5DNdpAY/o2Gxh0vWmE+wuQGuCh4mVCIP9KU/dbX4F0Ld9JEExzIpkdzKELibU2Akov0Krv0eWADSV++m/5ECLpaf6N6/VzkZEkt5XoOoL6OD/6ni4zojvo3O+X9Bn7Mdk2MnsQ1AccIohj5eN8Oi81QbD0a9b7jw==; ShimmerS=ET:20210210T114045%2c00Z&R:0&AT:M" "D:/Lotus/Domino/Data/mail/User.nsf"

What I would need is Client IP (85.85.85.85), VirtualHostname (webmail / webmail.company.com) , User (part after CN=, First Last), Time (14/Dec/2020:05:58:18), URL (GET /mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries&PresetFields=DBQuotaInfo;1,FolderName; ... ) and the Device Info ( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" )

I know it should start with below, however I can't get anyhow the User name to proceed with [%{HTTPDATE:timestamp}] and possible next would be "(?:%{WORD:verb} %{NOTSPACE:request} and not sure how to get the Device info. Any help would be appreciated!

%{IPORHOST:clientip} %{WORD:VirtualHost} ???

[ChrsMark (Chris Mark)]

Hi!

You can find the grok pattern which Filbeat apache module uses at beats/pipeline.yml at 829c3b7dcc6365161d83a3b10f05a9f9990f36c3 · elastic/beats · GitHub.

Also have a look into the sample logs that are used for testing at beats/filebeat/module/apache/access/test at 1ce20cbe17269eafc8dbb0d859413643e9a10886 · elastic/beats · GitHub.

If in your case, the logs are different and cannot be parsed by the pipeline then you might need to create custom grok pattern from scratch or update the one of the apache's module so as to cover your case.

C.

[radu990]

Thank you very much Chris. Very useful! Using the grok patterns from the link you've provided I have managed to get my right grok pattern.

%{IPORHOST:clientip} %{IPORHOST:destination.domain} "CN=%{DATA:username}" [%{HTTPDATE:apache.access.time}] "(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-)?" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-) ("%{DATA:http.request.referrer}") ("%{DATA:user_agent.original}")