Grok pattern failing for apache custom logs

Elastic Stack Beats
,
1

i am facing issue for my grok is failing for the apache custom logs as beolw

10.52.245.67 - - [12/Jul/2023:08:08:51 +0800] uibau1a "GET /login/runtime.6b0e772316ccb94a9291.js HTTP/1.1" 200 2289bytes "10.168.224.18, 10.52.80.51" "ZK3vE3gceATC133RawX2NAAAAJU" [-] 11ms
10.52.245.67 - - [12/Jul/2023:08:08:52 +0800] uibau1a "GET /login/assets/fonts/OpenSans/OpenSans-SemiBold.woff HTTP/1.1" 200 18696bytes "10.168.224.18, 10.52.80.51" "ZK3vFHgceATC133RawX2NgAAAJE" [-] 0ms

please help for creating the grok for this custom logs.

2

Hi Shailendra,

We have an updated Integration available for Apache_Tomcat.
Would suggest you to use that as it covers a lot more grok patterns.

3

thanks @ishleenk17 , i have gone through it and its usefull . i am also still facing the grok failures below is my custom log format from the apache logs.

LogFormat "%h %l %u %t %{BALANCER_WORKER_ROUTE}e \"%r\" %>s %bbytes \"%{X-Forwarded-For}i\" \"%{UNIQUE_ID}e\" [%{Authorization}i] %{ms}Tms" common

the issue is that in the x-forward , there are two IP's are coming after "," as per logs pasted above but grok is failing for recognized this IP . can suggest further ?

4

I have checked the log format. The grok is failing due to %{BALANCER_WORKER_ROUTE}e and %bbytes, not because the issue is in the x-forward.

There is one extra custom field %{BALANCER_WORKER_ROUTE}e (uibau1a) and should be 18696 instead of 18696bytes.

I would recommend performing using supported log formats. Please refer to this link: Apache Tomcat | Elastic docs

5

thank you @Harnish_Chavda for answering it. yes, we have custom logs where route is added in the apache custom logs.

  1. do you mean that %{BALANCE_WORKER_ROUTE}e => agent name should not be here ?
    2. %bbytes. -> 18696bytes ,

so 18696 is from %b => 18696 and %bytes => bytes . pl corrrect me if i am understanding correctly or anything can be imporve . thanks

6

Yes, you are correct. You can follow this log format %h %l %u %t "%r" %s %b %A %X %T "%{Referer}i" "%{User-Agent}i" X-Forwarded-For="%{X-Forwarded-For}i"

7

okay, so u mean to say it would be same patterns what we have defined and

%{BALANCE_WORKER_ROUTE}e => "%r"

would be replacing it ?
do we have any other ways to generate the dummy logs for these formats and test our grok ? appreciate your response . Thank you .

8

No, the field that you have defined %{BALANCE_WORKER_ROUTE}e represents the route of the balancer worker that handled the request. While %r represents the request line from the client, including the method, URI, and HTTP version that was already presented in your mentioned log format => %h %l %u %t %{BALANCER_WORKER_ROUTE}e \"%r\" %>s %bbytes "%{X-Forwarded-For}i" "%{UNIQUE_ID}e" [%{Authorization}i] %{ms}Tms.

do we have any other ways to generate the dummy logs for these formats and test our grok ?

If you mean to generate dummy logs in the current instance:

You don't need to perform any special activity. Access logs are generated automatically by Tomcat whenever a request is made to the server. However, you need to ensure that the access logging configuration is properly set up. Here's what you can do:

Configure Access Logging : Access logging is configured through the server.xml file of your Tomcat installation

<Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">
    <!-- Other configurations -->

    <!-- Configure Access Logging -->
    <Valve className="org.apache.catalina.valves.AccessLogValve"
           directory="logs" prefix="access_log" suffix=".txt"
           pattern="%h %l %u %t "%r" %s %b %A %X %T "%{Referer}i" "%{User-Agent}i" X-Forwarded-For="%{X-Forwarded-For}i"" />
</Host>

Restart Tomcat : After making changes to the server.xml file, you need to restart your Tomcat server for the changes to take effect.

Make Requests : Once Tomcat is up and running, simply make requests to your web applications hosted on Tomcat. You can do this by entering URLs in your web browser, using tools like curl , or sending HTTP requests through any other method.

View Access Logs : Tomcat will automatically generate access logs based on the configured pattern and save them in the specified logs directory. You can navigate to the logs directory within your Tomcat installation to find the access log files. The files will be named based on the prefix and suffix you specified in the <Valve> configuration.

If you want a docker configurations please use the docker image to generate the dummy logs! Make sure to follow the above-mentioned steps in the docker container!

Please find the example logs for

%h %l %u %t "%r" %s %b %A %X %T "%{Referer}i" "%{User-Agent}i" X-Forwarded-For="%{X-Forwarded-For}i" =>

0.0.0.0 - admin [02/Mar/2023:18:58:17 +0530] "POST /host-manager/images/asf-logo.svg HTTP/1.1" 200 20486 0.0.0.0 + "http://localhost:8080/host-manager/html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/0.0.0.0 Safari/537.36" X-Forwarded-For="

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.