Grok pattern matching both the logs

Elastic Stack Logstash
1

Hi i have two types of logs, which differ just by one value at the end. can i get a single grok pattern to match both the files

[08/Sep/2023:13:28:50 +0530] | 404 | 1 ms | 773 B | 127.0.0.1 |          - | - | - | "GET /status HTTP/1.1" | -

The last - in the field can be an email id or a -
And another log is below

	
[08/Sep/2023:13:28:53 +0530] | 200 | 0 ms | 632 B | 172.31.6.240 | - | - | - | "GET /server-status?auto HTTP/1.1"

I've a grok for the entire thing except the last value

\[(?<short_date>%{HTTPDATE:date})\] \| %{NUMBER:response} \| (?<duration>%{NUMBER} %{WORD}) \| (?<bytes>%{NUMBER} %{WORD}|%{DATA}) \| %{IP:clientip} \| (%{SPACE}%{DATA:remoteip}|%{SPACE}%{IP:remoteip}) \| (%{WORD:token}|%{DATA:token}) \| (?<tag1>%{NUMBER} %{WORD}|%{DATA}) \| \"(?<method>%{WORD}) (?<url>%{URIPATHPARAM:service}) (?:HTTP/%{NUMBER:http_version})\"

What can i add to match both the logs?

2

Are you referring to log lines above. They Seems to already match.

[
  {
    "short_date": "08/Sep/2023:13:28:50 +0530",
    "date": "08/Sep/2023:13:28:50 +0530",
    "response": 404,
    "duration": "1 ms",
    "bytes": "773 B",
    "clientip": "127.0.0.1",
    "remoteip": "-",
    "token": "-",
    "tag1": "-",
    "method": "GET",
    "url": "/status",
    "service": "/status",
    "http_version": 1.1
  },
  {
    "short_date": "08/Sep/2023:13:28:53 +0530",
    "date": "08/Sep/2023:13:28:53 +0530",
    "response": 200,
    "duration": "0 ms",
    "bytes": "632 B",
    "clientip": "172.31.6.240",
    "remoteip": "-",
    "token": "-",
    "tag1": "-",
    "method": "GET",
    "url": "/server-status?auto",
    "service": "/server-status?auto",
    "http_version": 1.1
  }
]
3

there is a difference of | - in both the log lines
in one type of logs there is nothing after http version. and in another there is | - or | email@test.com

4

You can have 2 patterns like

        "\[%{HTTPDATE:date}\] \| %{NUMBER:response:int} \| %{NUMBER:duration:int} ms \| %{NUMBER:bytes:int} B \| %{IP:clientip} \| (-|%{IP:remoteip}) \| (-|%{DATA:token}) \| (-|%{NUMBER} %{WORD}|%{DATA}) \| \"%{WORD:method} %{URIPATHPARAM:service}(?: HTTP/%{NUMBER:http_version})?(?: \| %{GREEDYDATA:extra_data}(?: \| %{EMAILADDRESS:email}))?",
        "\[%{HTTPDATE:date}\] \| %{NUMBER:response:int} \| %{NUMBER:duration:int} ms \| %{NUMBER:bytes:int} B \| %{IP:clientip} \| (-|%{IP:remoteip}) \| (-|%{DATA:token}) \| (-|%{NUMBER} %{WORD}|%{DATA}) \| \"%{WORD:method} %{URIPATHPARAM:service}(?: HTTP/%{NUMBER:http_version})?"
5

can i have two match in same grok? because i want these both logs to be redirected to one index. i have two indexes, with a difference of just one field. one have email and one has nothing. i used to merge the two indexes in one data view. but now i want them to be in a single index and transport the data to another tool for further analysis.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.