Add new filter questions

Elastic Stack Logstash
1

Hi.

I'm trying to parse the id filed in my message column so that I can group by id to have a number of unique transactions in my dashboard.

Currently, the filter is:

    if [type] == "web_industry_log" {
        mutate {
            replace => { 'host' => 'appserver.datacentre.example.com' }
            add_field => { 'environment' => 'production'
                           'service' => 'web_industry'
            }
        }

        grok {
            match => { 
                "message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp }%{SPACE}%{DATA:class}%{SPACE}-%{SPACE}%{GREEDYDATA:message}"
            }
            overwrite => [ 'message' ]
        }

        date {
            match => ["timestamp", "ISO8601"]
            target => "@timestamp"
        }
    }

Log output:

{
  "_index": "elk-log-index-2019.w42",
  "_type": "_doc",
  "_id": "PM92zG0oBL5xOADC",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-10-14T22:49:58.357Z",
    "@version": "1",
    "environment": "production",
    "timestamp": "2019-10-15 11:49:58,357",
    "host": "appserver.datacentre.example.com",
    "type": "web_industry_log",
    "message": "Transaction traderStockMonitor a19d5f | id: 1930808 timestamp: 2019-10-15T11:49:58.000+13:00 user: Customer name here Ltd (5648575) request address: null login id: null",
    "log-level": "INFO",
    "class": "transaction.TransactionDocumentPersisterImpl",
    "path": "/mnt/logs/app/industry/industry.log",
    "service": "web_industry"
  },
  "fields": {
    "@timestamp": [
      "2019-10-14T22:49:58.357Z"
    ]
  },
  "highlight": {
    "environment": [
      "@kibana-highlighted-field@production@/kibana-highlighted-field@"
    ],
    "service": [
      "@kibana-highlighted-field@web_industry@/kibana-highlighted-field@"
    ],
    "class.keyword": [
      "@kibana-highlighted-field@transaction.TransactionDocumentPersisterImpl@/kibana-highlighted-field@"
    ],
    "log-level.keyword": [
      "@kibana-highlighted-field@INFO@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1571093398357
  ]
}

So, the message column "message": "Transaction traderStockMonitor a19d5f | id: 1930808 timestamp: 2019-10-15T11:49:58.000+13:00 user: Customer name here Ltd (5648575) request address: null login id: null",. I need to filter the id: number as well, so it is searchable and then I can group by later on.

How to do that, please?

Thanks!

2

grok patterns are not anchored by default, so you could try something like

grok { match => { "message" => " id: %{NUMBER:id}" } }

If the id is something with numerical meaning, like the number of bytes in a response, then you could add :int to that so that you can apply aggregations like average id. But if it is just an identifier I would leave it as a string.

1 Like
3

hmmm Thanks! But it is not working.

grok {
            match => {
                "message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp }%{SPACE}%{NUMBER:id:int}%{SPACE}%{JAVACLASS:class}%{SPACE}-%{SPACE}%{GREEDYDATA:message}"
            }
            overwrite => [ 'message' ]

Then I restarted logstash and did a refresh field list. I still cannot see it.

Edit:

I've also tried with: "message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp }%{SPACE}, id %{NUMBER:id:int}%{SPACE}%{JAVACLASS:class}%{SPACE}-%{SPACE}%{GREEDYDATA:message}"

4

Oh.. I see... I managed to get it working. Will mark your answer as resolution. Thanks @Badger.

Edit:

Sorry... I was able to play around with the filter, but the id is still not working.

"message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp }%{SPACE}%{DATA:class}%{SPACE}-%{SPACE}Transaction%{SPACE}%{WORD:transaction-type}%{SPACE}%{WORD:transaction-reference}%{SPACE}|%{SPACE}id:%{SPACE}%{NUMBER:transaction-id:int}%{GREEDYDATA:message}"

transaction-type / transaction-reference are working... but id is not.

What am I doing wrong?

Edit 2:

The syntax on Grok Debug is working...

%{WORD:transaction-reference} \| id: %{NUMBER:transaction-id:int}

However, it does not work in my setup.

5

I've found the problem.

This final syntax works:

"message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp } %{DATA:class} - Transaction %{WORD:transaction-type} %{WORD:transaction-reference} \| id: %{NUMBER:transaction-id}%{GREEDYDATA:message}"

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.