# Add new filter questions
**URL:** https://discuss.elastic.co/t/add-new-filter-questions/203533
**Category:** Logstash
**Created:** [October 14, 2019, 11:08pm UTC](https://discuss.elastic.co/t/add-new-filter-questions/203533 "2019-10-14T23:08:39Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@sud0](https://discuss.elastic.co/u/sud0)
#### Post date: [October 14, 2019, 11:08pm UTC](https://discuss.elastic.co/t/add-new-filter-questions/203533/1 "2019-10-14T23:08:40Z")
Hi.
I'm trying to parse the id filed in my message column so that I can group by id to have a number of unique transactions in my dashboard.
Currently, the filter is:
```
if [type] == "web_industry_log" {
mutate {
replace => { 'host' => 'appserver.datacentre.example.com' }
add_field => { 'environment' => 'production'
'service' => 'web_industry'
}
}
grok {
match => {
"message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp }%{SPACE}%{DATA:class}%{SPACE}-%{SPACE}%{GREEDYDATA:message}"
}
overwrite => ['message']
}
date {
match => ["timestamp", "ISO8601"]
target => "@timestamp"
}
}
```
Log output:
```
{
"_index": "elk-log-index-2019.w42",
"_type": "_doc",
"_id": "PM92zG0oBL5xOADC",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-10-14T22:49:58.357Z",
"@version": "1",
"environment": "production",
"timestamp": "2019-10-15 11:49:58,357",
"host": "appserver.datacentre.example.com",
"type": "web_industry_log",
"message": "Transaction traderStockMonitor a19d5f | id: 1930808 timestamp: 2019-10-15T11:49:58.000+13:00 user: Customer name here Ltd (5648575) request address: null login id: null",
"log-level": "INFO",
"class": "transaction.TransactionDocumentPersisterImpl",
"path": "/mnt/logs/app/industry/industry.log",
"service": "web_industry"
},
"fields": {
"@timestamp": [
"2019-10-14T22:49:58.357Z"
]
},
"highlight": {
"environment": [
"@kibana-highlighted-field@production@/kibana-highlighted-field@"
],
"service": [
"@kibana-highlighted-field@web_industry@/kibana-highlighted-field@"
],
"class.keyword": [
"@kibana-highlighted-field@transaction.TransactionDocumentPersisterImpl@/kibana-highlighted-field@"
],
"log-level.keyword": [
"@kibana-highlighted-field@INFO@/kibana-highlighted-field@"
]
},
"sort": [
1571093398357
]
}
```
So, the message column `"message": "Transaction traderStockMonitor a19d5f | id: 1930808 timestamp: 2019-10-15T11:49:58.000+13:00 user: Customer name here Ltd (5648575) request address: null login id: null",`. I need to filter the `id: number` as well, so it is searchable and then I can `group by` later on.
How to do that, please?
Thanks!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 14, 2019, 11:35pm UTC](https://discuss.elastic.co/t/add-new-filter-questions/203533/2 "2019-10-14T23:35:30Z")
grok patterns are not anchored by default, so you could try something like
```
grok { match => { "message" => " id: %{NUMBER:id}" } }
```
If the id is something with numerical meaning, like the number of bytes in a response, then you could add :int to that so that you can apply aggregations like average id. But if it is just an identifier I would leave it as a string.
---
### Author:  [@sud0](https://discuss.elastic.co/u/sud0)
#### Post date: [October 15, 2019, 12:11am UTC](https://discuss.elastic.co/t/add-new-filter-questions/203533/3 "2019-10-15T00:11:22Z")
hmmm Thanks! But it is not working.
```
grok {
match => {
"message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp }%{SPACE}%{NUMBER:id:int}%{SPACE}%{JAVACLASS:class}%{SPACE}-%{SPACE}%{GREEDYDATA:message}"
}
overwrite => ['message']
```
Then I `restarted logstash` and did a `refresh field list`. I still cannot see it.
Edit:
I've also tried with: `"message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp }%{SPACE}, id %{NUMBER:id:int}%{SPACE}%{JAVACLASS:class}%{SPACE}-%{SPACE}%{GREEDYDATA:message}"`
---
### Author:  [@sud0](https://discuss.elastic.co/u/sud0)
#### Post date: [October 15, 2019, 2:10am UTC](https://discuss.elastic.co/t/add-new-filter-questions/203533/4 "2019-10-15T02:10:14Z")
Oh.. I see... I managed to get it working. Will mark your answer as resolution. Thanks @Badger.
Edit:
Sorry... I was able to play around with the filter, but the `id` is still not working.
`"message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp }%{SPACE}%{DATA:class}%{SPACE}-%{SPACE}Transaction%{SPACE}%{WORD:transaction-type}%{SPACE}%{WORD:transaction-reference}%{SPACE}|%{SPACE}id:%{SPACE}%{NUMBER:transaction-id:int}%{GREEDYDATA:message}"`
`transaction-type / transaction-reference` are working... but `id` is not.
What am I doing wrong?
Edit 2:
The syntax on [Grok Debug](https://grokdebug.herokuapp.com/) is working...
`%{WORD:transaction-reference} \| id: %{NUMBER:transaction-id:int}`
However, it does not work in my setup.
---
### Author:  [@sud0](https://discuss.elastic.co/u/sud0)
#### Post date: [October 15, 2019, 3:16am UTC](https://discuss.elastic.co/t/add-new-filter-questions/203533/5 "2019-10-15T03:16:41Z")
I've found the problem.
This final syntax works:
`"message" => "(?m)%{LOGLEVEL:log-level}%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:timestamp } %{DATA:class} - Transaction %{WORD:transaction-type} %{WORD:transaction-reference} \| id: %{NUMBER:transaction-id}%{GREEDYDATA:message}"`
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 12, 2019, 3:16am UTC](https://discuss.elastic.co/t/add-new-filter-questions/203533/6 "2019-11-12T03:16:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.