# Need help in using logstash with grok and fingerprint
**URL:** https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109
**Category:** Logstash
**Created:** [September 25, 2019, 9:45pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109 "2019-09-25T21:45:17Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Nara725](https://discuss.elastic.co/u/Nara725)
#### Post date: [September 25, 2019, 9:45pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/1 "2019-09-25T21:45:17Z")
Hi,
I'm new to logstash and i have the below requirement:
My logs look like below and i dont want to send all 3 lines to Kibana, instead want to send only one line for a given time range.
```
[2019-09-25 17:17:33.153] [logger] [info] normal_task: Output queue occupancy: 0.00.
[2019-09-25 17:17:34.154] [logger] [info] normal_task: Output queue occupancy: 0.00.
[2019-09-25 17:17:35.154] [logger] [info] normal_task: Output queue occupancy: 0.00.
[2019-09-25 17:17:36.155] [logger] [info] normal_task: Output queue occupancy: 0.00.
[2019-09-25 17:17:37.155] [logger] [info] normal_task: Output queue occupancy: 0.00.
```
I tried a combination of grok and fingerprint, it didnt worked as expected. Can i know how i can accomplish the above?
```
filter {
grok{
match => { "message" => "\[%{TIMESTAMP_ISO8601:time}\] %{DATA:log_message}"}
}
fingerprint {
source => "message"
target => "[@metadata][fingerprint]"
method => "MURMUR3"
}
}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 26, 2019, 4:39am UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/2 "2019-09-26T04:39:26Z")
The `message` field contains a varying timestamp so including this will not work. You will need to calculate the fingerprint based on a set of specific fields that defines what is unique. You should also not use a MURMUR3 hash as you will risk losing data as it is only 32-bit. I would recommend reading these blog posts:
> **[Little Logstash Lessons: Handling Duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)**
>
> Approaches for de-duplicating data in Elasticsearch using Logstash. We also go into examples of how you can use IDs in Elasticsearch Output.
> **[Efficient Duplicate Prevention for Event-Based Data in Elasticsearch](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch)**
>
> Need to prevent duplicates in the Elastic Stack with minimal performance impact? In this blog we look at different options in Elasticsearch and provide some practical guidelines.
---
### Author:  [@Nara725](https://discuss.elastic.co/u/Nara725)
#### Post date: [October 2, 2019, 4:55pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/4 "2019-10-02T16:55:51Z")
What if i include log\_message in fingerprint to generate the hash?
```auto
fingerprint {
source => "log_message"
target => "[@metadata][fingerprint]"
method => "MURMUR3"
}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 2, 2019, 5:20pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/5 "2019-10-02T17:20:51Z")
That might work, but I would still recommend not using MURMUR3 hash.
---
### Author:  [@Nara725](https://discuss.elastic.co/u/Nara725)
#### Post date: [October 2, 2019, 6:23pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/6 "2019-10-02T18:23:40Z")
sure, i'll use a different hash like SHA256..
---
### Author:  [@Nara725](https://discuss.elastic.co/u/Nara725)
#### Post date: [October 2, 2019, 9:42pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/7 "2019-10-02T21:42:29Z")
I'm using this now,
```
input {
stdin {
}
}
filter {
grok{
match => { "message" => "\[%{TIMESTAMP_ISO8601:time}\] %{DATA:log_message}"}
}
fingerprint {
source => ['log_message']
target => "[@metadata][fingerprint]"
method => "SHA256"
}
}
output {
stdout {
codec => rubydebug {
metadata => true
}
}
}
```
But when i test it i'm not getting expected results. I'm trying to compare the fingerprint value generated for two same messages with different timestamps, but here the debug mode doesnt work
```
[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.00%.
{
"@timestamp" => 2019-10-02T21:45:00.706Z,
"host" => "host",
"@version" => "1",
"time" => "2019-10-02 20:53:17.749",
"message" => "[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.00%."
}
[2019-10-02 20:53:07.749] [logger] [info] harvester: Output queue occupancy: 0.00%.
{
"@timestamp" => 2019-10-02T21:45:16.942Z,
"host" => "host",
"@version" => "1",
"time" => "2019-10-02 20:53:07.749",
"message" => "[2019-10-02 20:53:07.749] [logger] [info] harvester: Output queue occupancy: 0.00%."
}
```
Where as when i set source to `message` instead of `log_message`, the output is in debug mode and i can see the fingerprint hash.
```
[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.00%.
{
"@timestamp" => 2019-10-02T21:40:12.891Z,
"@metadata" => {
"fingerprint" => "7a289d620e1128f8d19e1976744efa090921b1ad7edf42a004e5323ec40c5ce3"
},
"@version" => "1",
"host" => "host",
"time" => "2019-10-02 20:53:17.749",
"message" => "[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.00%."
}
[2019-10-02 20:53:07.749] [logger] [info] harvester: Output queue occupancy: 0.00%.
{
"@timestamp" => 2019-10-02T21:40:30.346Z,
"@metadata" => {
"fingerprint" => "440b70520e778dfda160f3258fd4ba5b6efbcd221478e850ff8dbc70b41aa5c7"
},
"@version" => "1",
"host" => "host",
"time" => "2019-10-02 20:53:07.749",
"message" => "[2019-10-02 20:53:07.749] [logger] [info] harvester: Output queue occupancy: 0.00%."
}
```
Any help regarding this will be helpful, also pls point to any examples that i can refer.
my logs look like the below:
```
[2019-10-02 21:17:00.742] [logger] [info] harvester: Output queue occupancy: 0.00%.
[2019-10-02 21:17:10.742] [logger] [info] harvester: Output queue occupancy: 0.00%.
[2019-10-02 21:17:20.742] [logger] [info] harvester: Output queue occupancy: 0.00%.
[2019-10-02 21:17:30.743] [logger] [info] harvester: Output queue occupancy: 0.00%.
[2019-10-02 21:17:40.743] [logger] [info] harvester: Output queue occupancy: 0.00%.
[2019-10-02 21:17:50.743] [logger] [info] harvester: Output queue occupancy: 0.00%.
[2019-10-02 21:18:00.743] [logger] [info] harvester: Output queue occupancy: 0.00%.
[2019-10-02 21:18:10.743] [logger] [info] harvester: Output queue occupancy: 0.00%.
```
My requirement is to send one log message/minute to Kibana rather than sending all 6 messages...
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 3, 2019, 5:29am UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/8 "2019-10-03T05:29:38Z")
I would expect to see a `log_message` field in the output, which seems to be missing. Is this the full config? Can you change the `DATA` to a `GREEDYDATA`?
---
### Author:  [@Nara725](https://discuss.elastic.co/u/Nara725)
#### Post date: [October 3, 2019, 2:57pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/9 "2019-10-03T14:57:53Z")
Yes thats the full config. Its working when i tried with GREEDYDATA
```
[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.00%.
{
"@timestamp" => 2019-10-03T14:53:15.782Z,
"@metadata" => {
"fingerprint" => "6ba223861a09243d169c74c046ce239b9455687468a1dd8bb6ae76f2b4bcbaf6"
},
"@version" => "1",
"host" => "host",
"log_message" => "[logger] [info] harvester: Output queue occupancy: 0.00%.",
"time" => "2019-10-02 20:53:17.749",
"message" => "[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.00%."
}
[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.01%.
{
"@timestamp" => 2019-10-03T14:53:29.444Z,
"@metadata" => {
"fingerprint" => "c551cb6c4d5871768fc0ecb6d4e2e8461c5b9f9d6fc0de5a0c397c0f24454123"
},
"@version" => "1",
"host" => "host",
"log_message" => "[logger] [info] harvester: Output queue occupancy: 0.01%.",
"time" => "2019-10-02 20:53:17.749",
"message" => "[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.01%."
}
[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.00%.
{
"@timestamp" => 2019-10-03T14:53:41.426Z,
"@metadata" => {
"fingerprint" => "6ba223861a09243d169c74c046ce239b9455687468a1dd8bb6ae76f2b4bcbaf6"
},
"@version" => "1",
"host" => "host",
"log_message" => "[logger] [info] harvester: Output queue occupancy: 0.00%.",
"time" => "2019-10-02 20:53:17.749",
"message" => "[2019-10-02 20:53:17.749] [logger] [info] harvester: Output queue occupancy: 0.00%."
}
```
Now the pending part is time constraint, i need to generate only one message for a give timestamp.. can i provide an timestamp as source for fingerprint: where i give the date with hours and minutes?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 3, 2019, 4:58pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/10 "2019-10-03T16:58:57Z")
You could extract the relevant time stamp component and add these to the fields used by the fingerprint plugin.
---
### Author:  [@Nara725](https://discuss.elastic.co/u/Nara725)
#### Post date: [October 3, 2019, 9:50pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/11 "2019-10-03T21:50:30Z")
I'm able to get what i want using the below config:
```
filter {
dissect {
mapping => { "message" => "[%{ts}-%{+ts}-%{+ts} %{+ts}:%{+ts}:%{seconds}] %{remaining}" }
}
fingerprint {
source => ['remaining','ts']
target => "[@metadata][fingerprint]"
method => "SHA256"
}
}
output {
kafka {
bootstrap_servers => "kafka broker host"
topic_id => "kafka topic"
id => "%{[@metadata][fingerprint]}"
codec => plain {
format => "%{message}"
}
}
}
```
but unfortunately the team who supports Elastic Search blocked users in setting up the `_id` , hence i cannot use the replace functionality. Is their any other way where i can do this?
if not ... then the issue i had is resolved, thanks for the help Christian !!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 31, 2019, 9:50pm UTC](https://discuss.elastic.co/t/need-help-in-using-logstash-with-grok-and-fingerprint/201109/12 "2019-10-31T21:50:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.