# Query that finds duplicate log lines and their respective count
**URL:** https://discuss.elastic.co/t/query-that-finds-duplicate-log-lines-and-their-respective-count/199937
**Category:** Elasticsearch
**Created:** [September 18, 2019, 7:35am UTC](https://discuss.elastic.co/t/query-that-finds-duplicate-log-lines-and-their-respective-count/199937 "2019-09-18T07:35:42Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@AndriesN](https://discuss.elastic.co/u/AndriesN)
#### Post date: [September 18, 2019, 7:35am UTC](https://discuss.elastic.co/t/query-that-finds-duplicate-log-lines-and-their-respective-count/199937/1 "2019-09-18T07:35:42Z")
Hi all,
I am doing a school project where I am using your product for a log management system.
I have lots of data and I want to know which log lines are duplicate and how many duplicates there are for that particular log line.
I tried this query in which I succesfully extracted the duplicate numbers.
```
GET /_all/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"beat.hostname": "server-x"
}
},
{
"match": {
"log_level": "WARNING"
}
},{
"range": {
"@timestamp" : {
"gte" : "now-48h",
"lte" : "now"
}
}
}
]
}
},
"aggs": {
"duplicateNames": {
"terms": {
"field": "message_description.keyword",
"min_doc_count": 2,
"size": 10000
}
}
}
}
```
This works for a log line that contains: 'AuthToken not Found' as you can see here:
```
"aggregations" : {
"duplicateNames" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "AuthToken not found []",
"doc_count" : 657
}
]
}
}
```
But it doesn't work for a log line that contains more characters for some weird reason. I tried the very same query only with log\_level : "CRITICAL". In that way I'll get other log lines from the CRITICAL level but somehow the bucket is empty.
I hope someone can help me with this weird problem.
Thanks,
Andries
---
### Author:  [@abdon](https://discuss.elastic.co/u/abdon)
#### Post date: [September 19, 2019, 1:29pm UTC](https://discuss.elastic.co/t/query-that-finds-duplicate-log-lines-and-their-respective-count/199937/2 "2019-09-19T13:29:02Z")
What do you mean by "a log line that contains more characters" exactly? Can you give an example of those log lines?
---
### Author:  [@AndriesN](https://discuss.elastic.co/u/AndriesN)
#### Post date: [September 19, 2019, 2:07pm UTC](https://discuss.elastic.co/t/query-that-finds-duplicate-log-lines-and-their-respective-count/199937/3 "2019-09-19T14:07:38Z")
Well I have a feeling it doesn't work when the log line has more than x characters for example this log line: `"""Uncaught PHP Exception ErrorException: "Warning: include(/data/httpd/api/xxx/var/cache/dev/overblog/graphql-bundle/ __definitions__ /QueryType.php): failed to open stream: No such file or directory" at /data/httpd/api/xxx/vendor/composer/ClassLoader.php line 444"""`
I have multiple log lines who are exactly like this but for some reason the query mentioned above doesn't give me a bucket
Can it be that the .keyword messes it up? Or is my query incorrect?
---
### Author:  [@abdon](https://discuss.elastic.co/u/abdon)
#### Post date: [September 19, 2019, 3:21pm UTC](https://discuss.elastic.co/t/query-that-finds-duplicate-log-lines-and-their-respective-count/199937/4 "2019-09-19T15:21:25Z")
You are absolutely right - by default a `.keyword` field will only contain values up to 256 characters. You can see that by looking at your index' mappings:
```auto
GET my_index/_mapping
```
You will see that the `.keyword` fields in the mapping have an [`ignore_above` parameter](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html) with a value of 256.
You can change the value of `ignore_above`. You would typically do that when creating the index, by providing [an explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#create-mapping). You can also [change it dynamically on existing indexes](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html) but be aware that this is quite an expensive operation as it requires Elasticsearch to rewrite all the data.
To update existing indexes, first update the mapping:
```auto
PUT my_index/_mapping
{
"properties": {
"message_description": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
```
Next you can then reindex the data by executing an [`_update_by_query`](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html) request:
```auto
POST my_index/_update_by_query?wait_for_completion=false
```
The last operation will run in the background and will take some time to complete, depending on how much data you have.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 17, 2019, 3:21pm UTC](https://discuss.elastic.co/t/query-that-finds-duplicate-log-lines-and-their-respective-count/199937/5 "2019-10-17T15:21:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.