# Logstash使用fingerprint插件去重复出现问题

**URL:** <https://discuss.elastic.co/t/logstash-fingerprint/42327>\
**Category:** 中文提问与讨论\
**Created:** [February 21, 2016, 10:55am UTC](https://discuss.elastic.co/t/logstash-fingerprint/42327 "2016-02-21T10:55:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jiangzhifang](https://avatars.discourse-cdn.com/v4/letter/j/3d9bf3/32.png) [@jiangzhifang](https://discuss.elastic.co/u/jiangzhifang)\
**Post date:** [February 21, 2016, 10:55am UTC](https://discuss.elastic.co/t/logstash-fingerprint/42327/1 "2016-02-21T10:55:02Z")

</div>

我是新手，可能有些细节描述不清楚，请大家耐心看完。我的测试环境如下：  
1)elasticsearch版本2.2.0，安装目录/opt/elasticsearch  
2)logstash版本2.2.2，安装目录/opt/logstash  
3)测试文件为/tmp/test\_file，内容只有一行  
{"b":{"b1":0.0}}  
4)logstash的配置文件内容

input {  
file {  
path =\> ["/tmp/test\_file"]  
sincedb\_path =\> "/tmp/sincedb"  
codec =\> json  
type =\> "nginx\_visit"  
start\_position =\> "beginning"

}  
}

filter {  
fingerprint {  
method =\> "SHA1"  
key =\> "160129"  
source =\> ["[b]"]  
}  
}

output {  
elasticsearch {  
hosts =\> "192.168.233.103"  
document\_id =\> "%{fingerprint}"  
}  
}

问题描述：  
每次停止logstash之后，删除sincedb，然后再启动logstash，在elasticsearch中会生成一个新的document，但是原来的document仍然保留。因此就产生了重复记录

另外我发现的问题还有：  
1）正如我的配置文件logstash.conf中，如果logstash的input插件是file，file指定的文件test\_file的内容中某一个字段含有子字段，并且子字段的数据类型是浮点数时，每次重新启动logstash，fingerprint就会产生不一样的值。  
2）如果第1）条中的子字段的类型不是浮点数，而是整型或字符串型等，每次重启logstash，fingerprint就能计算出唯一值  
3）我用checksum插件代替fingerprint用来生成documentID，出现的情况和fingerprint的情况相同。

请大家帮我看看我的错误出在什么地方。谢谢

---

<div class="post-metadata">

**Author:** ![medcl.net](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl.net/32/4414_2.png) [@medcl.net](https://discuss.elastic.co/u/medcl.net)\
**Post date:** [March 8, 2016, 4:16am UTC](https://discuss.elastic.co/t/logstash-fingerprint/42327/2 "2016-03-08T04:16:28Z")

</div>

> [@jiangzhifang](#):
>
> input { file { path =\> ["/tmp/test\_file"] sincedb\_path =\> "/tmp/sincedb" codec =\> json type =\> "nginx\_visit" start\_position =\> "beginning"
> 
> }}
> 
> filter { fingerprint { method =\> "SHA1" key =\> "160129" source =\> ["[b]"] }}
> 
> output { elasticsearch { hosts =\> "192.168.233.103" document\_id =\> "%{fingerprint}" }}

@jiangzhifang 感谢反馈，这个是logstash的一个bug，issue跟踪：[The result of Float type is not unique · Issue #12 · logstash-plugins/logstash-filter-fingerprint · GitHub](https://github.com/logstash-plugins/logstash-filter-fingerprint/issues/12)

---

<div class="post-metadata">

**Author:** ![jiangzhifang](https://avatars.discourse-cdn.com/v4/letter/j/3d9bf3/32.png) [@jiangzhifang](https://discuss.elastic.co/u/jiangzhifang)\
**Post date:** [March 8, 2016, 7:44am UTC](https://discuss.elastic.co/t/logstash-fingerprint/42327/3 "2016-03-08T07:44:46Z")

</div>

谢谢您的回复。我发现用checksum插件，也遇到同样问题，logstash配置文件如下：

input {  
file {  
path =\> ["/tmp/test\_file"]  
sincedb\_path =\> "/tmp/sincedb"  
codec =\> json  
type =\> "nginx\_visit"  
start\_position =\> "beginning"  
}  
}

filter {  
checksum {  
keys =\> "[b]"  
}  
}

output {  
elasticsearch {  
hosts =\> "192.168.233.103"  
document\_id =\> "%{[logstash\_checksum]}"  
}  
}

问题出现的条件：

1.测试文件是json格式  
2.某个字段包含子字段，且子字段的值的类型是float

那么，使用fingerprint或者checksum，每次对这个字段计算，都会得出不同的值

---

<div class="post-metadata">

**Author:** ![medcl.net](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl.net/32/4414_2.png) [@medcl.net](https://discuss.elastic.co/u/medcl.net)\
**Post date:** [March 27, 2016, 2:17am UTC](https://discuss.elastic.co/t/logstash-fingerprint/42327/4 "2016-03-27T02:17:06Z")

</div>

看来也是同样的有bug

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:48pm UTC](https://discuss.elastic.co/t/logstash-fingerprint/42327/5 "2017-07-06T13:48:54Z")

</div>