# Occasional duplicate entries going in to different indices

**URL:** <https://discuss.elastic.co/t/occasional-duplicate-entries-going-in-to-different-indices/107040>\
**Category:** Logstash\
**Created:** [November 9, 2017, 12:12pm UTC](https://discuss.elastic.co/t/occasional-duplicate-entries-going-in-to-different-indices/107040 "2017-11-09T12:12:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![paulglavin](https://avatars.discourse-cdn.com/v4/letter/p/b38774/32.png) [@paulglavin](https://discuss.elastic.co/u/paulglavin)\
**Post date:** [November 9, 2017, 12:12pm UTC](https://discuss.elastic.co/t/occasional-duplicate-entries-going-in-to-different-indices/107040/1 "2017-11-09T12:12:39Z")

</div>

I have a syslog node that is outputting to logstash on TCP.  
I seem to occasionally get duplicate log entries, one going to where I expect, the other goes to a new index.  
My logstash syslog config is this:

input {  
tcp {  
port =\> 5140  
}  
udp {  
port =\> 5140  
}  
}  
filter {  
if [type] == "syslog" {  
if "reverseproxy" in [message]{  
grok {  
break\_on\_match =\> false  
match =\> [  
"message", "\<%{INT:priority}\>(?%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:interface}: id="%{INT:id}" srcip="%{IP:source\_ip}" localip="%{IP:local\_ip}" size="%{INT:size}" user="%{USER:user}" host="%{HOSTNAME:host}" method="%{WORD:method}" statuscode="%{INT:statuscode}" reason="%{DATA:reason}" extra="%{DATA:extra}" exceptions="%{DATA:exceptions}" time="%{INT:time}" url="%{DATA:url}" server="%{DATA:server}" referer="%{DATA:referer}" cookie="%{GREEDYDATA:cookie}" set-cookie="%{DATA:set\_cookie}" ",  
"message", "\<%{INT:priority}\>(?%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:interface}: id="%{INT:id}" srcip="%{IP:source\_ip}" localip="%{IP:local\_ip}" size="%{INT:size}" user="%{USER:user}" host="%{HOSTNAME:host}" method="%{WORD:method}" statuscode="%{INT:statuscode}" reason="%{DATA:reason}" extra="%{DATA:extra}" exceptions="%{DATA:exceptions}" time="%{INT:time}" url="%{DATA:url}" server="%{DATA:server}" referer="%{DATA:referer}"; cookie="%{GREEDYDATA:cookie}" set-cookie="%{DATA:set\_cookie}" "]  
}  
mutate {  
add\_tag =\> "reverseproxy"  
remove\_tag =\> "\_grokparsefailure"  
}  
geoip {  
source =\> "source\_ip"  
}  
}  
if "ulogd" in [message]{  
grok {  
break\_on\_match =\> false  
match =\> [  
"message", "\<%{INT:fw\_id}\>(?%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}[%{INT:pid}]: id="%{INT:id}" severity="%{WORD:severity}" sys="%{WORD:sys}" sub="%{WORD:sub}" name="%{DATA:name}" action="%{WORD:action}" fwrule="%{INT:fwrule}" outitf="%{NOTSPACE:outitf}" mark="%{DATA:mark}" app="%{DATA:app}" srcmac="%{MAC:srcmac}" srcip="%{IPV4:source\_ip}" dstip="%{IPV4:destination\_ip}" proto="%{WORD:protocol}" length="%{INT:length}" tos="%{DATA:tos}" prec="%{DATA:prec}" ttl="%{INT:ttl}" srcport="%{INT:srcport}" dstport="%{INT:dstport}" "]  
}  
mutate {  
remove\_tag =\> "\_grokparsefailure"  
add\_tag =\> "ulogd" }  
geoip {  
source =\> "source\_ip"  
}  
}  
if "aua" in [message]{  
grok {  
break\_on\_match =\> false  
match =\> [  
"message", "\<%{INT:fw\_id}\>(?%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}[%{INT:pid}]: id="%{INT:id}" severity="%{WORD:severity}" sys="%{WORD:sys}" sub="%{WORD:sub}" name="%{DATA:name}" srcip="%{IP:source\_ip}" host="%{DATA:external\_host}" user="%{DATA:user}" caller="%{WORD:caller}" reason="%{WORD:reason}" ",  
"message", "\<%{INT:fw\_id}\>(?%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:program}[%{INT:pid}]: id="%{INT:id}" severity="%{WORD:severity}" sys="%{WORD:sys}" sub="%{WORD:sub}" name="Trying %{IP:ip\_intern} %{DATA}" "]  
}  
mutate {  
remove\_tag =\> "\_grokparsefailure"  
add\_tag =\> "aua" }  
geoip {  
source =\> "source\_ip"  
}  
}  
if "confd" in [message]{  
grok {  
break\_on\_match =\> false  
match =\> [  
"message", "(?%{YEAR}:%{MONTHNUM}:%{MONTHDAY}-%{HOUR}:%{MINUTE}:%{SECOND}) %{HOSTNAME:logsource} %{WORD:confd}[%{INT:pid}]: id="%{INT:id}" severity="%{WORD:severity}" sys="%{WORD:sys}" sub="%{WORD:sub}" name="%{DATA:name}" user="%{WORD:user}" host="%{DATA:external\_host}" user="%{DATA:user}" caller="%{WORD:caller}" reason="%{WORD:reason}" "]  
}  
mutate {  
remove\_tag =\> "\_grokparsefailure"  
add\_tag =\> "confd" }  
geoip {  
source =\> "source\_ip"  
}  
}

}  
}

output {  
if "heimdallr" in [message] {  
elasticsearch {  
hosts =\> ["172.x.x.x:9200"]  
index =\> "utm-%{+YYYY.MM.dd}"  
sniffing =\> false  
}  
}

# stdout { codec =\> rubydebug }

}

From that output the majority of my logs go into a utm date index and look like this:

{  
"\_index": "utm-2017.11.09",  
"\_type": "logs",  
"\_id": "AV-eFAFmFdBBEmEyn9Ro",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"@version": "1",  
"host": "172.16.x.x",  
"@timestamp": "2017-11-09T00:00:32.092Z",  
"message": "\<30\>2017:11:08-23:58:32 heimdallr httpproxy[4344]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="172.16.x.x" dstip="x.x.x.x" user="" group="" ad\_domain="" statuscode="200" cached="0" profile="REF\_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF\_DefaultHTTPCFFAction (Default content filter action)" size="21127" request="0xe23df000" url="[https://ec2messages.eu-west-1.amazonaws.com/](https://ec2messages.eu-west-1.amazonaws.com/)" referer="" error="" authtime="0" dnstime="2" cattime="107" avscantime="0" fullreqtime="167138144" device="0" auth="0" ua="" exceptions="" category="178" reputation="trusted" categoryname="Internet Services" country="Ireland"",  
"port": 35837  
},  
"fields": {  
"@timestamp": [  
1510185632092  
]  
},  
"sort": [  
1510185632092  
]  
}

But I'm finding that there are some events that are duplicated and I can't seem to see why. The event above has a duplicate that was sent to the "%{[@metadata][beat]}-2017.11.09" index. The message field is identical, the only difference is the index.

```
  {

```

"\_index": "%{[@metadata][beat]}-2017.11.09",  
"\_type": "%{[@metadata][type]}",  
"\_id": "AV-eFAFiFdBBEmEyn9Rn",  
"\_score": 1,  
"\_source": {  
"@version": "1",  
"host": "172.16.x.x",  
"@timestamp": "2017-11-09T00:00:32.092Z",  
"message": """\<30\>2017:11:08-23:58:32 heimdallr httpproxy[4344]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="172.16.x.x" dstip="x.x.x.x" user="" group="" ad\_domain="" statuscode="200" cached="0" profile="REF\_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF\_DefaultHTTPCFFAction (Default content filter action)" size="21127" request="0xe23df000" url="[https://ec2messages.eu-west-1.amazonaws.com/](https://ec2messages.eu-west-1.amazonaws.com/)" referer="" error="" authtime="0" dnstime="2" cattime="107" avscantime="0" fullreqtime="167138144" device="0" auth="0" ua="" exceptions="" category="178" reputation="trusted" categoryname="Internet Services" country="Ireland"""",  
"port": 35837  
}  
}

The index name that it is being output to is defined in a different output that this host can't talk to.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 7, 2017, 12:12pm UTC](https://discuss.elastic.co/t/occasional-duplicate-entries-going-in-to-different-indices/107040/2 "2017-12-07T12:12:54Z")

</div>

Closed on December 7, 2017, 12:12pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.