# Logstash adding duplicate rows in every run
**URL:** https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453
**Category:** Logstash
**Created:** [September 14, 2017, 5:26am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453 "2017-09-14T05:26:30Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@saafianihan](https://discuss.elastic.co/u/saafianihan)
#### Post date: [September 14, 2017, 5:26am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/1 "2017-09-14T05:26:30Z")
i am using Logstash to pull the date from MySQL database. I am using two different configuration files to create two indexes with different MySQL queries. I run the conf files as
logstash -f path\to\bin\*.conf .
While running the above command duplication problem is solved. But one additional row added to first result from second result and vice versa. Why?
My configuration files is
1. input {
jdbc {
jdbc\_driver\_library =\> "C:/New\_ELK/logstash-5.5.2/mysql-connector-java-5.1.42/mysql-connector-java-5.1.42-bin.jar"
jdbc\_driver\_class =\> "com.mysql.jdbc.Driver"
jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/emob\_report"
jdbc\_user =\> "root"
jdbc\_password =\> ""
schedule =\>"\* \* \* \* \*"
statement =\> "select attendance\_fact.id as attendance\_id,date(attendance\_fact.punchin\_time) as clocked\_in\_date,org\_dimension.name as organization,org\_dimension.id as org\_id,
concat(user\_dimenson.first\_name,' ',user\_dimenson.last\_name) as user
from attendance\_fact
inner join user\_dimenson on attendance\_fact.user\_fk=user\_dimenson.ID
inner join org\_dimension on user\_dimenson.org\_fk=org\_dimension.ID
where date(attendance\_fact.punchin\_time)=curdate()"
}
}
output {
elasticsearch {
index =\> "users\_clocked\_in"
hosts =\> ["localhost:9200"]
document\_id =\> "%{[attendance\_id]}"
}
stdout { codec =\> json\_lines }
}
1. input {
jdbc {
jdbc\_driver\_library =\> "C:/New\_ELK/logstash-5.5.2/mysql-connector-java-5.1.42/mysql-connector-java-5.1.42-bin.jar"
jdbc\_driver\_class =\> "com.mysql.jdbc.Driver"
jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/emob\_report"
jdbc\_user =\> "root"
jdbc\_password =\> ""
schedule =\>"\* \* \* \* \*"
statement =\> "select complaint\_fact.id as complaint\_id,workflow\_dimension.name as workflow,work\_location\_dimension.name as worklocation,date(complaint\_fact.complaint\_create\_time) as jobsadded\_createddate,
org\_dimension.name as organization,org\_dimension.id as org\_id
from complaint\_fact
left join workflow\_dimension on complaint\_fact.workflow\_fk=workflow\_dimension.id
left join work\_location\_dimension on complaint\_fact.worklocation\_fk=work\_location\_dimension.id
inner join org\_dimension on complaint\_fact.org\_fk = org\_dimension.id
where date(complaint\_fact.complaint\_create\_time)=curdate()"
}
}
output {
elasticsearch {
index =\> "new\_jobs\_count"
hosts =\> ["localhost:9200"]
document\_id =\> "%{[complaint\_id]}"
}
stdout { codec =\> json\_lines }
}
---
### Author:  [@saafianihan](https://discuss.elastic.co/u/saafianihan)
#### Post date: [September 14, 2017, 6:15am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/2 "2017-09-14T06:15:23Z")
No hints from anyone. I am still looking for answer. Please help me!!!
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 17, 2017, 5:05pm UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/3 "2017-09-17T17:05:50Z")
Logstash has a single event pipeline and doesn't care if you have multiple configuration files. All events from all inputs will be sent through all filters and then to all outputs. If you don't want that you need to wrap the filters and outputs in conditionals to route the events correctly.
This question pops up here at least once a week.
---
### Author:  [@saafianihan](https://discuss.elastic.co/u/saafianihan)
#### Post date: [September 18, 2017, 4:25am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/4 "2017-09-18T04:25:45Z")
I am new to ELK. Could you please give me an example of wrap the filters and outputs in conditionals!!!
---
### Author:  [@saafianihan](https://discuss.elastic.co/u/saafianihan)
#### Post date: [September 18, 2017, 4:26am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/5 "2017-09-18T04:26:33Z")
While executing the configuration file, logstash adding one additional row with the Query result. The additional row contains the "document\_id" details. I am using "document\_id" to prevent duplication of data for every run. The duplication issue solved, but one additional row is added to the result with document\_id details. Why?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 18, 2017, 5:27am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/6 "2017-09-18T05:27:14Z")
> I am new to ELK. Could you please give me an example of wrap the filters and outputs in conditionals!!!
See [Accessing event data and fields | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html) for examples.
> While executing the configuration file, logstash adding one additional row with the Query result. The additional row contains the "document\_id" details.
It's not clear what you mean by this. Please give an example. The output from your stdout output would be useful to see.
---
### Author:  [@saafianihan](https://discuss.elastic.co/u/saafianihan)
#### Post date: [September 18, 2017, 5:36am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/7 "2017-09-18T05:36:54Z")
My configuration file is
input {
jdbc {
jdbc\_driver\_library =\> "C:/New\_ELK/logstash-5.5.2/mysql-connector-java-5.1.42/mysql-connector-java-5.1.42-bin.jar"
jdbc\_driver\_class =\> "com.mysql.jdbc.Driver"
jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/emob\_report"
jdbc\_user =\> "root"
jdbc\_password =\> ""
schedule =\>"\* \* \* \* \*"
statement =\> "select attendance\_fact.id as attendance\_id,date(attendance\_fact.punchin\_time) as clocked\_in\_date,org\_dimension.name as organization,org\_dimension.id as org\_id,
concat(user\_dimenson.first\_name,' ',user\_dimenson.last\_name) as user
from attendance\_fact
inner join user\_dimenson on attendance\_fact.user\_fk=user\_dimenson.ID
inner join org\_dimension on user\_dimenson.org\_fk=org\_dimension.ID
where date(attendance\_fact.punchin\_time)=curdate()"
}
}
output {
elasticsearch {
index =\> "users\_clocked\_in"
hosts =\> ["localhost:9200"]
document\_id =\> "%{[attendance\_id]}"
}
stdout { codec =\> json\_lines }
}
my stdout output is
just see the output, the second row from the last is an additional row which appended to my result with "document\_id" details. That row is not needed in my result. Why it is appended?
---
### Author:  [@saafianihan](https://discuss.elastic.co/u/saafianihan)
#### Post date: [September 18, 2017, 7:34am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/8 "2017-09-18T07:34:33Z")
Could you please help me to avoid that unnecessary row from showing/fetching?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 18, 2017, 10:08am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/9 "2017-09-18T10:08:16Z")
That event was probably produced by one of your other jdbc inputs. The way the configuration above looks it couldn't possibly have produced the event you've circled in red.
---
### Author:  [@saafianihan](https://discuss.elastic.co/u/saafianihan)
#### Post date: [September 18, 2017, 10:25am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/10 "2017-09-18T10:25:57Z")
When i execute above logstash configuration file i did not get the additional row. But when i execute the two more configuration files i am getting the result as shown.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 18, 2017, 11:31am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/11 "2017-09-18T11:31:37Z")
Right, and that's because you're not wrapping your outputs in conditionals so that events from the input in configuration file A is routed only to file A's output.
---
### Author:  [@saafianihan](https://discuss.elastic.co/u/saafianihan)
#### Post date: [September 18, 2017, 11:34am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/12 "2017-09-18T11:34:56Z")
How can i do that? Could you please give me an example?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 18, 2017, 11:36am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/13 "2017-09-18T11:36:34Z")
Did you look at the documentation link I sent? You can set a field or a tag in the jdbc input and look at that field or tag in the conditional block in the output.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 16, 2017, 11:36am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453/14 "2017-10-16T11:36:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.