# Logstash not creating right number of documents when passing folder as an argument
**URL:** https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720
**Category:** Logstash
**Created:** [December 21, 2016, 11:25pm UTC](https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720 "2016-12-21T23:25:34Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@saisn](https://discuss.elastic.co/u/saisn)
#### Post date: [December 21, 2016, 11:25pm UTC](https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720/1 "2016-12-21T23:25:34Z")
Hi,
What I'm doing: Created 4 conf files under a root directory called XYZ. Each of these conf files will import 1000 rows from SQL and tables being imported are unique in all 4 conf files. when ran separately the number of documents created are 1000 in each index but when running conf file with root folder as argument the count is not 1000 in indexes.
I've also observed each index is picking different document. I'm also using templates for each of the index and template name is different all across.
I've given different index names and different document names in all configfiles. But, somehow the number of documents created in each of the indexes is different from expected.
However, when i run configfiles seperately the number of docs created are correct
---
### Author:  [@jsvd](https://discuss.elastic.co/u/jsvd)
#### Post date: [December 22, 2016, 9:19am UTC](https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720/2 "2016-12-22T09:19:54Z")
I'm not sure what is happening without seeing the configuration files, but remember that in logstash, if you run it with multiple configuration files, they will all be concatenated and evaluated as a single one.
So, be sure to confirm if you have unnecessarily duplicate input/filter/output sections.
---
### Author:  [@saisn](https://discuss.elastic.co/u/saisn)
#### Post date: [December 22, 2016, 6:29pm UTC](https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720/3 "2016-12-22T18:29:26Z")
i cant share the whole config file but each conf file is fetching data from single table and writing to a single table. Similar to below i have 4 conf files for each table under a root directory
input {
jdbc {
jdbc\_driver\_library =\> "/h"
jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc\_connection\_string =\> "jdbc:sqlserver://"
jdbc\_user =\> "ReadOnly"
jdbc\_password =\> ""
#lowercase\_column\_names =\> false
#schedule =\> "\*/10 \* \* \* \*"
clean\_run =\> true
use\_column\_value =\> true
tracking\_column =\> \*\*\*
record\_last\_run =\> true
#used for incremental updates of record by having a reference point in ../jinfo
last\_run\_metadata\_path =\> "/etc/logstash/run\_metadata.d/"
statement =\> "SELECT \*
FROM [a] where [a] \> :sql\_last\_value order by [amd] asc"
#jdbc\_paging\_enabled =\> "true"
#jdbc\_page\_size =\> "50000"
#statement\_filepath =\> "query.sql"
}
}
filter {
}
output {
elasticsearch {
user =\> ""
password =\> ""
#ssl =\> true
#ssl\_certificate\_verification =\> true
truststore =\> ""
truststore\_password =\> ""
hosts =\> [""]
index =\> "h1"
document\_type =\> ""
document\_id =\>"%{cd}"
#protocol =\> "http"
}
}
---
### Author:  [@jsvd](https://discuss.elastic.co/u/jsvd)
#### Post date: [December 23, 2016, 10:39am UTC](https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720/4 "2016-12-23T10:39:31Z")
when you say different count than expected, is it more or less?
---
### Author:  [@saisn](https://discuss.elastic.co/u/saisn)
#### Post date: [December 24, 2016, 4:40pm UTC](https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720/5 "2016-12-24T16:40:30Z")
it's more.
---
### Author:  [@jsvd](https://discuss.elastic.co/u/jsvd)
#### Post date: [December 27, 2016, 10:37am UTC](https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720/6 "2016-12-27T10:37:56Z")
does each of your individual configuration files, have the same structure like the one below?
```auto
input {
jdbc {
# ...
}
}
filter {
}
output {
elasticsearch {
# ...
}
}
```
If so, when you execute `bin/logstash -f *.conf`, all your N files will be merged into one, which means that you now have N jdbc blocks, but also N elasticsearch blocks, which means that for each event one of the jdbc blocks produces, you're sending it N times instead of one.
you need 1 file with:
```auto
filter {
}
output {
elasticsearch {
user => ""
password => ""
truststore => ""
truststore_password => ""
hosts => [""]
index => "h1"
document_type => ""
document_id =>"%{cd}"
#protocol => "http"
}
}
```
and then N files, each with just the jdbc:
```auto
input {
jdbc {
}
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 24, 2017, 10:38am UTC](https://discuss.elastic.co/t/logstash-not-creating-right-number-of-documents-when-passing-folder-as-an-argument/69720/7 "2017-01-24T10:38:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.