# Logstash to elasticsearch

**URL:** <https://discuss.elastic.co/t/logstash-to-elasticsearch/128021>\
**Category:** Logstash\
**Created:** [April 13, 2018, 8:32pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch/128021 "2018-04-13T20:32:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Senthiles](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@Senthiles](https://discuss.elastic.co/u/Senthiles)\
**Post date:** [April 13, 2018, 8:32pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch/128021/1 "2018-04-13T20:32:17Z")

</div>

Hi,  
while inserting data from database to elasticsearch index. the query statement run to fetch the no of count.  
Can we alter the query which runs to get the count.

---

<div class="post-metadata">

**Author:** ![Mike.Barretta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike.barretta/32/16688_2.png) [@Mike.Barretta](https://discuss.elastic.co/u/Mike.Barretta)\
**Post date:** [April 14, 2018, 7:07pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch/128021/2 "2018-04-14T19:07:14Z")

</div>

@Senthiles You can specify the query that is used by the Logstash JDBC input plugin. See: [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#\_configuring\_sql\_statement](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#_configuring_sql_statement)

The query is actually for all the data you want to fetch, not just a count. But, if that query doesn't return any data, then none would be available for inserting into ES.

That said, the JDBC input keeps track of which data it has already seen, stored on disk at the location configured in [last\_run\_metadata\_path](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#plugins-inputs-jdbc-last_run_metadata_path). You can configure how to determines what it thinks it's "seen" before by configuring the [tracking fields](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#plugins-inputs-jdbc-tracking_column)

---

<div class="post-metadata">

**Author:** ![Senthiles](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@Senthiles](https://discuss.elastic.co/u/Senthiles)\
**Post date:** [April 17, 2018, 7:19am UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch/128021/3 "2018-04-17T07:19:45Z")

</div>

hi,  
the below is the id value i am getting while running the logstash.  
"\_index": "general-report-hub1",  
"\_type": "doc",  
**"\_id": "%{task\_id}",**  
"\_score": 1.0,

my conf file is

jdbc\_driver\_class =\> "org.postgresql.Driver"  
#scheduler  
schedule =\> "\* \* \* \* \*"  
#pagination  
jdbc\_paging\_enabled =\> true  
#fetch size  
jdbc\_fetch\_size =\> 50  
#page size  
jdbc\_page\_size =\> 100  
#tracking column definition  
tracking\_column\_type =\> "timestamp"  
#tracking column  
tracking\_column =\> "modified\_date"  
statement =\> "SELECT \* from table where modified\_date = :sql\_last\_value

output {  
elasticsearch {  
index =\> "index1"  
#document\_type =\> "task"  
document\_id =\> "%{task\_id}"  
manage\_template =\> false  
hosts =\> "_ **.**.__._\*"  
}  
}

who to remove the document with id value "%{task\_id}" in the index

---

<div class="post-metadata">

**Author:** ![Mike.Barretta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike.barretta/32/16688_2.png) [@Mike.Barretta](https://discuss.elastic.co/u/Mike.Barretta)\
**Post date:** [April 20, 2018, 1:56pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch/128021/4 "2018-04-20T13:56:18Z")

</div>

@Senthiles is `task_id` a field in your incoming documents? In other words, is it already defined someplace?

Also, unless you absolutely want to use that task\_id as the underlying Elasticsearch document id (`_id`), I'd not set `document_id` and just let Elasticsearch generate one of its own. Since documents are assigned to shards based on their "place" in the ID keyspace, manually setting the `_id` field can result in shard imbalance issues. A better way to store an external id would be in a dedicated field that you create, like `original_id` or in this case, `task_id`

That make sense?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 18, 2018, 1:57pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch/128021/5 "2018-05-18T13:57:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.