# Logstash Failing after setting a custom document ID **URL:** \ **Category:** Logstash\ **Created:** [September 11, 2018, 7:08am UTC](https://discuss.elastic.co/t/logstash-failing-after-setting-a-custom-document-id/148068 "2018-09-11T07:08:48Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ganessen23](https://avatars.discourse-cdn.com/v4/letter/g/b5a626/32.png) [@ganessen23](https://discuss.elastic.co/u/ganessen23)\ **Post date:** [September 11, 2018, 7:08am UTC](https://discuss.elastic.co/t/logstash-failing-after-setting-a-custom-document-id/148068/1 "2018-09-11T07:08:48Z")
Hello, I am using ELK version 6.4.0 on a Linux machine. My logstash is failing when I'm trying to set a custom document id, but I am able to import data from the database to Elasticsearch if I remove the document\_id parameter in my output conf file. The reason I am using a custom document id is to prevent any duplicates of data being imported to Elasticsearch. Error found in logstash logs: > [2018-09-11T06:36:16,953][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=\>400, :url=\>"[http://localhost:9200/\_bulk](http://localhost:9200/_bulk)"} This is my configuration file: ``` input { jdbc{ jdbc_driver_library => "/data/sqljdbc_6.0/enu/jre8/sqljdbc42.jar" jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc_connection_string => "jdbc:sqlserver://xx.xx.xx.xx:1433;instanceName=xxx;databasename=xxx" jdbc_user => "Elastic" jdbc_password => "xxxx" schedule => "* * * * *" statement => "SELECT TOP 1000 * FROM [xxxx].[xx].[table_x]" } } output { elasticsearch { hosts => ["localhost:9200"] user => "elastic" password => "password" index => "rejection" document_type => "rejection" document_id => "%{[date_run]}_%{[nat_key]}" } stdout {} } ``` I found a post on that same matter in this forum, which tells us this is a permission problem: (see [link](https://discuss.elastic.co/t/encountered-a-retryable-error-will-retry-with-exponential-backoff/109883)) ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/f/4f64006a9d8f7c46fefda47ff1ab32f2230e0243.png) I am infact using the **elastic** user, user which normally has the high level of permissions, but logstash still is failing. I have even tried the following users logstash\_internal, logstash\_admin which i created using this [link](https://www.elastic.co/guide/en/logstash/6.x/ls-security.html) but to no avail. So, my questions would be: Is there any way to use the document id without having logstash to fail? Or is there any user that I have to create that would have the necessary permissions to overwrite the document\_id? Is there any way to ingest unique documents without the need of the parameter document\_id? Thanks in advance. Ganessen. ---
**Author:** ![ganessen23](https://avatars.discourse-cdn.com/v4/letter/g/b5a626/32.png) [@ganessen23](https://discuss.elastic.co/u/ganessen23)\ **Post date:** [September 12, 2018, 5:24am UTC](https://discuss.elastic.co/t/logstash-failing-after-setting-a-custom-document-id/148068/2 "2018-09-12T05:24:43Z")
Hello, Can someone please help.. Thanks, Ganessen ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 12, 2018, 5:31am UTC](https://discuss.elastic.co/t/logstash-failing-after-setting-a-custom-document-id/148068/3 "2018-09-12T05:31:09Z")
What is the format of the fields you are using to create the document I’d? Can any of them contain inappropriate characters? Are these fields set for all documents? ---
**Author:** ![ganessen23](https://avatars.discourse-cdn.com/v4/letter/g/b5a626/32.png) [@ganessen23](https://discuss.elastic.co/u/ganessen23)\ **Post date:** [September 12, 2018, 6:26am UTC](https://discuss.elastic.co/t/logstash-failing-after-setting-a-custom-document-id/148068/4 "2018-09-12T06:26:15Z")
Hello Christian, I'm using the following 2 fields to construct the document. Format in SQL server: date\_run ==\> datatype datetime nat\_key ==\> datatype nchar Once imported to Elasticsearch, it turns out that the **nat\_key** contains some inapropriate characters and some extra unnecessary space. ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/d/8d314686bf4309753a5b2627d4944d1fd049fae6.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9c4b65b75549a3bc1fd95c2bc858b4da40b78d8e.png) Could it be the reason for this failure? Thanks, Ganessen ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 12, 2018, 6:35am UTC](https://discuss.elastic.co/t/logstash-failing-after-setting-a-custom-document-id/148068/5 "2018-09-12T06:35:59Z")
Yes, if the field contains inappropriate characters that is likely to be the reason. One way to get around this could perhaps be to hash the value into a SHA1 hash using the fingerprint module and use this as the document ID instead. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 10, 2018, 6:36am UTC](https://discuss.elastic.co/t/logstash-failing-after-setting-a-custom-document-id/148068/6 "2018-10-10T06:36:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.