# Invalid document\_id not handled well with elasticsearch output

**URL:** <https://discuss.elastic.co/t/invalid-document-id-not-handled-well-with-elasticsearch-output/123811>\
**Category:** Logstash\
**Created:** [March 13, 2018, 9:17pm UTC](https://discuss.elastic.co/t/invalid-document-id-not-handled-well-with-elasticsearch-output/123811 "2018-03-13T21:17:02Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 13, 2018, 9:17pm UTC](https://discuss.elastic.co/t/invalid-document-id-not-handled-well-with-elasticsearch-output/123811/1 "2018-03-13T21:17:02Z")

</div>

Due to an issue in my multiline codec, I had an event that did not have the expected format. When I parsed the document\_id out of it, it was part of a JSON object. This results in logstash looping retrying the request and no indication of what the retryable error was in either the logstash or elasticsearch logs, making it really hard to track down.

```

```

Logstash logs these ERRORs

```
[2018-03-13T17:08:47,546][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
[2018-03-13T17:08:49,550][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
[2018-03-13T17:08:53,557][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
[2018-03-13T17:09:01,563][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
[2018-03-13T17:10:42,717][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
[2018-03-13T17:10:44,756][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
[2018-03-13T17:10:48,764][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
[2018-03-13T17:10:56,772][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
[2018-03-13T17:11:12,784][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
[2018-03-13T17:11:44,794][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://localhost:9200/_bulk"}
```

Whilst elasticsearch does not log any errors

```
[2018-03-13T17:08:40,187][INFO][o.e.m.j.JvmGcMonitorService] [sd-0364-9b52] [gc][344193] overhead, spent [327ms] collecting in the last [1s]
[2018-03-13T17:10:45,338][INFO][o.e.m.j.JvmGcMonitorService] [sd-0364-9b52] [gc][344318] overhead, spent [380ms] collecting in the last [1s]
[2018-03-13T17:12:01,411][INFO][o.e.m.j.JvmGcMonitorService] [sd-0364-9b52] [gc][344394] overhead, spent [265ms] collecting in the last [1s]
[2018-03-13T17:12:13,417][INFO][o.e.m.j.JvmGcMonitorService] [sd-0364-9b52] [gc][344406] overhead, spent [317ms] collecting in the last [1s]
```

This can be reproduced using

```
input { generator { message => '2018-02-23T13:20:29.020Z' count => 1 } }
output { stdout { codec => rubydebug } }
output {
  elasticsearch {
    hosts => "localhost"
    index => "logstash-test-%{+YYYY.MM.dd}"
    document_id => ' "XXXXXXXXXXXXXXXXXXXXXX": null,
            "XXXXXXXXXXXXXXXXXXXXXX": null,
            "XXXXXXXXXXXXXXXXXXXXXX": "3.50",
            "XXXXXXXXXXXXXXXXXXXXXX": "02/16/2018",
            "XXXXXXXXXXXXXXXXXXXXXX": 3.50,
            
    "XXXXXXXXXXXXXXXXXXXXXX": {
              "XXXXXXXXXXXXXXXXXXXXXX": null,
              "XXXXXXXXXXXXXXXXXXXXXX": "XXXXXXXXXXXXXXXXXXXXXX",
              "XXXXXXXXXXXXXXXXXXXXXX": null,
              "XXXXXXXXXXXXXXXXXXXXXX": "XXXXXXXXXXXXXXXXXXXXXX"
            },
            "XXXXXXXXXXXXXXXXXXXXXX": "",
            "XXXXXXXXXXXXXXXXXXXXXX": false,
    
    "XXXXXXXXXXXXXXXXXXXXXX": {
              "XXXXXXXXXXXXXXXXXXXXXX": "3.50",
              "XXXXXXXXXXXXXXXXXXXXXX": null,
              "XXXXXXXXXXXXXXXXXXXXXX": 3.50,
              "XXXXXXXXXXXXXXXXXXXXXX": "USD"
            },
            "XXXXXXXXXXXXXXXXXXXXXX": "XXXXXXXXXXXXXXXXXXXXXX",
    
    "XXXXXXXXXXXXXXXXXXXXXX": "02/15/2018",
            "XXXXXXXXXXXXXXXXXXXXXX": "XXXXXXXXXXXXXXXXXXXXXX"
          }
        ],
        "XXXXXXXXXXXXXXXXXXXXXX": 0
      },
      "XXXXXXXXXXXXXXXXXXXXXX": null,
      "XXXXXXXXXXXXXXXXXXXXXX": false,
    
    "XXXXXXXXXXXXXXXXXXXXXX": "USD",
      "XXXXXXXXXXXXXXXXXXXXXX": true,
      "XXXXXXXXXXXXXXXXXXXXXX": false,
      "XXXXXXXXXXXXXXXXXXXXXX": "-3.50",
      "XXXXXXXXXXXXXXXXXXXXXX": "02/22/2018",
      "XXXXXXXXXXXXXXXXXXXXXX": "01/22/2018"
    }
  }'
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 10, 2018, 9:17pm UTC](https://discuss.elastic.co/t/invalid-document-id-not-handled-well-with-elasticsearch-output/123811/2 "2018-04-10T21:17:11Z")

</div>

Closed on April 10, 2018, 9:17pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.