# Unknown error occurred sending a bulk request to Elasticsearch

**URL:** https://discuss.elastic.co/t/unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/353920
**Category:** Logstash
**Tags:** docker
**Created:** [February 22, 2024, 8:19pm UTC](https://discuss.elastic.co/t/unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/353920 "2024-02-22T20:19:08Z")
**Posts on this page:** 1
**Showing post:** 2

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 22, 2024, 10:41pm UTC](https://discuss.elastic.co/t/unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/353920/2 "2024-02-22T22:41:19Z")

</div>

> [@JRicha](#):
>
> {:message=\>""\xD1" from ASCII-8BIT to UTF-8", :exception=\>LogStash::Json::GeneratorError, :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:43:in `jruby_dump`

That's blowing up [here](https://github.com/elastic/logstash/blob/e9c9865f4066b54048f8d708612a72d25e2fe5d9/logstash-core/lib/logstash/json.rb#L43). Per [this](https://github.com/elastic/logstash/issues/1582#issuecomment-50792120) comment on the same error in a different context it means you have non-UTF-8 characters in one of your fields.

OK, if in a ruby filter you do

```
            text = [0xD1].pack("C*")
            text.to_json

```

you will get

> "\xD1" from ASCII-8BIT to UTF-8 {:class=\>"Encoding::UndefinedConversionError"

You have a couple of options. If you know the encoding of the string you have you  
could try this ruby code

```
text = event.get("problemField") 
text = text.force_encoding("iso-8859-1").encode("utf-8")
event.set("problemField", text)

```

which will get you

```
"problemField" => "Ñ",

```

I am not telling you that you have iso-8859-1 encoded text. It's plausible but you need to determine if this is true.

If the encoding is unknown or varies then a more aggressive approach would be

```
event.set("problemField", text.encode("UTF-8", "binary", :invalid => :replace, :undef => :replace)

```

which will get you

```
"problemField" => "�",

```

I recognize that losing data is bad, but your fields have to be valid UTF-8 to be sent to elasticsearch. It is not optional.

I assume you know which fields are likely to have non-UTF-8 data in them. If you do not know then you will have to iterate over the fields of the event. [This](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/8) code should give you some ideas.

I hope this helps you to understand the issue.

Another thing you could do in ruby is

```
            initialEvent = event.to_json # String.to_json crashes, Hash.to_json does not
            fixedEvent = initialEvent.encode("UTF-8", "binary", "replace" => "x", :invalid => :replace, :undef => :replace)
            if fixedEvent != initialEvent
                event.tag("encodingProblem")
            end

```

then route to a different output based on `"encodingProblem" in [tags]`, if you use a rubydebug output then start looking for \x, as in

```
"problemField" => "\xD1",
        "tags" => [
    [0] "encodingProblem"
],

```

---

_[View the full topic](https://discuss.elastic.co/t/unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/353920)._