# Unknown error occurred sending a bulk request to Elasticsearch
**URL:** https://discuss.elastic.co/t/unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/353920
**Category:** Logstash
**Tags:** docker
**Created:** [February 22, 2024, 8:19pm UTC](https://discuss.elastic.co/t/unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/353920 "2024-02-22T20:19:08Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@JRicha](https://discuss.elastic.co/u/JRicha)
#### Post date: [February 22, 2024, 8:19pm UTC](https://discuss.elastic.co/t/unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/353920/1 "2024-02-22T20:19:08Z")
Hello community,
I have found other posts here to be helpful in solving my previous issues so I am hoping that someone can help me resolve this issue. I researched my problem in the community pages and have not found any recent solutions to my problem. I am running the ELK stack via 4 Docker containers. My versions are:
- elasticsearch:8.3.2
- kibana:8.3.2
- logstash:8.3.2
- metricbeat:8.3.3
The problem is I receive data from this pipeline, usa03000xxxxx, for a while (hours, days) then suddenly it will stop receiving data in this pipeline. When I look in the logs I see the following error but it does not tell me what data column is causing the problem:
[2024-02-12T17:49:23,088]**[ERROR]**[logstash.outputs.elasticsearch][usa03000xxxxx][usa03000xxxxx\_jdbc\_logstash] An unknown error occurred sending a bulk request to Elasticsearch (will retry indefinitely) {:message=\>""\xD1" from ASCII-8BIT to UTF-8", :exception=\>LogStash::Json::GeneratorError, :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:43:in `jruby_dump'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:127:in `block in bulk'", "org/jruby/RubyArray.java:2589:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:127:in `block in bulk'", "org/jruby/RubyArray.java:1821:in `each'", "org/jruby/RubyEnumerable.java:1258:in `each\_with\_index'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http\_client.rb:125:in `bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:296:in `safe\_bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin\_mixins/elasticsearch/common.rb:228:in `submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:177:in `retrying\_submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:342:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi\_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300:in `block in start\_workers'"]}
**My logstash config file:**
```auto
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mssql-jdbc-10.2.1.jre11.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://usa03000xxxxx.net:45688;databaseName=AuditDbwork;trustServerCertificate=true;encrypt=false"
jdbc_user => "${msuser}"
jdbc_password => "${mspw}"
connection_retry_attempts => 5
schedule => "*/10 * * * *"
statement => "SELECT * FROM radarX_Log WHERE 'starttime' >= '2022-07-30' and 'starttime' > :sql_last_value ORDER BY starttime ASC"
columns_charset => { "transactionid" => "ISO-8859-1" }
columns_charset => { "binarydata" => "ISO-8859-1" }
last_run_metadata_path => "/usr/share/logstash/data/last_run/usa03000xxxxx"
record_last_run => true
tracking_column => "starttime"
tracking_column_type => "timestamp"
use_column_value => true
}
}
filter {
mutate {
add_field => {
"log_source" => "usa03000xxxxx.net"
"DB_Type" => "MSSQL"
"Database_Name" => "AuditDBwork"
}
}
grok {
match => { "starttime" => "(?%{YEAR}[\/\-\s]%{MONTHNUM}[\/\-\s]%{MONTHDAY})" }
}
fingerprint {
source => ["textdata", "dbusername", "eventclass", "ownername", "date", "databasename", "ntusername", "objectname"]
target => "[@metadata][fingerprint]"
method => "SHA1"
key => "7y47ST9qg5F8ezy7"
base64encode => true
concatenate_sources => true
}
}
output {
elasticsearch {
hosts => "https://elk030001.net:9200"
id => "usa03000xxxxx_jdbc_logstash"
data_stream => "true"
data_stream_type => "logs"
data_stream_dataset => "jdbc"
data_stream_namespace => "usa03000xxxxx"
document_id => "%{[@metadata][fingerprint]}"
user => "${logstash_user}"
password => "${logstash_pw}"
ssl => true
cacert => "${cacert}"
}
}
```
I found a few posts that look similar to my problem but they appear to be related to older versions of the ELK stack.
- [My column\_charset is already lowercase](https://discuss.elastic.co/t/unable-to-import-to-es-error-occurred-sending-a-bulk-request-to-elasticsearch/197565/2)
- [I think my logstash-plugin is already up to date](https://discuss.elastic.co/t/an-unknown-error-occurred-sending-a-bulk-request-to-elasticsearch-we-will-retry-indefinitely-error-message-xb0-from-ascii-8bit-to-utf-8/220906)
- [There was already a logstash-plugin fix pushed to the codec](https://discuss.elastic.co/t/jdbc-input-for-oracle-blob-field-error/139861/5)
Is there somewhere in the ELK stack where I can see the data types of the columns selected from this pipeline? So far I have only found the column names and their values but not the data types.
Is this problem usually caused by a date/datetime data type?
Is it one of the columns in my filter.fingerprint.source that is the problem?
Will adding characterEncoding=utf8 to my connection string cover everything? Will that cause problems with other data?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 22, 2024, 10:41pm UTC](https://discuss.elastic.co/t/unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/353920/2 "2024-02-22T22:41:19Z")
> [@JRicha](#):
>
> {:message=\>""\xD1" from ASCII-8BIT to UTF-8", :exception=\>LogStash::Json::GeneratorError, :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:43:in `jruby_dump`
That's blowing up [here](https://github.com/elastic/logstash/blob/e9c9865f4066b54048f8d708612a72d25e2fe5d9/logstash-core/lib/logstash/json.rb#L43). Per [this](https://github.com/elastic/logstash/issues/1582#issuecomment-50792120) comment on the same error in a different context it means you have non-UTF-8 characters in one of your fields.
OK, if in a ruby filter you do
```
text = [0xD1].pack("C*")
text.to_json
```
you will get
> "\xD1" from ASCII-8BIT to UTF-8 {:class=\>"Encoding::UndefinedConversionError"
You have a couple of options. If you know the encoding of the string you have you
could try this ruby code
```
text = event.get("problemField")
text = text.force_encoding("iso-8859-1").encode("utf-8")
event.set("problemField", text)
```
which will get you
```
"problemField" => "Ñ",
```
I am not telling you that you have iso-8859-1 encoded text. It's plausible but you need to determine if this is true.
If the encoding is unknown or varies then a more aggressive approach would be
```
event.set("problemField", text.encode("UTF-8", "binary", :invalid => :replace, :undef => :replace)
```
which will get you
```
"problemField" => "�",
```
I recognize that losing data is bad, but your fields have to be valid UTF-8 to be sent to elasticsearch. It is not optional.
I assume you know which fields are likely to have non-UTF-8 data in them. If you do not know then you will have to iterate over the fields of the event. [This](https://discuss.elastic.co/t/how-to-exclude-xml-json-key-value-if-key-length-is-greater-than-15-char-and-value-length-is-greater-than-100-char/270248/8) code should give you some ideas.
I hope this helps you to understand the issue.
Another thing you could do in ruby is
```
initialEvent = event.to_json # String.to_json crashes, Hash.to_json does not
fixedEvent = initialEvent.encode("UTF-8", "binary", "replace" => "x", :invalid => :replace, :undef => :replace)
if fixedEvent != initialEvent
event.tag("encodingProblem")
end
```
then route to a different output based on `"encodingProblem" in [tags]`, if you use a rubydebug output then start looking for \x, as in
```
"problemField" => "\xD1",
"tags" => [
[0] "encodingProblem"
],
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 21, 2024, 10:42pm UTC](https://discuss.elastic.co/t/unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/353920/3 "2024-03-21T22:42:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.