Hi,
I have installed Elasticsearch, Logstash, Kibana, and the SQL JDBC driver.
All of these things run and I have created an index and have used Logstash to index a SQL Server table. I've then used Kibana to query the index. All of this works perfectly for a table that does NOT have a varbinary column.
PROBLEM
If I create a new index and try to populate it from a SQL Server table that has a varbinary column, logstash reports the error below. Note, if I populate my index from all fields in that table EXCEPT the varbinary column, it works as expected.
[2019-07-09T07:55:01,869][ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely
{:error_message=>"\"\\xE4\" from ASCII-8BIT to UTF-8",
:error_class=>"LogStash::Json::GeneratorError",
:backtrace=>["C:/Cabinet/Downloads/ElasticSearch/logstash-7.2.0/logstash-core/lib/logstash/json.rb:27:in `jruby_dump'",
"C:/Cabinet/Downloads/ElasticSearch/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:119:in `block in bulk'",
"org/jruby/RubyArray.java:2577:in `map'", "C:/Cabinet/Downloads/ElasticSearch/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:119:in `block in bulk'", "org/jruby/RubyArray.java:1792:in `each'",
"C:/Cabinet/Downloads/ElasticSearch/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:117:in `bulk'",
"C:/Cabinet/Downloads/ElasticSearch/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:296:in `safe_bulk'",
"C:/Cabinet/Downloads/ElasticSearch/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:201:in `submit'",
"C:/Cabinet/Downloads/ElasticSearch/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:169:in `retrying_submit'",
"C:/Cabinet/Downloads/ElasticSearch/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:38:in `multi_receive'",
"org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in `multi_receive'",
"C:/Cabinet/Downloads/ElasticSearch/logstash-7.2.0/logstash-core/lib/logstash/java_pipeline.rb:239:in `block in start_workers'"]}
QUESTION
Is it possible to populate an Elasticsearch index with varbinary data in a SQL Server table? The binary data is from MS Office documents, PDF's, etc.
Or, do those files need to reside on the file system (as actual .docx, xlsx, pdf) files?
Miscellaneous Info
Thinking it would somehow be useful, I've created an attachment processor pipeline, following these instructions:
Attachment Processing Pipeline
But I guess I don't know how to use that pipeline. So if using that pipeline is the key to solving my problem, I could use instructions on how to use that pipeline.
I'm a total novice at this, so take that into consideration when replying (in other words, the more info you give, the better).
Thanks, Brent