I am sure the responseerror is more verbose than just Exception - can you please share the full error message? You can also increase debugging by enabling stacktraces.
Did you install the field mapper attachment plugin before doing this?
Adding "error_trace: true" to my call, the elasticsearch.log:
[2019-08-05T09:09:40,350][DEBUG][o.e.a.b.TransportBulkAction] [DESKTOP-SBLSQL0] failed to execute pipeline [attachment] for document [descriptions/_doc/101]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:194) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:133) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:100) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:427) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService.access$100(IngestService.java:70) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:355) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
... 11 more
Caused by: java.lang.IllegalArgumentException: Illegal base64 character 3a
at java.util.Base64$Decoder.decode0(Base64.java:743) ~[?:?]
at java.util.Base64$Decoder.decode(Base64.java:535) ~[?:?]
at java.util.Base64$Decoder.decode(Base64.java:558) ~[?:?]
at org.elasticsearch.ingest.IngestDocument.getFieldValueAsBytes(IngestDocument.java:183) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.attachment.AttachmentProcessor.execute(AttachmentProcessor.java:79) ~[?:?]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:123) ~[elasticsearch-7.3.0.jar:7.3.0]
... 9 more
Also, I tried looking for the plugin you suggested by Googling elastic.co, and there is no "field mapper attachment plugin" that I can find, only the following:
Hi Christian. I already do have this plugin installed. I suppose I should have mentioned it, but I thought it would be obvious since I am using a pipeline.
// Kibana
GET _ingest/pipeline
{
"attachment" : {
"description" : "Extract attachment information",
"processors" : [
{
"attachment" : {
"field" : "data"
}
}
]
},
"xpack_monitoring_6" : {
"description" : "This pipeline upgrades documents from the older version of the Monitoring API to the newer version (7) by fixing breaking changes in those older documents before they are indexed from the older version (6).",
"version" : 7000199,
"processors" : [
{
"script" : {
"source" : "ctx._type = null"
}
},
{
"gsub" : {
"field" : "_index",
"pattern" : "(.monitoring-\\w+-)6(-.+)",
"replacement" : "$17$2"
}
}
]
},
"xpack_monitoring_7" : {
"description" : "This is a placeholder pipeline for Monitoring API version 7 so that future versions may fix breaking changes.",
"version" : 7000199,
"processors" : [ ]
}
}
Let me ask you this: is my approach using client.index() from the Elasticsearch Node.js Client correct in your opinion?
[2019-08-06T08:08:58,619][DEBUG][o.e.a.b.TransportBulkAction] [DESKTOP-SBLSQL0] failed to execute pipeline [attachment] for document [descriptions/_doc/101]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:194) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:133) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:100) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:427) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService.access$100(IngestService.java:70) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:355) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
... 11 more
Caused by: java.lang.IllegalArgumentException: Illegal base64 character 3a
at java.util.Base64$Decoder.decode0(Base64.java:743) ~[?:?]
at java.util.Base64$Decoder.decode(Base64.java:535) ~[?:?]
at java.util.Base64$Decoder.decode(Base64.java:558) ~[?:?]
at org.elasticsearch.ingest.IngestDocument.getFieldValueAsBytes(IngestDocument.java:183) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.attachment.AttachmentProcessor.execute(AttachmentProcessor.java:79) ~[?:?]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:123) ~[elasticsearch-7.3.0.jar:7.3.0]
... 10 more
And as you can see, the ingest plugin is installed:
// KIBANA
GET /_cat/plugins?v
// Results in:
name component version
DESKTOP-SBLSQL0 ingest-attachment 7.3.0
THANK YOU!!! Alexander for continuing to help me on this.
I especially like the Elasticsearch Node.js Client, and I want to learn how to ingest base64 encoded files very much indeed.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.