Exception on ingest an attachment using client.index()

n2nelastic · August 3, 2019, 4:01pm

I have properly set up an index and pipeline to ingest base64 encoded documents.

Using the elasticsearch client, I call client.index() to process a base64 document, but I get "ResponseError: Exception" from client.index()

// KIBANA
PUT /descriptions
PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data"
      }
    }
  ]
}

// NODE.JS code
// The 'data' variable is base64 encoded
function indexBase64Data(data) {

    console.log(" ");
    console.log("calling client.index()");
    const index = client.index({
        id: 101,
        index: "descriptions",
        pipeline: "attachment",
        body: {
            "data": data
        }
    });
    index.then(() => {
        console.log("Index was successful...");
    },
    (error) => {
        console.error("ERROR in client.index: " + error);
    })

}

// ERROR from client.index() is: "ResponseError: Exception"

Should I be using something other than client.index()??
What am I doing wrong here?
-David

n2nelastic · August 4, 2019, 2:39pm

Maybe I should put my question a different way.

What is the Elasticsearch Node.js Client call to ingest a base64 encoded document/attachment into an existing index (here "descriptions").

spinscale · August 5, 2019, 10:00am

I am sure the responseerror is more verbose than just Exception - can you please share the full error message? You can also increase debugging by enabling stacktraces.

Did you install the field mapper attachment plugin before doing this?

n2nelastic · August 5, 2019, 3:02pm

Hi Alexander!

Adding "error_trace: true" to my call, the elasticsearch.log:


[2019-08-05T09:09:40,350][DEBUG][o.e.a.b.TransportBulkAction] [DESKTOP-SBLSQL0] failed to execute pipeline [attachment] for document [descriptions/_doc/101]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:194) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:133) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:100) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:427) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService.access$100(IngestService.java:70) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:355) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
... 11 more
Caused by: java.lang.IllegalArgumentException: Illegal base64 character 3a
at java.util.Base64$Decoder.decode0(Base64.java:743) ~[?:?]
at java.util.Base64$Decoder.decode(Base64.java:535) ~[?:?]
at java.util.Base64$Decoder.decode(Base64.java:558) ~[?:?]
at org.elasticsearch.ingest.IngestDocument.getFieldValueAsBytes(IngestDocument.java:183) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.attachment.AttachmentProcessor.execute(AttachmentProcessor.java:79) ~[?:?]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:123) ~[elasticsearch-7.3.0.jar:7.3.0]
... 9 more

Also, I tried looking for the plugin you suggested by Googling elastic.co, and there is no "field mapper attachment plugin" that I can find, only the following:

Mapper plugins:

Mapper Size Plugin
Mapper Murmur3 Plugin
Mapper Annotated Plugin

Please continue to help me on this.

Christian_Dahlqvist · August 5, 2019, 3:13pm

I believe this is the plugin you need. This processor is used within an ingest pipeline.

n2nelastic · August 5, 2019, 8:25pm

Hi Christian. I already do have this plugin installed. I suppose I should have mentioned it, but I thought it would be obvious since I am using a pipeline.

// Kibana
GET _ingest/pipeline

{
  "attachment" : {
    "description" : "Extract attachment information",
    "processors" : [
      {
        "attachment" : {
          "field" : "data"
        }
      }
    ]
  },
  "xpack_monitoring_6" : {
    "description" : "This pipeline upgrades documents from the older version of the Monitoring API to the newer version (7) by fixing breaking changes in those older documents before they are indexed from the older version (6).",
    "version" : 7000199,
    "processors" : [
      {
        "script" : {
          "source" : "ctx._type = null"
        }
      },
      {
        "gsub" : {
          "field" : "_index",
          "pattern" : "(.monitoring-\\w+-)6(-.+)",
          "replacement" : "$17$2"
        }
      }
    ]
  },
  "xpack_monitoring_7" : {
    "description" : "This is a placeholder pipeline for Monitoring API version 7 so that future versions may fix breaking changes.",
    "version" : 7000199,
    "processors" : [ ]
  }
}

Let me ask you this: is my approach using client.index() from the Elasticsearch Node.js Client correct in your opinion?

n2nelastic · August 5, 2019, 8:36pm

Alexander,

Also I should have mentioned that "ResponseError: Exception" is in fact all that is returned. Nothing more.

Adding "error_trace: true" to the client.index() call returned a lot, and I added part of the log file earlier.

spinscale · August 6, 2019, 7:58am

a part of the exception states

can you share the original request including the base64 string?

n2nelastic · August 6, 2019, 1:21pm

The base64 string:

DATA> data:application/octet-stream;base64,Ym9vazoxfGFiY3xkZWYKYm9vazoyfE1ha2luZyB0aGUgTW9kZXJuIFdvcmxkfFZhY2xhdiBTbWlsCmJvb2s6M3xUYXAgRGFuY2luZyB0byBXb3JrOiBXYXJyZW4gQnVmZmV0fENhcm9sIExvb21pcwpib29rOjZ8VGhlIEhlYWRzcGFjZSBHdWlkZSB0byBNZWRpdGF0aW9uIGFuZCBNaW5kZnVsbmVzc3xBbmR5IFB1ZGRpY29tYmUKYm9vazo3fDIxIExlc3NvbnMgZm9yIHRoZSAyMXN0IENlbnR1cnl8WXV2YWwgTm9haCBIYXJhcmkKYm9vazo4fEJhZCBCbG9vZHxKb2huIENhcnJleXJvdQpib29rOjl8QXJteSBvZiBOb25lOiBBdXRvbm9tb3VzIFdlYXBvbnMgYW5kIHRoZSBGdXR1cmUgb2YgV2FyfFBhdWwgU2NoYXJyZQpib29rOjEwfEVkdWNhdGVkfFRhcmEgV2VzdG92ZXIK

Original request to client.index() using "Elasticsearch Node.js Client"

function indexBase64Data(data) {

    console.log("DATA> " + data);
    console.log(" ");
    console.log("calling client.index()");
    const index = client.index({
        id: 102,
        index: "descriptions",
        pipeline: "attachment",
        error_trace: true,
        body: {
            "data": data
        }
    });
    index.then(() => {
        console.log("Index was successful...");
    },
    (error) => {
        console.log("index> " + index);
        console.error("ERROR in client.index: " + error);
    })

}

The elasticsearch.log file:

[2019-08-06T08:08:58,619][DEBUG][o.e.a.b.TransportBulkAction] [DESKTOP-SBLSQL0] failed to execute pipeline [attachment] for document [descriptions/_doc/101]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
	at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:194) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:133) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:100) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:427) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.IngestService.access$100(IngestService.java:70) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:355) [elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
	... 11 more
Caused by: java.lang.IllegalArgumentException: Illegal base64 character 3a
	at java.util.Base64$Decoder.decode0(Base64.java:743) ~[?:?]
	at java.util.Base64$Decoder.decode(Base64.java:535) ~[?:?]
	at java.util.Base64$Decoder.decode(Base64.java:558) ~[?:?]
	at org.elasticsearch.ingest.IngestDocument.getFieldValueAsBytes(IngestDocument.java:183) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.attachment.AttachmentProcessor.execute(AttachmentProcessor.java:79) ~[?:?]
	at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:123) ~[elasticsearch-7.3.0.jar:7.3.0]
	... 10 more

And as you can see, the ingest plugin is installed:

// KIBANA
GET /_cat/plugins?v

// Results in:
name            component         version
DESKTOP-SBLSQL0 ingest-attachment 7.3.0

THANK YOU!!! Alexander for continuing to help me on this.
I especially like the Elasticsearch Node.js Client, and I want to learn how to ingest base64 encoded files very much indeed.

David

spinscale · August 6, 2019, 9:32pm

try removing the prefix, marking this as application/octet-stream and just go with the raw base 64 string

n2nelastic · August 6, 2019, 11:48pm

Yes Alexander... that worked.

system · September 3, 2019, 11:57pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Exception when indexing pdf document Elasticsearch	11	2047	January 17, 2017
Error while using ingest attachment plugin on some docs Elasticsearch	13	1737	November 29, 2018
Ingest attachment Plugin exception : Elasticsearch	8	4423	January 19, 2017
ElasticsearchParseException using Ingest Attachment Processor Plugin in Elasticsearch 6.4.2 Elasticsearch	9	1716	April 17, 2019
Mapper parsing exception even after installed ingest attachment 5.0.1 Elasticsearch	5	1302	December 15, 2016

Exception on ingest an attachment using client.index()

Related topics