Exception on ingest an attachment using client.index()

Elastic Stack Elasticsearch
1

I have properly set up an index and pipeline to ingest base64 encoded documents.

Using the elasticsearch client, I call client.index() to process a base64 document, but I get "ResponseError: Exception" from client.index()

// KIBANA
PUT /descriptions
PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data"
      }
    }
  ]
}

// NODE.JS code
// The 'data' variable is base64 encoded
function indexBase64Data(data) {

    console.log(" ");
    console.log("calling client.index()");
    const index = client.index({
        id: 101,
        index: "descriptions",
        pipeline: "attachment",
        body: {
            "data": data
        }
    });
    index.then(() => {
        console.log("Index was successful...");
    },
    (error) => {
        console.error("ERROR in client.index: " + error);
    })

}

// ERROR from client.index() is: "ResponseError: Exception"

Should I be using something other than client.index()??
What am I doing wrong here?
-David

2

Maybe I should put my question a different way.

What is the Elasticsearch Node.js Client call to ingest a base64 encoded document/attachment into an existing index (here "descriptions").

3

I am sure the responseerror is more verbose than just Exception - can you please share the full error message? You can also increase debugging by enabling stacktraces.

Did you install the field mapper attachment plugin before doing this?

4

Hi Alexander!

Adding "error_trace: true" to my call, the elasticsearch.log:


[2019-08-05T09:09:40,350][DEBUG][o.e.a.b.TransportBulkAction] [DESKTOP-SBLSQL0] failed to execute pipeline [attachment] for document [descriptions/_doc/101]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:194) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:133) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:100) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:427) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService.access$100(IngestService.java:70) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:355) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
... 11 more
Caused by: java.lang.IllegalArgumentException: Illegal base64 character 3a
at java.util.Base64$Decoder.decode0(Base64.java:743) ~[?:?]
at java.util.Base64$Decoder.decode(Base64.java:535) ~[?:?]
at java.util.Base64$Decoder.decode(Base64.java:558) ~[?:?]
at org.elasticsearch.ingest.IngestDocument.getFieldValueAsBytes(IngestDocument.java:183) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.ingest.attachment.AttachmentProcessor.execute(AttachmentProcessor.java:79) ~[?:?]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:123) ~[elasticsearch-7.3.0.jar:7.3.0]
... 9 more

Also, I tried looking for the plugin you suggested by Googling elastic.co, and there is no "field mapper attachment plugin" that I can find, only the following:

Mapper plugins:

  • Mapper Size Plugin
  • Mapper Murmur3 Plugin
  • Mapper Annotated Plugin

Please continue to help me on this.

5

I believe this is the plugin you need. This processor is used within an ingest pipeline.

6

Hi Christian. I already do have this plugin installed. I suppose I should have mentioned it, but I thought it would be obvious since I am using a pipeline.

// Kibana
GET _ingest/pipeline

{
  "attachment" : {
    "description" : "Extract attachment information",
    "processors" : [
      {
        "attachment" : {
          "field" : "data"
        }
      }
    ]
  },
  "xpack_monitoring_6" : {
    "description" : "This pipeline upgrades documents from the older version of the Monitoring API to the newer version (7) by fixing breaking changes in those older documents before they are indexed from the older version (6).",
    "version" : 7000199,
    "processors" : [
      {
        "script" : {
          "source" : "ctx._type = null"
        }
      },
      {
        "gsub" : {
          "field" : "_index",
          "pattern" : "(.monitoring-\\w+-)6(-.+)",
          "replacement" : "$17$2"
        }
      }
    ]
  },
  "xpack_monitoring_7" : {
    "description" : "This is a placeholder pipeline for Monitoring API version 7 so that future versions may fix breaking changes.",
    "version" : 7000199,
    "processors" : [ ]
  }
}

Let me ask you this: is my approach using client.index() from the Elasticsearch Node.js Client correct in your opinion?

7

Alexander,

Also I should have mentioned that "ResponseError: Exception" is in fact all that is returned. Nothing more.

Adding "error_trace: true" to the client.index() call returned a lot, and I added part of the log file earlier.

8

a part of the exception states

can you share the original request including the base64 string?

9

The base64 string:

DATA> data:application/octet-stream;base64,Ym9vazoxfGFiY3xkZWYKYm9vazoyfE1ha2luZyB0aGUgTW9kZXJuIFdvcmxkfFZhY2xhdiBTbWlsCmJvb2s6M3xUYXAgRGFuY2luZyB0byBXb3JrOiBXYXJyZW4gQnVmZmV0fENhcm9sIExvb21pcwpib29rOjZ8VGhlIEhlYWRzcGFjZSBHdWlkZSB0byBNZWRpdGF0aW9uIGFuZCBNaW5kZnVsbmVzc3xBbmR5IFB1ZGRpY29tYmUKYm9vazo3fDIxIExlc3NvbnMgZm9yIHRoZSAyMXN0IENlbnR1cnl8WXV2YWwgTm9haCBIYXJhcmkKYm9vazo4fEJhZCBCbG9vZHxKb2huIENhcnJleXJvdQpib29rOjl8QXJteSBvZiBOb25lOiBBdXRvbm9tb3VzIFdlYXBvbnMgYW5kIHRoZSBGdXR1cmUgb2YgV2FyfFBhdWwgU2NoYXJyZQpib29rOjEwfEVkdWNhdGVkfFRhcmEgV2VzdG92ZXIK

Original request to client.index() using "Elasticsearch Node.js Client"

function indexBase64Data(data) {

    console.log("DATA> " + data);
    console.log(" ");
    console.log("calling client.index()");
    const index = client.index({
        id: 102,
        index: "descriptions",
        pipeline: "attachment",
        error_trace: true,
        body: {
            "data": data
        }
    });
    index.then(() => {
        console.log("Index was successful...");
    },
    (error) => {
        console.log("index> " + index);
        console.error("ERROR in client.index: " + error);
    })

}

The elasticsearch.log file:

[2019-08-06T08:08:58,619][DEBUG][o.e.a.b.TransportBulkAction] [DESKTOP-SBLSQL0] failed to execute pipeline [attachment] for document [descriptions/_doc/101]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
	at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:194) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:133) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:100) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:427) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.IngestService.access$100(IngestService.java:70) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:355) [elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character 3a
	... 11 more
Caused by: java.lang.IllegalArgumentException: Illegal base64 character 3a
	at java.util.Base64$Decoder.decode0(Base64.java:743) ~[?:?]
	at java.util.Base64$Decoder.decode(Base64.java:535) ~[?:?]
	at java.util.Base64$Decoder.decode(Base64.java:558) ~[?:?]
	at org.elasticsearch.ingest.IngestDocument.getFieldValueAsBytes(IngestDocument.java:183) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.ingest.attachment.AttachmentProcessor.execute(AttachmentProcessor.java:79) ~[?:?]
	at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:123) ~[elasticsearch-7.3.0.jar:7.3.0]
	... 10 more

And as you can see, the ingest plugin is installed:

// KIBANA
GET /_cat/plugins?v

// Results in:
name            component         version
DESKTOP-SBLSQL0 ingest-attachment 7.3.0

THANK YOU!!! Alexander for continuing to help me on this.
I especially like the Elasticsearch Node.js Client, and I want to learn how to ingest base64 encoded files very much indeed.

  • David
10

try removing the prefix, marking this as application/octet-stream and just go with the raw base 64 string

11

Yes Alexander... that worked.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.