415 error when uploading ndjson to kibana

[amesl]

I need to upload an ndjson dashboard file to the kibana endpoint in Java. The below command is successful with the ndjson files:

curl -X POST api/saved_objects/_import?overwrite=true -H "kbn-xsrf: true" --form file=@file.ndjson

When calling the curl command from Java with the ndjson files it runs successfully, however we would prefer to use Java connections. When attempting to do the same thing with an HttpsURLConnection we receive the 415 error (Unsupported media type) with the same files.

Any ideas why this continues to return 415 error? We are using kibana 8.4.2

							    URL url = new URL("https://" + host + "/api/saved_objects/_import?overwrite=true");
							    HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
							    conn.setDoOutput(true);
							    conn.setDoInput(true);
							    conn.setUseCaches(false);
	
							    conn.setRequestMethod("POST");
							    conn.setRequestProperty("kbn-xsrf", "true");
							    conn.setRequestProperty(HttpHeaders.CONTENT_TYPE, "application/x-ndjson; charset=utf-8");
							    conn.setRequestProperty(HttpHeaders.ACCEPT, "*/*");
							   
                                // Reads in ndjson file 
							    String json = Files.readString(file.toPath());
							    try (OutputStream os = conn.getOutputStream())
							    {
							    	byte[] input = json.getBytes("utf-8");
							    	os.write(input);
							    	os.flush();
								    os.close();
							    }
							    							    						    							    
							    int responseCode = conn.getResponseCode();
							    String response = conn.getResponseMessage();

[gsoldevila (Gerard Soldevila)]

Hello, and welcome to the community!

I believe the problem is that the /api/saved_objects/_import endpoint expects the request to be of type multipart/form-data.

Please note that this affects the way your NDJSON is appended to the request. The Java API provides the necessary tools for that approach, and there are some examples of usage on the internet. Please let us know if you have any trouble attaching your NDJSON that way.

Kind regards

[amesl]

Thanks, and I appreciate your response.

I modified the code to handle the ndjson files as multipart files (using the HttpPostMultipart class) but the connection is returning 400 errors. The error is being returned from the line:

throw new IOException("Server returned non-OK status: " + status);

I added some logging to the HttpPostMultipart class and found that the Content-Type is set to null:

writer.append("Content-Type: " + URLConnection.guessContentTypeFromName(fileName)).append(LINE);

The URLConnection.guessContentTypeFromName returns null for the *.ndjson file names.

Since the URLConnection.guessContentTypeFromName returns null should it be hardcoded? The files are ndjson format.

[gsoldevila (Gerard Soldevila)]

Hello!

I am not familiar with the HttpPostMultipart class, is it part of the standard Java API?

When using multipart POST, the Content-Type header should have the multipart/form-data value.
If the library you are using is not setting it for you, then you might have to hardcode it.

Kind regards

[amesl]

I truly appreciate your responses! Any response is better than none

The HttpPostMultipart class is the code from the example you linked to. The first part shows how to write the HttpPostMultipart class.

In the HttpPostMultipart class example the Content-Type header is hardcoded to multipart/form-data, this is the header on the multipart/form-data message:

httpConn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);

However, in the addFilePart method that attaches the file the Content-Type is pulled from the file type.

writer.append("Content-Disposition: form-data; name=\"" + fieldName + "\"; filename=\"" + fileName + "\"").append(LINE);
writer.append("Content-Type: " + URLConnection.guessContentTypeFromName(fileName)).append(LINE);
writer.append("Content-Transfer-Encoding: binary").append(LINE);

I changed this to the following because the URLConnection.guessContentTypeFromName(fileName) returned null:

writer.append("Content-Disposition: form-data; name=\"" + fieldName + "\"; filename=\"" + fileName + "\"").append(LINE);
writer.append("Content-Type: application/x-ndjson").append(LINE);
writer.append("Content-Transfer-Encoding: binary").append(LINE);

So the headers on the POST would look like:

Content-Type: multipart/form-data; boundary=XXXX
...

Content-Disposition: form-data; name=updateFile; filename=dashboard1.ndjson
Content-Type: application/x-ndjson
Content-Transfer-Encoding: binary
...

Each form or message in the multipart message should have a header indicating the contents. The kibana endpoint is returning 400 for messages at this point.

[gsoldevila (Gerard Soldevila)]

Sorry I just posted the link as an illustrative example, did not pay too much attention to the class names.

Comparing your requests against the ones performed by Kibana UI when using the Import feature, there's a couple differences:

• The name of the form field is simply "file" instead of the "updateFile" you are currently using. I believe that should make a difference (considering this validation).
• The Content-Transfer-Encoding header is not sent by the UI, so we can get rid of it probably.

Kind regards

[amesl]

Thank you - I'll certainly try this out.

[amesl]

@gsoldevila THANK YOU! That was the secret sauce to getting this working. Changing the form field to "file" and getting rid of the Content-Transfer-Encoding header were the missing pieces.

[amesl]

Just to document what worked in Java to upload the ndjson files to Kibana. Our server is https and we used a username and password for authentication.

Use the link referenced in first response then made the following changes

Implementing the HttpPostMultipart in the application:

URL url = new URL("https://" + host + "/api/saved_objects/_import?overwrite=true");

Map<String, String> headers = new HashMap<>();
headers.put(HttpHeaders.AUTHORIZATION, getBasicAuthenticationHeader(user, password));
headers.put("kbn-xsrf", "true");
						    		
HttpPostMultipart multipart = new HttpPostMultipart(url.toString(), "utf-8", headers);
// has to be named file to work with Kibana - the file value is a File object
multipart.addFilePart("file", file);
							    
String response = multipart.finish();

Modified the addFilePart method in the HttpPostMultipart class to work for Kibana POST.

• Set the Content-Type to application/x-ndjson
• removed the Content-Transfer-Encoding

public void addFilePart(String fieldName, File uploadFile) throws IOException 
{
    String fileName = uploadFile.getName();
    writer.append("--" + boundary).append(LINE);
    writer.append("Content-Disposition: form-data; name=\"" + fieldName + "\"; filename=\"" + fileName + "\"").append(LINE);
    writer.append("Content-Type: application/x-ndjson").append(LINE);
    writer.append(LINE);
    writer.flush();

These configurations worked for me.

[gsoldevila (Gerard Soldevila)]

Thanks for sharing your working configuration! I'm sure it'll be very useful for future reference.

Cheers

[amesl]

Thanks for all of your help! It got me pointed in the right direction.