Java client put logstash pipeline last_modified error

Elastic Stack Elasticsearch
1

Hi,
I'm using the new Java elasticsearch client to PUT a logstash pipeline (client version 8.4.1, server 8.4.1)

client.logstash().putPipeline(b -> b
                        .id(id)
                        .pipeline(p -> p
                                .description(id)
                                .lastModified(DateTime.ofEpochMilli(System.currentTimeMillis()))
                                .pipeline("input {}\n filter { grok {} }\n output {}")
                                .username("elastic")
                                .pipelineMetadata(m -> m
                                        .type("logstash_pipeline")
                                        .version("1")
                                )
                                .pipelineSettings(s -> s
                                        .pipelineBatchDelay(50)
                                        .pipelineBatchSize(125)
                                        .pipelineWorkers(1)
                                        .queueCheckpointWrites(1024)
                                        .queueMaxBytesNumber(1)
                                        .queueMaxBytesUnits("gb")
                                        .queueType("memory")
                                )
                        )
                )

Bu I get an error on the last_modified field:

DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "PUT /_logstash/pipeline/org_dataset HTTP/1.1[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "X-Elastic-Client-Meta: es=8.4.1,jv=17,hl=2,t=8.4.1,hc=4.1.5[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "User-Agent: elastic-java/8.4.1 (Java/17.0.4)[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "Accept: application/vnd.elasticsearch+json; compatible-with=8[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "Content-Length: 1593[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "Content-Type: application/vnd.elasticsearch+json; compatible-with=8[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "Host: xxxxxx:9243[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "Authorization: Basic xxxxxxx[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "[\r][\n]"
DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "{"description":"org_dataset","last_modified":1662024280651,"pipeline_metadata":{"type":"logstash_pipeline",
[...]

DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "HTTP/1.1 400 Bad Request[\r][\n]"
DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "Content-Length: 287[\r][\n]"
DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "Content-Type: application/json;charset=utf-8[\r][\n]"
DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "X-Cloud-Request-Id: xxxxxxxxx[\r][\n]"
DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "X-Elastic-Product: Elasticsearch[\r][\n]"
DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "X-Found-Handling-Cluster: xxxxxxxxxx[\r][\n]"
DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "X-Found-Handling-Instance: instance-0000000000[\r][\n]"
DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "Date: Thu, 01 Sep 2022 09:15:08 GMT[\r][\n]"
DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "[\r][\n]"
DEBUG [2022-09-01 09:15:08,399] org.apache.http.wire: http-outgoing-1 << "{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:46] [pipeline] last_modified doesn't support values of type: VALUE_NUMBER"}],"type":"x_content_parse_exception","reason":"[1:46] [pipeline] last_modified doesn't support values of type: VALUE_NUMBER"},"status":400}"

Am I doing something wrong or is it a bug?

Cheers,

2

@swallez maybe?

3

nobody?

4

Check the documentation.

last_modified
(Required, string) Date the pipeline was last updated. Must be in the yyyy-MM-dd'T'HH:mm:ss.SSSZZ strict_date_time format.

The value for the last_modified needs to be in this format: yyyy-MM-dd'T'HH:mm:ss.SSSZZ

But from your log you are sending it as a number, with the date in epoch time.

"{"description":"org_dataset","last_modified":1662024280651

5

As you can see, I'm sending it as a DateTime object which is what is required by the lastModified(DateTime value) method of the java client.

6

This is not what your log shows, the Debug line of your request shows that the last_modified is a number.

DEBUG [2022-09-01 09:24:40,766] org.apache.http.wire: http-outgoing-1 >> "{"description":"org_dataset","last_modified":1662024280651

Maybe this is a bug then? You should open an issue to see if someone can help you better, but have you tried to format the date to see if it works?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.