Ingest Node

[Changra (neha changra)]

I added the ingest pipeline with the name "transfer-pipeline" in the logstash output as below,

output {

if "mm" in [service_name]

{

elasticsearch {

ssl => true

ssl_certificate_verification => true

ilm_rollover_alias => "mm"

ilm_pattern => "{now/d}-000001"

ilm_policy => "rollover-size-wise"

pipeline => "transf"

}

}

But it constantly gives me this error while reloading the pipeline "reason"=>"unable to parse date ", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"cannot parse empty date"}}}}}

Can u please help me out what could be the reason , the ingest pipeline is as below

PUT _ingest/pipeline/transfer/

{

"description" : "log",

"processors" : [

{

"set": {

"field": "request_time",

"value": "{{_source.request_time}}"

}

},

{

"set": {

"field": "response_time",

"value":"{{_source.response_time}}"

}

},

{

"date":{

"field":"request_time",

"formats":["yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"]

}

},

{

"date":{

"field":"response_time",

"formats":["yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"]

}

},

{

"script": {

"lang": "painless",

"source": """

[stephenb (Stephen Brown)]

Hi @Changra Welcome to the community.

So that we can help you can you please edit your post and format your code using the </> format button.

2nd I see this

pipeline => "transf"

and yet you said it should be transfer

2nd these are probably redundant, if the _source.request_time exist you do not need to set it again

{
"set": {
  "field": "request_time",
  "value": "{{_source.request_time}}"
}

Finally you probably need guards something like this, you probably have empty fields.

"date":{
  "if": "ctx.response_time != null",
  "field":"response_time",
  "formats":["yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"]
}

There is also a nice section on handling errors in ingest pipelines here

[Changra (neha changra)]

HI Stephenb,

Thank you for the reply i changed the pipeline code a bit but the still see the error in my script sectipn below is the ingest pipeline code

PUT _ingest/pipeline/transfer-pipeline/
{
  "description" : "transfer of ownership logs",
  "processors" : [         
    {
    "set": {
      "field": "request_time",
      "value": "{{_source.request_time}}"
      
    }
  },
  {
    "set": {
      "field": "response_time",
      "value":"{{_source.response_time}}"
     
    }
  },
  {
  "date":{
    "if": "ctx.request_time != null",
    "field":"request_time", 
    "formats":["yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"]
  }
  },
{
  "date":{
    "if":"ctx.response_time != null",
    "field":"response_time", 
    "formats":["yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"]
  }
  },
  {
  "script": {
    "lang": "painless",
    "source": """
    
 String datetime = ctx['request_time'];
ZonedDateTime zdt = ZonedDateTime.parse(datetime);
String datetime_new = ctx['response_time'];
ZonedDateTime edt = ZonedDateTime.parse(datetime_new);

long request_time_milli = zdt.toInstant().toEpochMilli();
ctx['request_time_milli']= request_time_milli;

long response_time_milli = edt.toInstant().toEpochMilli();
ctx['response_time_milli']= response_time_milli;
if (response_time_milli >= request_time_milli)
{
        ctx['duration'] = (ctx['response_time_milli'] - ctx['request_time_milli']); 
 }
     """
  }
  }




here is the error which i see


"script"=>"\n    \n String datetime = ctx['request_time'];\nZonedDateTime zdt = ZonedDateTime.parse(datetime);\nString datetime_new = ctx['response_time'];\nZonedDateTime edt = ZonedDateTime.parse(datetime_new);\n\nlong request_time_milli = zdt.toInstant().toEpochMilli();\nctx['request_time_milli']= request_time_milli;\n\nlong response_time_milli = edt.toInstant().toEpochMilli();\nctx['response_time_milli']= response_time_milli;\nif (response_time_milli >= request_time_milli)\n{\n        ctx['duration'] = (ctx['response_time_milli'] - ctx['request_time_milli']); \n }\n     ", "lang"=>"painless", "position"=>{"offset"=>86, "start"=>60, "end"=>104}, "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Text '' could not be parsed at index 0



here is the output section 

output {
if "mm" in [service_name]
{
elasticsearch {Preformatted text
ssl => true
ssl_certificate_verification => true
ilm_rollover_alias => "provisioning_service"
ilm_pattern => "{now/d}-000001"
ilm_policy => "rollover-size-wise"
pipeline => "transfer-pipeline"
}
}

I still get the same error

["index", {:_id=>nil, :_index=>"provisioning_service", :routing=>nil, :_type=>"_doc", :pipeline=>"transfer-pipeline"}, #LogStash::Event:0x4a13394], :response=>{"index"=>{"_index"=>"provisioning_service", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"unable to parse date ", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"cannot parse empty date"}}}}}

[stephenb (Stephen Brown)]

Well that is a lot more complex code you are going to need to debug, you still have and empty or an empty string date somewhere.

Did you read the docs on error handling you can put in messages

Also you can try the simulate API to make it easier to debug.