Ingest node: Invalid format...date and time is malformed

Elastic Stack Beats
1

I try to parse a csv file using a pipeline but got into problem with the date format. Appreciate any help!

Here is my pipiline:

{
  "pipeline": {
  "description" : "parse stat logs",
  "processors": [
      {
      "grok": {
        "field": "message",
        "patterns": ["%{DATESTAMP:log.datetime}\",\"%{HOSTNAME:log.machineName}\",\"%{HOSTNAME:log.domainDS}\",\"%{DATA:log.requestID}\",\"%{URIPATH:log.pageUrl}\",\"%{NUMBER:log.totalDuration}\",\"%{NUMBER:log.actionDuration}\",\"%{NUMBER:log.viewDuration}\",\"%{NUMBER:log.prefetchData-1-Duration}\",\"%{NUMBER:log.prefetchData-2-Duration}\",\"%{NUMBER:log.routingDuration}\",\"%{NUMBER:log.unknownDuration}\""],
        "on_failure": [
        	{
        		"set":{
        			"field": "ingestError",
        			"value": "{{ _ingest.on_failure_message }}"
        		}
        	},
           	{
    						"date": {
    						"field":"log.datetime",
    						"target_field":"log.datetime",
    						"formats": ["dd/MMM/yyyy HH:mm:ss.SSS"]
    						}
        	},
        		{
        		"set":{
        			"field": "log.datetime",
        			"value": "{{ @timestamp }}"
        		}
        	}
 
        ]
      }
    }
  ]
},
"docs":[
  {
    "_source": {"message":"\"7/31/2017 15:37:56.362\",\"EKL-DSDEV\",\"localhost\",\"+hn7XkuM+06ACvXijjo5Ww\",\"/modelos\",\"3417\",\"8\",\"538\",\"2026\",\"92\",\"2851\",\"-12\""}
  }
  ]
}

2017-08-15T18:16:52-04:00 DBG PublishEvents: 5 events have been published to elasticsearch in 9.9982ms.
2017-08-15T18:16:52-04:00 DBG Bulk item insert failed (i=0, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: "2017-08-15T22:14:15.635Z" is malformed at "17-08-15T22:14:15.635Z""}}},"header":{"processor_type":"date"}}
2017-08-15T18:16:52-04:00 DBG Bulk item insert failed (i=1, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: "2017-08-15T22:14:15.635Z" is malformed at "17-08-15T22:14:15.635Z""}}},"header":{"processor_type":"date"}}
2017-08-15T18:16:52-04:00 DBG Bulk item insert failed (i=2, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: "2017-08-15T22:14:15.635Z" is malformed at "17-08-15T22:14:15.635Z""}}},"header":{"processor_type":"date"}}
2017-08-15T18:16:52-04:00 DBG Bulk item insert failed (i=3, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: "2017-08-15T22:14:15.635Z" is malformed at "17-08-15T22:14:15.635Z""}}},"header":{"processor_type":"date"}}
2017-08-15T18:16:52-04:00 DBG Bulk item insert failed (i=4, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"unable to parse date [2017-08-15T22:14:15.635Z]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: "2017-08-15T22:14:15.635Z" is malformed at "17-08-15T22:14:15.635Z""}}},"header":{"processor_type":"date"}}
2017-08-15T18:16:52-04:00 DBG Registry file updated. 6 states written.
2017-08-15T18:16:52-04:00 INFO Total non-zero values: filebeat.harvester.closed=6 filebeat.harvester.started=6 libbeat.es.call_count.PublishEvents=47635 libbeat.es.publish.read_bytes=21396008 libbeat.es.publish.write_bytes=117496007 libbeat.es.published_and_acked_events=493 libbeat.es.published_but_not_acked_events=141840 libbeat.publisher.published_events=2037 registrar.states.current=6 registrar.writes=1
2017-08-15T18:16:52-04:00 INFO Uptime: 2m36.7110224s

2

The date format looks to be incorrect. Try using mm/DD/yyyy HH:mm:ss.SSS.

Also the pipeline looks to be setup incorrectly because you have the date and set processors defined as part of the on_failure handler. I'm guessing you mean for those to come after the grok processor as part of the main pipeline.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.