Mapping date in milliseconds to basic_date_time

[paolovalladolid]

I followed the instructions here:

format | Elasticsearch Guide [7.17] | Elastic

I ran this command as instructed:

PUT /_index_template/itential_jobs_template
{
	"index_patterns": ["itential-jobs-*"],
	"template": {
		"mappings": {
			"properties": {
				"start_time": {
					"format": "basic_date_time",
					"type": "date"
				}
			}
		}
	}
}

The start_time field is still being indexed into Elasticsearch as a number instead of as a timestamp.

"start_time": [
      1694806840000
    ]

Part of the problem is the source JSON had start_time as a float

"metrics": {
		"start_time": 1.694726079118E+12,
		"user": {
			"$oid": "64654876fb0d9c4f8d6cccc5"
		},
		"progress": 0.23076923076923078
	}

I do have my Logstash pipeline config copying the start_time out of the metrics JSON object before attempting to convert it to date.

[paolovalladolid]

I modified the pipeline config because according to the docs the date filter looks for seconds instead of milliseconds if the "UNIX" literal is used.

Adding to my problems, the ruby filter does not like my code.

input {
  file {
    path => "/var/log/mongodb/errored-jobs.json"
    start_position => "beginning"
    codec => json
    #codec => multiline {
    #  pattern => "^s"
    #  what => "previous"
    #
    #sincedb_path => "/dev/null"
  }
}

filter {
  mutate {
    remove_field => ["tasks", "transitions", "variables"]
    rename => {"[metrics][start_time]" => "start_time"}
  }

  ruby {
    code => "event.set('start_time', event.get('start_time')/1000))"
  }

  mutate {
    rename => {"_id" => "jobId"}
  }

  date {
    match => ["start_time", "UNIX"]
    target => "start_time"
    timezone => "UTC"
  }
}

Error message in logstash-plain.log

[2023-09-15T21:45:38,999][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<RuntimeError: unexpected error: (ruby filter code):2: syntax error, unexpected ')'
...event.get('start_time')/1000))

[paolovalladolid]

I reversed the single and double quotes to fix the ruby filter.

I also added more fields to remove as Logstash/ES started complaining about them, and I don't think we need them. Now documents get indexed with start_time as a timestamp instead of as a float.

input {
  file {
    path => "/var/log/mongodb/errored-jobs.json"
    start_position => "beginning"
    codec => json
    #codec => multiline {
    #  pattern => "^s"
    #  what => "previous"
    #
    #sincedb_path => "/dev/null"
  }
}

filter {
  mutate {
    remove_field => ["created", "last_updated", "tasks", "transitions", "variables", "watchers", "ancestors", "decorators"]
    rename => {"[metrics][start_time]" => "start_time"}
  }

  ruby {
    code => 'event.set("start_time", event.get("start_time") / 1000)'
  }

  mutate {
    rename => {"_id" => "jobId"}
  }

  date {
    match => ["start_time", "UNIX"]
    target => "start_time"
    timezone => "UTC"
  }
}