Cannot convert date to string by mutate

Elastic Stack Logstash
1

I want to force my search_varchar_1 data type as string, but it always auto detect my data and parse as date type. Is there something I missed?

My config

filter {
	split {}
	csv {
		separator => ","
		skip_header => "true"
		columns => ["log_no","log_dtm","operation_cd","correlation_id","no_hash","search_varchar_1","search_varchar_2","search_varchar_3","search_varchar_4","search_varchar_5","search_varchar_6","search_varchar_7","search_varchar_8","search_varchar_9","search_varchar_10","search_int_1","search_int_2","search_int_3","search_int_4","search_int_5","search_int_6","search_int_7","search_int_8","search_int_9","search_int_10","birth_month","birth_year","login_flag"]
		convert => {
			"search_int_1" => "integer"
			"search_int_2" => "integer"
			"search_int_3" => "integer"
			"search_int_4" => "integer"
			"search_int_5" => "integer"
			"search_int_6" => "integer"
			"search_int_7" => "integer"
			"search_int_8" => "integer"
			"search_int_9" => "integer"
			"search_int_10" => "integer"
			"birth_year" => "integer"
			"birth_month" => "integer"
		}
	}
	mutate {
		convert => {
			"search_varchar_1" => "string"
		}
	}
}

My data

log_no,log_dtm,operation_cd,correlation_id,no_hash,search_varchar_1,search_varchar_2,search_varchar_3,search_varchar_4,search_varchar_5,search_varchar_6,search_varchar_7,search_varchar_8,search_varchar_9,search_varchar_10,search_int_1,search_int_2,search_int_3,search_int_4,search_int_5,search_int_6,search_int_7,search_int_8,search_int_9,search_int_10,birth_month,birth_year,login_flag
1,24-NOV-20 11.35.18.445000000 PM,operation,17,,E,,,,,,,,,,,,,,,,,,,,,,N
4,24-NOV-20 11.35.18.445000000 PM,operation,17,,E,,,,,,,,,,,,,,,,,,,,,,N
5,24-NOV-20 11.35.18.445000000 PM,operation,17,,E,,,,,,,,,,,,,,,,,,,,,,N
6,24-NOV-20 11.35.18.445000000 PM,operation,17,,E,,,,,,,,,,,,,,,,,,,,,,N
3,24-NOV-20 11.35.18.445000000 PM,operation,17,,2022-02-01T00:00:18.224+08:00,,,,,,,,,,,,,,,,,,,,,,N
7,24-NOV-20 11.35.18.445000000 PM,operation,17,,2022-02-01,,,,,,,,,,,,,,,,,,,,,,N
2

At which phase "detect as date type" happen? Indexing to Elasticsearch?

3

In elasticsearch, if dynamic mapping and date detection are enabled then if it looks like a date it will get mapped as a date.

4

Thank you, is there any way to force the field to be mapped as string even it contains date format? Because I am only a consumer of my company's Elasticsearch service, which I don't have access control, thank you.

5

Elasticsearch does the mapping. You can either disable dynamic mapping or use an index template to set the mapping.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.