Ingest Custom date Format data to Elastic Index

[Debasis_Mallick]

Hi Team,

We are trying to ingest a custom date format data to Elastic Index where the field mapping is as below.

 "edate":{
        "type":"date",
        "format":["yyyy-mm-dd"] 
	  	}

We are using filebeat to ingest data to Elasticsearch. In the record we are getting the value as "2024-08-25" . We can see the other record data is getting populated but the edate field data is not getting populated.

Could you please advise if anything we are missing to add in the index settings.

Thanks,
Debasis

[dadoonet (David Pilato)]

It looks good to me.

[Debasis_Mallick]

@dadoonet thanks for response.
Actually I want to convert the timestamp data (which receives as epoch_millis) in csv file . I want to convert the same timestamp value to a yyyy-mm-dd format and store in index , with the help of processor which is defined in pipeline.

Below is my pipeline definition.

PUT _ingest/pipeline/parse_elastic_data_v1
{
  "processors": [
    {
      "csv": {
        "description": "Parse elastic Data From CSV Files",
        "field": "message",
        "target_fields": ["sequence",
        "component","tenant",
        "service_id","session_id",
        "timestamp","edr_version",
        "prov_version","action_id",
        "action_extra_info","rule_type",
        "rule_id", "traffic_type",
        "ac_group","ac_version",
        "opcode","cg_plan", "cg_gt",
        "cg_ssn","cg_operator",
        "cd_plan","cd_gt",
        "cd_ssn", "cd_operator",
        "imsi", "location_operator",
        "location_timestamp","sms_content",
        "message_id","fragment_number",
        "sms_analytics_case_id","msisdn",
        "msisdn_ton","msisdn_npi","tpoa",        
        "sender_type","sms_hub_type",      
        "embedded_url","domain",      
        "msg_source","smsc",      
        "pid","dcs","message_type"],
        "separator": ",",
        "ignore_missing":true,
        "trim":true
      }
      },
	  {
	  "remove": {"field": "message" }
	  },
	  { "date": {
        "field": "timestamp",
        "target_field": "timestamp_em",
        "formats": ["epoch_millis"]
      }
      },
	   {"date": {
        "field": "timestamp",
        "target_field": "timestamp_tf",
        "formats": ["epoch_millis"]
      }
    }
 ]
}

The target fields (timestamp_em,timestamp_tf) are index cols which are defined while creating Index. Below is the snapshot of Index creation settings.

  "timestamp_em":{
        "type":"date",
        "format": "epoch_millis"
	  	},
	  	"timestamp_tf":{
        "type":"date",
        "format": ["yyyy-MM-dd'T'HH:mm:ss.SSSZZ"]
	  	  
    	}
  }

But while doing ingest process through filebeat getting error as "failed to parse field [timestamp_em] of type [date] " .

Any suggestion how we can proceed further.

Thanks,
Debasis

[dadoonet (David Pilato)]

Could you reproduce the problem with the _simulate API?

If you can, please share it here and we can iterate from that.
If you can't, check that you are not sending the CSV header to Elasticsearch. That's a common mistake.