# Ingest Custom date Format data to Elastic Index

**URL:** <https://discuss.elastic.co/t/ingest-custom-date-format-data-to-elastic-index/356736>\
**Category:** Elasticsearch\
**Created:** [April 4, 2024, 5:22am UTC](https://discuss.elastic.co/t/ingest-custom-date-format-data-to-elastic-index/356736 "2024-04-04T05:22:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [April 4, 2024, 5:22am UTC](https://discuss.elastic.co/t/ingest-custom-date-format-data-to-elastic-index/356736/1 "2024-04-04T05:22:55Z")

</div>

Hi Team,

We are trying to ingest a custom date format data to Elastic Index where the field mapping is as below.

```auto
 "edate":{
        "type":"date",
        "format":["yyyy-mm-dd"] 
	  	}

```

We are using filebeat to ingest data to Elasticsearch. In the record we are getting the value as `"2024-08-25"` . We can see the other record data is getting populated but the `edate` field data is not getting populated.

Could you please advise if anything we are missing to add in the index settings.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 4, 2024, 7:44am UTC](https://discuss.elastic.co/t/ingest-custom-date-format-data-to-elastic-index/356736/2 "2024-04-04T07:44:34Z")

</div>

It looks good to me.

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [April 4, 2024, 10:04am UTC](https://discuss.elastic.co/t/ingest-custom-date-format-data-to-elastic-index/356736/3 "2024-04-04T10:04:06Z")

</div>

@dadoonet thanks for response.  
Actually I want to convert the timestamp data (which receives as epoch\_millis) in csv file . I want to convert the same timestamp value to a yyyy-mm-dd format and store in index , with the help of processor which is defined in pipeline.

Below is my pipeline definition.

```auto
PUT _ingest/pipeline/parse_elastic_data_v1
{
  "processors": [
    {
      "csv": {
        "description": "Parse elastic Data From CSV Files",
        "field": "message",
        "target_fields": ["sequence",
        "component","tenant",
        "service_id","session_id",
        "timestamp","edr_version",
        "prov_version","action_id",
        "action_extra_info","rule_type",
        "rule_id", "traffic_type",
        "ac_group","ac_version",
        "opcode","cg_plan", "cg_gt",
        "cg_ssn","cg_operator",
        "cd_plan","cd_gt",
        "cd_ssn", "cd_operator",
        "imsi", "location_operator",
        "location_timestamp","sms_content",
        "message_id","fragment_number",
        "sms_analytics_case_id","msisdn",
        "msisdn_ton","msisdn_npi","tpoa",        
        "sender_type","sms_hub_type",      
        "embedded_url","domain",      
        "msg_source","smsc",      
        "pid","dcs","message_type"],
        "separator": ",",
        "ignore_missing":true,
        "trim":true
      }
      },
	  {
	  "remove": {"field": "message" }
	  },
	  { "date": {
        "field": "timestamp",
        "target_field": "timestamp_em",
        "formats": ["epoch_millis"]
      }
      },
	   {"date": {
        "field": "timestamp",
        "target_field": "timestamp_tf",
        "formats": ["epoch_millis"]
      }
    }
 ]
}

```

The target fields (`timestamp_em,timestamp_tf`) are index cols which are defined while creating Index. Below is the snapshot of Index creation settings.

```auto
  "timestamp_em":{
        "type":"date",
        "format": "epoch_millis"
	  	},
	  	"timestamp_tf":{
        "type":"date",
        "format": ["yyyy-MM-dd'T'HH:mm:ss.SSSZZ"]
	  	  
    	}
  }

```

But while doing ingest process through filebeat getting error as "`failed to parse field [timestamp_em] of type [date]` " .

Any suggestion how we can proceed further.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 4, 2024, 12:26pm UTC](https://discuss.elastic.co/t/ingest-custom-date-format-data-to-elastic-index/356736/4 "2024-04-04T12:26:48Z")

</div>

Could you reproduce the problem with the `_simulate` API?

If you can, please share it here and we can iterate from that.  
If you can't, check that you are not sending the CSV header to Elasticsearch. That's a common mistake.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 2, 2024, 12:27pm UTC](https://discuss.elastic.co/t/ingest-custom-date-format-data-to-elastic-index/356736/5 "2024-05-02T12:27:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.