Convert fields from text into date format using in Logstash config file

Elastic Stack Logstash
1

Logstash conf file:

filter {
json {
source => "message"
}

}

output {
elasticsearch {
hosts =>"http://xx.xx.xx.xx.com"
index => "sample_data_%{+YYYY.MM.dd}"
user => "elastic"
password => "xxxxxxxxx"
}

}

Index Mappings:

{
"mappings": {
"_doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"order": {
"properties": {
"createdOn": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"items": {
"properties": {
"itemId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"orderID": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"totalPrice": {
"type": "long"
},
"updatedOn": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
}

Sample message stored in Elasticsearch:

{
"_index": "sample_data_2024.07.05",
"_type": "_doc",
"_id": "pcABgpAB1lurpuJXc5JB",
"_version": 1,
"_score": null,
"_source": {
"order": {
"createdOn": "2024-05-27 13:23:29",
"totalPrice": 120,
"items": {
"itemId": "AG000284"
},
"updatedOn": "2024-05-27 13:23:29",
"orderID": "DE--499"
},
"@timestamp": "2024-07-05T08:27:31.407Z",
"@version": "1",
},
"fields": {
"@timestamp": [
"2024-07-05T08:27:31.407Z"
]
},
"sort": [
1720168051407
]
}

*We need HELP to store the fields as date format using logstash before store the data into Index like below but createdOn & updatedOn fields stored as text format *

"createdOn": {
"type": "date"
}

"updatedOn": {
"type": "date"
}

2

You could try

    date { match => [ "[order][createdOn]", "YYYY-MM-dd HH:mm:ss" ] target => "[order][createdOn]" }
    date { match => [ "[order][updatedOn]", "YYYY-MM-dd HH:mm:ss" ] target => "[order][updatedOn]" }

which (for me, in GMT-0400) results in

         "order" => {
    "totalPrice" => 120,
     "createdOn" => 2024-05-27T17:23:29.000Z,
                ....
     "updatedOn" => 2024-05-27T17:23:29.000Z
},

Set the timezone option on the date filters if you need to.

1 Like
3

Thanks ! we added

date { match => [ "[order][createdOn]", "YYYY-MM-dd HH:mm:ss" ] target => "[order][createdOn]" }
date { match => [ "[order][updatedOn]", "YYYY-MM-dd HH:mm:ss" ] target => "[order][updatedOn]" }

and then ingested the data to Logstash via Source system. Document stored and the fields createdOn & updatedOn stored as 'date' data type in the Index Mappings

"createdOn": {
"type": "date"
},
"updatedOn": {
"type": "date"
}

image
image1841×506 136 KB