How to insert date type column of a csv file in the format "dd:MM:YYYY HH:mm:ss" into elasticsearch through logstash config

Elastic Stack Logstash
#1

I'm trying to insert data from a csv file into elasticsearch using logstash. I have tried adding

  date {
                match => ["SIGHTING_TIME", "dd-MM-YYYY HH:mm:ss"]
                target =>"SIGHTING_TIME"
}

in the config file. But i find the mapping of the field "SIGHTING_TIME" as text only . I want it to be changed to date format to perform some search queries on the date range. Any help would be appreciable.

#2

I'm guessing your index already had a mapping for SIGHTING_TIME that was generated as text the first time you ingested the data. Adding this to your Logstash config can't change the mapping.

If you want the mapping to be a date then you will need to change your mapping of that field to be a date in elasticsearch.

#3

May I know how can I change the mapping in elasticsearch? and also i have inserted the data into the index and is updating the mapping possible? May i have some clear idea on that.

#4

You can't really update it, in this case.

If your data is disposable then the easiest way is to delete your index. Create your mapping. Elastic will automap based on best guess if you ingest data without a mapping. You don't need to put all your fields in your mapping, just the ones that didn't automap correctly. Then re-ingest your data.

If you can't just delete your data then most people use the reindex API to copy your data to a temporary index, change mapping on old, then reindex back to old index.

#5

I have tried creating mapping for a new index with the required date format in elasticsearch and then tried running the logstash config file to insert the csv data. Then i find the following warning and also found that no data was inserted into the index.

[WARN ][logstash.outputs.elasticsearch][main][65f4fc0a575c13127baac0a4b09f8dc1296cfc01417be78f54c2154302f77b55] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mtd_checkdate1", :routing=>nil, :_type=>"security_events"}, #<LogStash::Event:0xfbcc1a9>], :response=>{"index"=>{"_index"=>"mtd_checkdate1", "_type"=>"security_events", "_id"=>"HFsdaXgB2WWJNcKRoLkW", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [SIGHTING_TIME] cannot be changed from type [date] to [text]"}}}}

May i also know if iam proceeding in the correct way

#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.