# Pb with format during a CSV import
**URL:** https://discuss.elastic.co/t/pb-with-format-during-a-csv-import/323267
**Category:** Kibana
**Created:** [January 16, 2023, 3:44pm UTC](https://discuss.elastic.co/t/pb-with-format-during-a-csv-import/323267 "2023-01-16T15:44:56Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Phildefer](https://discuss.elastic.co/u/Phildefer)
#### Post date: [January 16, 2023, 3:44pm UTC](https://discuss.elastic.co/t/pb-with-format-during-a-csv-import/323267/1 "2023-01-16T15:44:56Z")
Hi all,
Sorry in advance for my bad english and my poor knowledge on ELK.
I need to import regularly a csv file like this :
| data\_id | iso | event\_id\_cnty | event\_id\_no\_cnty | event\_date | year | time\_precision | event\_type | sub\_event\_type | actor1 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 9722610 | 887 | YEM78146 | 78146 | 6 January 2023 | 2023 | 1 | Explosions/Remote violence | Remote explosive/landmine/IED | AQAP: Al Qaeda in the Arabian Peninsula |
| 9722612 | 887 | YEM78148 | 78148 | 6 January 2023 | 2023 | 1 | Explosions/Remote violence | Remote explosive/landmine/IED | National Resistance Forces |
| 9722613 | 887 | YEM78149 | 78149 | 6 January 2023 | 2023 | 1 | Battles | Armed clash | Military Forces of Yemen (2016-) Supreme Political Council |
| 9722615 | 887 | YEM78151 | 78151 | 6 January 2023 | 2023 | 1 | Battles | Armed clash | Military Forces of Yemen (2016-) Supreme Political Council |
| .... | | | | | | | | | |
My first problem is the format of the date field "event\_date". As you can see, the date is like "06 january 2023" but when I want to import it as a date field with Kibana (csv upload), I have an error during the upload process. Kibana cannot parse this field when I put the date type instead of a Keyword type.
My second problem is that I would like to find a way to apply (when I find the solution for the date field) a template to this file each time I want to import it with Kibana. I don't know if (and how) it's possible to apply a template (with the right types) to the upload csv process without changing manually the type of the date field.
UPDATE : I have found the problem with the date format, so now I have got a ingest pipeline in Kibana which can transform correctly the event\_date field, but I have a field (actor1) which is recognized as a text field and I would like to store it in my index as a keyword field. Is it possible to do that in the ingest pipeline ?
Thx for your help.
---
### Author:  [@jsanz](https://discuss.elastic.co/u/jsanz)
#### Post date: [January 18, 2023, 5:25pm UTC](https://discuss.elastic.co/t/pb-with-format-during-a-csv-import/323267/2 "2023-01-18T17:25:49Z")
In the File Upload interface you can set up both the Ingest Pipeline AND the mappings for your new index so you can define the data types there
I took your table and added the following mappings and pipeline:
```auto
{
"properties": {
"actor1": {
"type": "keyword"
},
"data_id": {
"type": "long"
},
"event_date": {
"type": "date"
},
"event_id_cnty": {
"type": "keyword"
},
"event_id_no_cnty": {
"type": "long"
},
"event_type": {
"type": "keyword"
},
"iso": {
"type": "long"
},
"sub_event_type": {
"type": "keyword"
},
"time_precision": {
"type": "long"
},
"year": {
"type": "long"
}
}
}
{
"description": "Ingest pipeline created by text structure finder",
"processors": [
{
"csv": {
"field": "message",
"target_fields": [
"data_id",
"iso",
"event_id_cnty",
"event_id_no_cnty",
"event_date",
"year",
"time_precision",
"event_type",
"sub_event_type",
"actor1"
],
"ignore_missing": false
}
},
{
"convert": {
"field": "data_id",
"type": "long",
"ignore_missing": true
}
},
{
"convert": {
"field": "event_id_no_cnty",
"type": "long",
"ignore_missing": true
}
},
{
"convert": {
"field": "iso",
"type": "long",
"ignore_missing": true
}
},
{
"convert": {
"field": "time_precision",
"type": "long",
"ignore_missing": true
}
},
{
"convert": {
"field": "year",
"type": "long",
"ignore_missing": true
}
},{
"date": {
"field": "event_date",
"formats": ["d MMMM yyyy"],
"timezone" : "Europe/Amsterdam",
"target_field": "event_date"
}
},
{
"remove": {
"field": "message"
}
}
]
}
```
See the types for `actor1` and `event_date` and the processor for the date.
And the discover screen after import:
Some notes about this (for you and whoever comes here in the future):
- [This is a nice tool to check for Java date format](https://javadevtools.com/datetimeformatter)
- You can [simulate the pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html) to test things isolated. Kibana has also an interface to test things but maybe with DevTools you can test smaller things and then go to the UI for more advanced cases. Example I used to test the date parsing:
```auto
POST /_ingest/pipeline/_simulate
{
"pipeline" :
{
"description": "_description",
"processors": [
{
"date": {
"field": "event_date",
"formats": ["d MMMM yyyy"],
"timezone" : "Europe/Amsterdam",
"target_field": "event_date"
}
}
]
},
"docs": [
{
"_index": "index",
"_id": "id",
"_source": {
"event_date": "6 January 2023"
}
}
]
}
```
- Finally, if you want to run this repeatedly maybe it is more convenient to you to store the pipeline, and put the mapping inside an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) and configure [filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) to read your CSVs.
---
### Author:  [@Phildefer](https://discuss.elastic.co/u/Phildefer)
#### Post date: [January 19, 2023, 1:33am UTC](https://discuss.elastic.co/t/pb-with-format-during-a-csv-import/323267/3 "2023-01-19T01:33:58Z")
Hi Jorge,
Thx for your answer.
I have created all files (Pipeline, template) and as you mentioned it, I need to find a way to apply automatically and repeatedly these files to my next uploads. If I understand well, it's not possible to automatically apply a pipeline and a template with the upload menu of Kibana so I'm going to try your solution with Filebeat.
Thx so much.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 16, 2023, 1:34am UTC](https://discuss.elastic.co/t/pb-with-format-during-a-csv-import/323267/4 "2023-02-16T01:34:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.