# Convert a date format in logstash pipeline

**URL:** <https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000>\
**Category:** Logstash\
**Created:** [July 28, 2017, 6:28pm UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000 "2017-07-28T18:28:36Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 28, 2017, 6:28pm UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/1 "2017-07-28T18:28:36Z")

</div>

hello , i am trying to load a csv file into elasticsearch via logstash .this is a snippet of my data sets

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/4/c455fe4bf4be40db1656aca22499210526ac33fe.png)transaction\_date,customer\_name,product\_name,product\_id,price\_unit,quantity,fidelity\_card\_id,discount,category  
2-24-2017,Roberto Michel,Taillefine aux fruits 0% Fraise - Danone - 500 g (4 x 125Â g),1,556.26,256,4.175E+15,9,yaourt  
5-15-2016,Reiko Branchet,"Activia Fibre - Danone - 171,5 g (150 g+21,5Â g)",2,612.98,163,3.74284E+14,8,yaourt

i used in the configuration of the pipeline mutate and convert to change string fields into numeric ones because elasticsearch store fields by default as strings but the problem appeared when i used this

date {  
match =\> {transaction\_date , "MM dd yyyy"} }

in order to convert this field from string into date format  
this is th error i 've got when running logsatsh

Cannot create pipeline {:reason=\>"Expected one of #, =\> at line 26, column 29 (byte 713) after filter {\n csv {\n separator =\> ","\n\t#transaction\_date,customer\_name,product\_name,product\_id,price\_unit,quantity,fidelity\_card\_id,discount\n columns =\> ["transaction\_date","customer\_name","product\_name","product\_id","price\_unit","quantity","fidelity\_card\_id","discount"]\n} \nmutate\t{\t\t\nconvert =\>{"discount" =\> "integer"\t}\t\t\t\t\t\nconvert =\>{"price\_unit" =\>"float"}\t\t\t\t\t\t\t\nconvert =\>{"product\_id" =\>"integer"}\t\t\t\t\t\t\t\nconvert =\>{"quantity"=\> "integer"}\t\t\t\t\t\n\n}\n\ndate {\n\tmatch =\> {transaction\_date "}  
2017-07-28 19:26:49,905 Api Webserver ERROR No log4j2 configuration file found. Using default configuration: logging only errors to the console.

this my config file (the filter part)

filter {  
csv {  
separator =\> ","  
#transaction\_date,customer\_name,product\_name,product\_id,price\_unit,quantity,fidelity\_card\_id,discount  
columns =\> ["transaction\_date","customer\_name","product\_name","product\_id","price\_unit","quantity","fidelity\_card\_id","discount"]  
}  
mutate {  
convert =\>{"discount" =\> "integer" }  
convert =\>{"price\_unit" =\>"float"}  
convert =\>{"product\_id" =\>"integer"}  
convert =\>{"quantity"=\> "integer"}

}

date {  
match =\> {transaction\_date ,"MM dd yyyy"}}  
}

please any help !!! how could i change the type of this field ??

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 29, 2017, 7:53am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/2 "2017-07-29T07:53:57Z")

</div>

any help !!!! is there anyone who had faced the same problem and resolved it ???

---

<div class="post-metadata">

**Author:** ![josephjohney](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@josephjohney](https://discuss.elastic.co/u/josephjohney)\
**Post date:** [July 29, 2017, 7:56am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/3 "2017-07-29T07:56:35Z")

</div>

The date format has to allogn with the date string you're getting from the filter

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 29, 2017, 8:05am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/4 "2017-07-29T08:05:52Z")

</div>

so what am i supposed to do ???

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 29, 2017, 8:33am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/5 "2017-07-29T08:33:40Z")

</div>

Any help please need to modify this field into date i need it in the creation of the dashboard later on !!!

---

<div class="post-metadata">

**Author:** ![josephjohney](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@josephjohney](https://discuss.elastic.co/u/josephjohney)\
**Post date:** [July 29, 2017, 8:36am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/6 "2017-07-29T08:36:01Z")

</div>

Please refer date filter plugin...  
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html)

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 29, 2017, 8:45am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/7 "2017-07-29T08:45:50Z")

</div>

i did but found nothing to solve my promblem .i don't get why the date filter i used does not effect the type of the field stored at elasticsearch ??? other fields has changed but only the date field doesn't change. when running logstash no error appears ..

---

<div class="post-metadata">

**Author:** ![josephjohney](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@josephjohney](https://discuss.elastic.co/u/josephjohney)\
**Post date:** [July 29, 2017, 9:53am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/8 "2017-07-29T09:53:36Z")

</div>

Please recreate the existing index

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 29, 2017, 10:02am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/9 "2017-07-29T10:02:43Z")

</div>

i deleted the index and recreate it but nothing changed

---

<div class="post-metadata">

**Author:** ![josephjohney](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@josephjohney](https://discuss.elastic.co/u/josephjohney)\
**Post date:** [July 29, 2017, 10:15am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/10 "2017-07-29T10:15:39Z")

</div>

Can you check the mapping of the index

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 29, 2017, 10:19am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/11 "2017-07-29T10:19:57Z")

</div>

this the transactin\_date part of the result :

```
    "transaction_date": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}, 

```

it still with a string format

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 29, 2017, 10:31am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/12 "2017-07-29T10:31:01Z")

</div>

can i create a template where i define my mapping then use it in the pipeline configuration ???

---

<div class="post-metadata">

**Author:** ![josephjohney](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@josephjohney](https://discuss.elastic.co/u/josephjohney)\
**Post date:** [July 29, 2017, 1:35pm UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/13 "2017-07-29T13:35:23Z")

</div>

You have to delete your current index or create another index and try to insert the data again

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 29, 2017, 1:39pm UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/14 "2017-07-29T13:39:40Z")

</div>

i did create an otherr index but also nothing works

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [July 29, 2017, 1:53pm UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/15 "2017-07-29T13:53:46Z")

</div>

do you have any example of a config file of logstash where u did use date filter and it worked ??? if u do can u send me the example ????

---

<div class="post-metadata">

**Author:** ![josephjohney](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@josephjohney](https://discuss.elastic.co/u/josephjohney)\
**Post date:** [August 1, 2017, 5:25am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/16 "2017-08-01T05:25:43Z")

</div>

The below config worked for me ..

date {  
match =\> ["timestamp\_string" , "dd.MM.yyyy HH:mm:ss.SSS"]  
target =\> "new\_timestamp"  
}

---

<div class="post-metadata">

**Author:** ![marwa](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@marwa](https://discuss.elastic.co/u/marwa)\
**Post date:** [August 1, 2017, 8:22am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/17 "2017-08-01T08:22:31Z")

</div>

thank you for your reply ,  
i did it and it worked

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 29, 2017, 8:22am UTC](https://discuss.elastic.co/t/convert-a-date-format-in-logstash-pipeline/95000/18 "2017-08-29T08:22:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.