Convert a date format in logstash pipeline

Elastic Stack Logstash
1

hello , i am trying to load a csv file into elasticsearch via logstash .this is a snippet of my data sets

image
image.png801×61 4.34 KB
transaction_date,customer_name,product_name,product_id,price_unit,quantity,fidelity_card_id,discount,category
2-24-2017,Roberto Michel,Taillefine aux fruits 0% Fraise - Danone - 500 g (4 x 125Â g),1,556.26,256,4.175E+15,9,yaourt
5-15-2016,Reiko Branchet,"Activia Fibre - Danone - 171,5 g (150 g+21,5Â g)",2,612.98,163,3.74284E+14,8,yaourt

i used in the configuration of the pipeline mutate and convert to change string fields into numeric ones because elasticsearch store fields by default as strings but the problem appeared when i used this

date {
match => {transaction_date , "MM dd yyyy"} }

in order to convert this field from string into date format
this is th error i 've got when running logsatsh

Cannot create pipeline {:reason=>"Expected one of #, => at line 26, column 29 (byte 713) after filter {\n csv {\n separator => ","\n\t#transaction_date,customer_name,product_name,product_id,price_unit,quantity,fidelity_card_id,discount\n columns => ["transaction_date","customer_name","product_name","product_id","price_unit","quantity","fidelity_card_id","discount"]\n} \nmutate\t{\t\t\nconvert =>{"discount" => "integer"\t}\t\t\t\t\t\nconvert =>{"price_unit" =>"float"}\t\t\t\t\t\t\t\nconvert =>{"product_id" =>"integer"}\t\t\t\t\t\t\t\nconvert =>{"quantity"=> "integer"}\t\t\t\t\t\n\n}\n\ndate {\n\tmatch => {transaction_date "}
2017-07-28 19:26:49,905 Api Webserver ERROR No log4j2 configuration file found. Using default configuration: logging only errors to the console.

this my config file (the filter part)

filter {
csv {
separator => ","
#transaction_date,customer_name,product_name,product_id,price_unit,quantity,fidelity_card_id,discount
columns => ["transaction_date","customer_name","product_name","product_id","price_unit","quantity","fidelity_card_id","discount"]
}
mutate {
convert =>{"discount" => "integer" }
convert =>{"price_unit" =>"float"}
convert =>{"product_id" =>"integer"}
convert =>{"quantity"=> "integer"}

}

date {
match => {transaction_date ,"MM dd yyyy"}}
}

please any help !!! how could i change the type of this field ??

2

any help !!!! is there anyone who had faced the same problem and resolved it ???

3

The date format has to allogn with the date string you're getting from the filter

4

so what am i supposed to do ???

5

Any help please need to modify this field into date i need it in the creation of the dashboard later on !!!

6

Please refer date filter plugin...
https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

7

i did but found nothing to solve my promblem .i don't get why the date filter i used does not effect the type of the field stored at elasticsearch ??? other fields has changed but only the date field doesn't change. when running logstash no error appears ..

8

Please recreate the existing index

9

i deleted the index and recreate it but nothing changed

10

Can you check the mapping of the index

11

this the transactin_date part of the result :

    "transaction_date": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
},

it still with a string format

12

can i create a template where i define my mapping then use it in the pipeline configuration ???

13

You have to delete your current index or create another index and try to insert the data again

14

i did create an otherr index but also nothing works

15

do you have any example of a config file of logstash where u did use date filter and it worked ??? if u do can u send me the example ????

16

The below config worked for me ..

date {
match => [ "timestamp_string" , "dd.MM.yyyy HH:mm:ss.SSS" ]
target => "new_timestamp"
}

1 Like
17

thank you for your reply ,
i did it and it worked

1 Like
18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.