# CSV Timstamp issues

**URL:** <https://discuss.elastic.co/t/csv-timstamp-issues/43050>\
**Category:** Logstash\
**Created:** [February 29, 2016, 8:50pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050 "2016-02-29T20:50:22Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [February 29, 2016, 8:50pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/1 "2016-02-29T20:50:22Z")

</div>

Here is my conf file, and it seems to be working ok, except I need the timestamp to be the actually time in the csv file. Not the time the file was added. What am i doing wrong.... total Noob here. Sorry.

input {  
file {  
path =\> "/home/bkelley6/flights/\*.csv"  
type =\> "flights"  
start\_position =\> "beginning"  
}  
}

filter {  
csv {  
columns =\> ["Date", "Time", "SWR", "RSSI(dB)", "RxBt(V)", "Cels(gRe)", "Tmp2(@C)", "RPM(rpm)", "Tmp1(@C)", "Rud", "Ele", "Thr", "Ail", "S1", "S2", "S3", "LS", "RS", "SA", "SB", "SC", "SD" ,"SE", "SF", "SG", "SH"]  
separator =\> ","  
}

mutate {  
replace =\> ["date", "%{Date} %{Time}"]  
}  
}  
output {  
elasticsearch {  
action =\> "index"  
hosts =\> ["localhost:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
workers =\> 1  
}  
}

File sample

Date,Time,SWR,RSSI(dB),RxBt(V),Cels(gRe),Tmp2(@C),RPM(rpm),Tmp1(@C),Rud,Ele,Thr,Ail,S1,S2,S3,LS,RS,SA,SB,SC,SD,SE,SF,SG,SH,  
2016-02-21,04:11:14.640,30,75,5.2,23.4,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [February 29, 2016, 8:52pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/2 "2016-02-29T20:52:15Z")

</div>

Here is what I am getting.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/d/d3d500f55470181630025af014ecbee7cb7aefde.png)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 29, 2016, 9:39pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/3 "2016-02-29T21:39:10Z")

</div>

You then need to do something like;

```auto
  date {
    match => ["date", "dd/MM/YYYY hh:mm:ss a"]
    timezone => "YOUR TZ HERE"
    target => "date"
  }

```

That'll properly match the value. If you want to replace the `@tiemstamp` then just change the `target` value, then you may want to drop the original `date` field

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [February 29, 2016, 11:00pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/4 "2016-02-29T23:00:46Z")

</div>

So I would put that after the mutate, or in place of it?

Thanks,  
Brad

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 29, 2016, 11:01pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/5 "2016-02-29T23:01:59Z")

</div>

After the mutate, sorry!

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [February 29, 2016, 11:02pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/6 "2016-02-29T23:02:48Z")

</div>

Thank you,

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [February 29, 2016, 11:13pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/7 "2016-02-29T23:13:38Z")

</div>

input {  
file {  
path =\> "/home/bkelley6/flights/\*.csv"  
type =\> "flights"  
start\_position =\> "beginning"  
}  
}

filter {  
csv {  
columns =\> ["Date", "Time", "SWR", "RSSI(dB)", "RxBt(V)", "Cels(gRe)", "Tmp2(@C)", "RPM(rpm)", "Tmp1(@C)", "Rud", "Ele", "Thr", "Ail", "S1", "S2", "S3", "LS", "RS", "SA", "SB", "SC", "SD" ,"SE", "SF", "SG", "SH"]  
separator =\> ","  
}

mutate {  
replace =\> ["date", "%{Date} %{Time}"]  
}  
}  
date {  
match =\> ["date", "dd/MM/YYYY hh:mm:ss a"]  
timezone =\> "America/New\_York"  
target =\> "@tiemstamp"  
}

output {  
elasticsearch {  
action =\> "index"  
hosts =\> ["localhost:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
workers =\> 1  
}  
}

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [February 29, 2016, 11:14pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/8 "2016-02-29T23:14:27Z")

</div>

misspelled timestamp

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [February 29, 2016, 11:22pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/9 "2016-02-29T23:22:57Z")

</div>

Failed parsing date from field {:field=\>"date", :value=\>"2016-02-21 17:17:57.440", :exception=\>"Invalid format: "2016-02-21 17:17:57.440" is malformed at "16-02-21 17:17:57.440"", :config\_parsers=\>"dd/MM/ YYYY hh:mm:ss a", :config\_locale=\>"default=en\_US", :level=\>:warn}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 29, 2016, 11:29pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/10 "2016-02-29T23:29:47Z")

</div>

Oh sorry, that was a time format I was using for something else.

Change the match pattern to `ISO8601`.

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [February 29, 2016, 11:34pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/11 "2016-02-29T23:34:28Z")

</div>

Here is what I currently have, still not taking the date format.

input {  
file {  
path =\> "/home/bkelley6/flights/\*.csv"  
type =\> "flights"  
start\_position =\> "beginning"  
}  
}

filter {  
csv {  
columns =\> ["Date", "Time", "SWR", "RSSI(dB)", "RxBt(V)", "Cels(gRe)", "Tmp2(@C)", "RPM(rpm)", "Tmp1(@C)", "Rud", "Ele", "Thr", "Ail", "S1", "S2", "S3", "LS", "RS", "SA", "SB", "SC", "SD" ,"SE", "SF", "SG", "SH"]  
separator =\> ","  
}

mutate {  
replace =\> ["date", "%{Date} %{Time}"]  
}

date {  
match =\> ["date", "YYYY-MM-dd;HH:mm:ss.SSS", "ISO8601"]  
timezone =\> "America/New\_York"  
target =\> "@timestamp"  
}  
}  
output {  
elasticsearch {  
action =\> "index"  
hosts =\> ["localhost:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
workers =\> 1  
}  
}

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [March 1, 2016, 12:59am UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/12 "2016-03-01T00:59:07Z")

</div>

Still not parsing. I am at a loss.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 1, 2016, 1:15am UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/13 "2016-03-01T01:15:39Z")

</div>

What's the `date` field actually look like in the json?

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [March 1, 2016, 1:18am UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/14 "2016-03-01T01:18:03Z")

</div>

Here are the first two lines of the csv file I am trying to bring in, the first two records are the date and time.  
comma delimited.

Date,Time,SWR,RSSI(dB),RxBt(V),Cels(gRe),Tmp2(@C),RPM(rpm),Tmp1(@C),Rud,Ele,Thr,Ail,S1,S2,S3,LS,RS,SA,SB,SC,SD,SE,SF,SG,SH,  
2016-02-21,04:11:14.640,30,75,5.2,23.4,0,0,0,0,0,0,0,0,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [March 1, 2016, 1:23am UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/15 "2016-03-01T01:23:33Z")

</div>

This is what i have now.

input {  
file {  
path =\> "/home/bkelley6/flights/\*.csv"  
type =\> "flights"  
start\_position =\> "beginning"  
}  
}

filter {  
csv {  
columns =\> ["Date", "Time", "SWR", "RSSI(dB)", "RxBt(V)", "Cels(gRe)", "Tmp2(@C)", "RPM(rpm)", "Tmp1(@C)", "Rud", "Ele", "Thr", "Ail", "S1", "S2", "S3", "LS", "RS", "SA", "SB", "SC", "SD" ,"SE", "SF", "SG", "SH"]  
separator =\> ","  
}

mutate {  
replace =\> ["date", "%{Date} %{Time}"]  
}

date {  
"locale" =\> "en"  
match =\> ["date", "YYYY-MM-dd hh:mm:ss.SSS"]  
timezone =\> "America/New\_York"  
target =\> "@timestamp"  
}  
}  
output {  
elasticsearch {  
action =\> "index"  
hosts =\> ["localhost:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
workers =\> 1  
}  
}

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [March 2, 2016, 5:49pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/16 "2016-03-02T17:49:49Z")

</div>

Anyone?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 2, 2016, 9:53pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/17 "2016-03-02T21:53:28Z")

</div>

I think you need to start at the basics, with the CSV filter, and go from there. I did that and I can see a `_csvparsefailure` tag.

---

<div class="post-metadata">

**Author:** ![hursto75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hursto75/32/8168_2.png) [@hursto75](https://discuss.elastic.co/u/hursto75)\
**Post date:** [March 3, 2016, 2:57pm UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/18 "2016-03-03T14:57:27Z")

</div>

Little more information and some big hints to the problem.

I wrote a little different config file (notice the input method):  
filter {  
csv {  
columns =\> ["Date", "Time", "SWR", "RSSI(dB)", "RxBt(V)", "Cels(gRe)", "Tmp2(@C)", "RPM(rpm)", "Tmp1(@C)", "Rud", "Ele", "Thr", "Ail", "S1", "S2", "S3", "LS", "RS", "SA", "SB", "SC", "SD" ,"SE", "SF", "SG", "SH"]  
separator =\> ","  
}

mutate {  
replace =\> ["mydatetime", "%{Date}T%{Time}"]  
}

date {  
locale =\> "en"  
match =\> ["mydatetime", "yyyy-MM-dd'T'HH:mm:ss.SSS"]  
timezone =\> "America/New\_York"  
target =\> ["@timestamp"]  
remove\_field =\> ["mydatetime"]  
}  
}

input { stdin {} }  
output {  
elasticsearch {  
action =\> "index"  
hosts =\> ["localhost:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
workers =\> 1  
}  
}

And wrote this one liner to process one line at a time.

cat /home/bkelley6/flights/T-Rex\_500-2016-02-21-1-test.csv | while read line ; do echo $line | /opt/logstash/bin/logstash -f magnus-elastic.conf; done

This works!

So I am thinking the problem is with the input on the the other conf file:

input {  
file {  
path =\> "/home/bkelley6/flights/\*.csv"  
type =\> "flights"  
start\_position =\> "beginning"  
}  
}

Why would this be causing a problem?

Once again thank you for all your help.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 4, 2016, 12:06am UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/19 "2016-03-04T00:06:53Z")

</div>

Maybe there is some weird formatting in the file that isn't immediately obvious?

---

<div class="post-metadata">

**Author:** ![bryan\_stuhlsatz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bryan_stuhlsatz/32/49123_2.png) [@bryan\_stuhlsatz](https://discuss.elastic.co/u/bryan_stuhlsatz)\
**Post date:** [March 6, 2016, 6:55am UTC](https://discuss.elastic.co/t/csv-timstamp-issues/43050/20 "2016-03-06T06:55:14Z")

</div>

Verify that your csv file has a CRLF at end of each line. You could open in Notepad ++ to validate.  
So, nothing shows up in Elastic now, where earlier in your first post you were getting data in ES?

[Next page](https://discuss.elastic.co/t/csv-timstamp-issues/43050.md?page=2)