# Error parsing csv invalid file format

**URL:** <https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147>\
**Category:** Logstash\
**Created:** [May 16, 2018, 2:16pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147 "2018-05-16T14:16:53Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 16, 2018, 2:16pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/1 "2018-05-16T14:16:54Z")

</div>

hello all below is the log format and I was using ; as separator and trying to using the following logstash configuration and I am getting parsing error

Kindly suggest what can be done thanks in advance

csv {  
separator =\> ";"  
columns =\> ["Date","transactionID","TransactionType","Address","EndPointName","Status","FailureStatusCode","TransactionDuration","x-device-auth-mode","x-user-auth-mode","x-device-type","x-nsds-version","timestamp","conditiontype","reason","event-body"]  
}

2018-05-15 03:00:00,321 ;2607:fb90:a2e9:8275:6cf:8670:2dbe:81a3:VG1:1526378339989:MnBgMn;2;OUT:[https://dashapi.t-mobile.com/dash/eventnotification;DASH;1;500;60110;IP;null;Handheld](https://dashapi.t-mobile.com/dash/eventnotification;DASH;1;500;60110;IP;null;Handheld) with SIM;2.0;1526378340;device;activated;{"timeStamp":1526378340,"condition-type":"device","reason":"activated","event-body":{"devices-info":[{"device-id":"urn:gsma:imei:35826907-082230-0","imsi":"310260626967007","msisdn":"12246196061","user-ids":[]}]}}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 16, 2018, 7:44pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/2 "2018-05-16T19:44:54Z")

</div>

Is there anything in the Logstash log that might give clues about what's wrong?

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 16, 2018, 7:57pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/3 "2018-05-16T19:57:21Z")

</div>

@magnusbaeck thanks for getting back to me here is the error

message", :source=\>"activated;{"timeStamp":1526378340,"condition-type":"device","reason":"activated","event-body":{"devices-info":[{"device-id":"urn:gsma:imei:35826907-082230-0","imsi":"310260626967007","msisdn":"12246196061","user-ids":[]}]}}", :exception=\>#\<CSV::MalformedCSVError: Illegal quoting in line 1

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 16, 2018, 8:13pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/4 "2018-05-16T20:13:17Z")

</div>

Aha. It probably doesn't like the double quotes in the last column. Use a dissect or grok filter for the parsing.

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 16, 2018, 8:15pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/5 "2018-05-16T20:15:46Z")

</div>

Thanks Magnus one quick question when using grok can i map timestamp to DATA filed??

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 16, 2018, 9:10pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/6 "2018-05-16T21:10:35Z")

</div>

Hi Magnus,

Tried using the following for grok filter for the same input still no luck

```
    match => { "message" => "%{DATA:timestamp1} %{DATA:TID} %{DATA:Ttype} %{DATA:Add} %{DATA:stat} %{DATA:FailedScode} %{DATA:deviceautmode} %{DATA:userauthmode} %{DATA:devicetype} %{DATA:nsdsvers} %{DATA:timest1} %{DATA:conditionty} %{GREEDYDATA:eventbod}" }

```

Thanks,  
Sandeep

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 17, 2018, 6:01am UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/7 "2018-05-17T06:01:21Z")

</div>

Never use more than one DATA or GREEDYDATA pattern in the same grok expression. Use something more exact. For example, `[^;]+` matches zero or more characters of any kind except semicolon. That can be used to parse a string containing semicolon-delimited values. Secondly, you have spaces between the tokens you want to match. This is obviously incorrect since the columns are semicolon-delimited.

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 21, 2018, 4:31pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/8 "2018-05-21T16:31:55Z")

</div>

Thanks @magnusbaeck

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 21, 2018, 6:56pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/9 "2018-05-21T18:56:03Z")

</div>

input {  
file {  
path =\>"/opt/LogStashOutputFormatted/ENS/\*"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
type =\> "esn"  
}  
}  
filter {  
if [type] == "esn" {  
dissect {  
mapping =\> {  
"message" =\> "%{stat};%{{messagebody}}  
}  
}  
}  
}  
output {  
if [type] == "esn" {  
elasticsearch {  
action =\> "index"  
hosts =\> "[http://localhost:9200](http://localhost:9200)"  
index =\> "esn"  
}  
}  
}

sample input:  
activated;{"timeStamp":1526378340,"condition-type":"device","reason":"activated","event-body":{"devices-info":[{"device-id":"urn:gsma:imei:35826907-082230-0","imsi":"310260626967007","msisdn":"12246196061","user-ids":[]}]}}"event-body":{"devices-info":[{"device-id":"urn:gsma:imei:35282906-796690-0","imsi":"310260142105333","msisdn":"17274750820","user-ids":[]}]}}

error: The following is the error when using the following  
[ERROR] 2018-05-21 11:49:45.195 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, {, }

I tried debugging while commenting the lines the error is random

I want data between first which is ; separated and next was between {} and i am unable extract the data

Thanks in advance for the support

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 21, 2018, 7:11pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/10 "2018-05-21T19:11:49Z")

</div>

> [@sandeepnarla22322](#):
>
> "message" =\> "%{stat};%{{messagebody}}

You are missing a closing double quote. And you might mean {%{ rather than %{{, although I do not think either will actually do what you want.

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 21, 2018, 7:29pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/11 "2018-05-21T19:29:00Z")

</div>

Thanks Badger I changed the following config to this after your suggestion

"message" =\> "%{stat};{%{messagebody}}"

I don't see any error but index is not getting created in kibana

I all want is to get the entire message

I want filter the data after ; and assign to messagebody

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 21, 2018, 7:33pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/12 "2018-05-21T19:33:23Z")

</div>

282123K-\>2101K(314560K), 0.0032156 secs] 777032K-\>497010K(1013632K), 0.0033993 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]  
2018-05-21T12:21:05.488-0700: 2419872.268: Total time for which application threads were stopped: 0.0050471 seconds, Stopping threads took: 0.0001995 seconds  
2018-05-21T12:25:58.458-0700: 2420165.238: [GC (Allocation Failure) 2018-05-21T12:25:58.458-0700: 2420165.238: [ParNew  
Desired survivor size 17891328 bytes, new threshold 6 (max 6)

- age 1: 390272 bytes, 390272 total

I do see this error in elastic search

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 21, 2018, 8:09pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/13 "2018-05-21T20:09:51Z")

</div>

> [@sandeepnarla22322](#):
>
> I do see this error in Elasticsearch

That is not an error, it is just very verbose garbage collection logging.

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 21, 2018, 8:17pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/14 "2018-05-21T20:17:28Z")

</div>

Badger can you provide any inputs on how to filter the following i have tried everything I can

activated;**{"timeStamp":1526378340,"condition-type":"device","reason":"activated","event-body":{"devices-info":[{"device-id":"urn:gsma:imei:35826907-082230-0","imsi":"310260626967007","msisdn":"12246196061","user-ids":[]}]}}"event-body":{"devices-info":[{"device-id":"urn:gsma:imei:35282906-796690-0","imsi":"310260142105333","msisdn":"17274750820","user-ids":[]}]}}**

i want to filter activated =\> status  
rest to message body =\> messagebody

thanks,  
again

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 21, 2018, 9:51pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/15 "2018-05-21T21:51:20Z")

</div>

You should get that from

```
dissect { mapping => { "message" => "%{stat};%{messagebody}" } }
```

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [May 22, 2018, 9:09pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/16 "2018-05-22T21:09:13Z")

</div>

thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 19, 2018, 9:09pm UTC](https://discuss.elastic.co/t/error-parsing-csv-invalid-file-format/132147/17 "2018-06-19T21:09:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.