# Logstash CSV Kibana input error **URL:** \ **Category:** Logstash\ **Created:** [January 29, 2020, 7:25am UTC](https://discuss.elastic.co/t/logstash-csv-kibana-input-error/216948 "2020-01-29T07:25:29Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![ELK4Life](https://avatars.discourse-cdn.com/v4/letter/e/f4b2a3/32.png) [@ELK4Life](https://discuss.elastic.co/u/ELK4Life)\ **Post date:** [January 29, 2020, 7:25am UTC](https://discuss.elastic.co/t/logstash-csv-kibana-input-error/216948/1 "2020-01-29T07:25:29Z")
Dear Logstash community, I have challenged myself to capture my network data using TShark and make custom dashboards in Kibana. I have the following setup: - 1 Ubuntu 18.04 VM with ElasticSearch, Logstash and Kibana dockerized - 1 Ubuntu 18.04 VM with TShark and Filebeat running on the host I run TShark using the specified `-T ek` flag and export the capture to a rolling CSV file. Using Filebeat I send the CSV file to Logstash. Logstash will then send the file to Elastic Search which will be queried by Kibana. **My problem is: Kibana shows the Logstash logs as it's main input and per event includes one whole CSV capture line in a single field. How do I solve this that the CSV becomes the main input?** ![Kibana view](https://us1.discourse-cdn.com/elastic/original/3X/b/5/b56718828fd715f8e24dfce1dd45cecff1f6e2b8.png) This is my logstash.conf: ```auto input { beats { port => 5044 } } filter { csv { source => "message" columns => ["col.Time", "col.Source", "col.Destination", "ip.src", "ip.dst", "tcp.srcport", "tcp.dstport", "col.Protocol", "ip.len", "col.Info"] } mutate { convert => ["ip-len", "integer"] } date { match => ["col.time", "YYYY-MM-DD HH:mm:ss.SSSSSSSSS"] target => "@timestamp" } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" user => elastic password => changeme } } ``` This is my filebeat.yml: ```auto filebeat.modules: - module: system syslog: enabled: false auth: enabled: true var.paths: ["/home/user/Documents/tsharkcap/tshark.csv"] output.logstash: hosts: ["192.168.234.134:5044"] ``` I run ElasticSearch with default config: ```auto sudo docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.5.2 ``` I run Kibana with default config: ```auto sudo docker run --link docker-cont:elasticsearch -p5601:5601 docker.elastic.co/kibana/kibana:7.5.2 ``` Thanks in advance, ELK4Life ---
**Author:** ![ELK4Life](https://avatars.discourse-cdn.com/v4/letter/e/f4b2a3/32.png) [@ELK4Life](https://discuss.elastic.co/u/ELK4Life)\ **Post date:** [January 29, 2020, 7:30am UTC](https://discuss.elastic.co/t/logstash-csv-kibana-input-error/216948/2 "2020-01-29T07:30:26Z")
BTW, This is the Mapping as shown in Kibana, which is I believe auto generated?! ```auto { "mapping": { "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "agent": { "properties": { "ephemeral_id": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "hostname": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "id": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "ecs": { "properties": { "version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "event": { "properties": { "dataset": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "module": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "timezone": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "fileset": { "properties": { "name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "host": { "properties": { "name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "input": { "properties": { "type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "log": { "properties": { "file": { "properties": { "path": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "offset": { "type": "long" } } }, "message": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "service": { "properties": { "type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "tags": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } } ``` ---
**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Post date:** [January 29, 2020, 10:26am UTC](https://discuss.elastic.co/t/logstash-csv-kibana-input-error/216948/3 "2020-01-29T10:26:55Z")
You need to enable multiline configuration in your filebeat.yml Example: multiline.pattern: ',\d+,[^",]+$' multiline.negate: true multiline.match: before See this link for multiline configuration =\> [https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html) You can also check your pattern here =\> [https://play.golang.org/p/uAd5XHxscu](https://play.golang.org/p/uAd5XHxscu) ---
**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Post date:** [January 29, 2020, 10:29am UTC](https://discuss.elastic.co/t/logstash-csv-kibana-input-error/216948/4 "2020-01-29T10:29:34Z")
Yes. Elasticsearch auto detects schema. You can also control this via mapping templates etc. ---
**Author:** ![ELK4Life](https://avatars.discourse-cdn.com/v4/letter/e/f4b2a3/32.png) [@ELK4Life](https://discuss.elastic.co/u/ELK4Life)\ **Post date:** [January 29, 2020, 10:58am UTC](https://discuss.elastic.co/t/logstash-csv-kibana-input-error/216948/5 "2020-01-29T10:58:47Z")
Thank you for your reply! Personally I don't think this is a multiline configuration issue. Every Kibana entry has it's own, unique, message field consisting of one, unique, CSV row. The issue is, that the complete CSV row is inside this message component. Therefore I can't filter on any of the CSV columns ![Kibana specific](https://us1.discourse-cdn.com/elastic/original/3X/1/6/160d3115d701eb232e9913b922afa2f564812619.png) ---
**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Post date:** [January 29, 2020, 1:54pm UTC](https://discuss.elastic.co/t/logstash-csv-kibana-input-error/216948/6 "2020-01-29T13:54:31Z")
I can see a csvparsefailure in there. You are receiving the data in json format not csv. Can you confirm? ---
**Author:** ![ELK4Life](https://avatars.discourse-cdn.com/v4/letter/e/f4b2a3/32.png) [@ELK4Life](https://discuss.elastic.co/u/ELK4Life)\ **Post date:** [January 29, 2020, 2:32pm UTC](https://discuss.elastic.co/t/logstash-csv-kibana-input-error/216948/7 "2020-01-29T14:32:40Z")
You were completely right! I checked and I was using the wrong cmd input for TShark which indeed outputted json instead of csv Thanks for pointing that out to me 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 26, 2020, 2:32pm UTC](https://discuss.elastic.co/t/logstash-csv-kibana-input-error/216948/8 "2020-02-26T14:32:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.