# Problems when indexing csv files **URL:** \ **Category:** Elasticsearch\ **Created:** [May 25, 2019, 10:22am UTC](https://discuss.elastic.co/t/problems-when-indexing-csv-files/182753 "2019-05-25T10:22:58Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Ayumalex](https://avatars.discourse-cdn.com/v4/letter/a/c6cbf5/32.png) [@Ayumalex](https://discuss.elastic.co/u/Ayumalex)\ **Post date:** [May 25, 2019, 10:22am UTC](https://discuss.elastic.co/t/problems-when-indexing-csv-files/182753/1 "2019-05-25T10:22:59Z")
Hello, I use ELK (for the first time) to process CSV files. However, I have a problem when I run logstash with my configuration file: ```auto input { file { path => "/home/user/network_capt/csv/*.csv" start_position => "beginning" sincedb_path => "/dev/null" } } filter { csv { columns => ["frame.number","frame.time_epoch","frame.time","frame.len","ip.src","ip.dst","ip.proto","tcp.srcport","tcp.dstport","udp.srcport","udp.dstport","_ws.col.Protocol","_ws.col.Info","tcp.window_size","tcp.flags","tcp.flags.syn","tcp.flags.ack","tcp.flags.fin","tcp.flags.push","tcp.flags.reset","tcp.flags.urg","tcp.analysis.retransmission","icmp.type","icmp.code","dns.qry.name","dns.qry.type"] separator => ";" skip_empty_columns => "true" skip_empty_rows => "true" } mutate { add_field => { "hostname" => "%{ip.src}"} } date { match => ["timestamp", "UNIX", "UNIX_MS"] locale => "en" } geoip { source => "ip.src" target => "ip.src" } geoip { source => "ip.dst" target => "ip.dst" } } output { elasticsearch { hosts => "localhost:9200" index => "honeypot-%{+YYYYMMdd}" } stdout{} } ``` Here's what I get: ```auto Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN] 2019-05-24 16:03:55.748 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2019-05-24 16:03:55.765 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.1.0"} [INFO] 2019-05-24 16:04:04.562 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} [WARN] 2019-05-24 16:04:04.806 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"} [INFO] 2019-05-24 16:04:05.054 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7} [WARN] 2019-05-24 16:04:05.058 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [INFO] 2019-05-24 16:04:05.103 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]} [INFO] 2019-05-24 16:04:05.128 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template [INFO] 2019-05-24 16:04:05.135 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"} [INFO] 2019-05-24 16:04:05.183 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"} [INFO] 2019-05-24 16:04:05.242 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, :thread=>"#"} [INFO] 2019-05-24 16:04:05.342 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [INFO] 2019-05-24 16:04:05.827 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"} [INFO] 2019-05-24 16:04:06.023 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [INFO] 2019-05-24 16:04:06.051 [[main]9600} [WARN] 2019-05-24 16:04:09.679 [[main]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"honeypot-20190524", :_type=>"_doc", :routing=>nil}, #], :response=>{"index"=>{"_index"=>"honeypot-20190524", "_type"=>"_doc", "_id"=>"0Csn6moB1EZoSNMt-Ht2", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Can't merge a non object mapping [tcp.flags] with an object mapping [tcp.flags]"}}}} [WARN] 2019-05-24 16:04:09.684 [[main]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"honeypot-20190524", :_type=>"_doc", :routing=>nil}, #], :response=>{"index"=>{"_index"=>"honeypot-20190524", "_type"=>"_doc", "_id"=>"0Ssn6moB1EZoSNMt-Ht2", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Can't merge a non object mapping [tcp.flags] with an object mapping [tcp.flags]"}}}} ``` Then as a result of these warnings, here is what I get: ![Capture%20d%E2%80%99%C3%A9cran_2019-05-24_15-56-46](https://us1.discourse-cdn.com/elastic/original/3X/d/3/d3395d8f505dfb875494bd7c5efb1727fe5565d9.png) My index is well created, only I feel that it is empty. In addition, the field "mapping" is also empty which intrigues me: ![Capture%20d%E2%80%99%C3%A9cran_2019-05-24_15-55-32](https://us1.discourse-cdn.com/elastic/original/3X/d/0/d00d77514ebc55b41ebce31f5e712b097d74583d.png) In addition, when I want to create an index pattern, I do not see the index that was created, so I can not create an index pattern that matches with my index: ![Capture%20d%E2%80%99%C3%A9cran_2019-05-24_15-55-54](https://us1.discourse-cdn.com/elastic/original/3X/7/5/75f8316186ef2ac15e00a232d57eda85b3ad2781.png) I would like to have a visual on the data of my CSV, but I am currently blocked by the problem that I presented above. I am a beginner with ELK, I only use it since today, do not hesitate to give me your advice. **Excuse me** for the quality of this topic (& for my english) but I lack of time and I need to find a solution rather quickly ... **Thank you in advance to all for your help.** ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 25, 2019, 10:41am UTC](https://discuss.elastic.co/t/problems-when-indexing-csv-files/182753/2 "2019-05-25T10:41:49Z")
I answered you in the french version of the post you posted yesterday evening: [Problèmes lors d'indexation de fichiers CSV](https://discuss.elastic.co/t/problemes-lors-dindexation-de-fichiers-csv/182688) There is no need to post it twice. Read [this](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and specifically the "Also be patient" part. ---
**Author:** ![shin-higuchi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shin-higuchi/32/46524_2.png) [@shin-higuchi](https://discuss.elastic.co/u/shin-higuchi)\ **Post date:** [May 25, 2019, 10:45am UTC](https://discuss.elastic.co/t/problems-when-indexing-csv-files/182753/3 "2019-05-25T10:45:34Z")
tcp.flags cannot has a concrete value. Because of you having tcp.flags.reset field, Elasticsearch interprets tcp.flags as an object. However, tcp.flags seems to have a concrete text value in your output of logstash. Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 22, 2019, 10:45am UTC](https://discuss.elastic.co/t/problems-when-indexing-csv-files/182753/4 "2019-06-22T10:45:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.