# Not able to load custom data using csv file in logstash **URL:** \ **Category:** Logstash\ **Created:** [August 5, 2020, 3:11pm UTC](https://discuss.elastic.co/t/not-able-to-load-custom-data-using-csv-file-in-logstash/243891 "2020-08-05T15:11:29Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Divya\_Mathur\_ELK](https://avatars.discourse-cdn.com/v4/letter/d/c0e974/32.png) [@Divya\_Mathur\_ELK](https://discuss.elastic.co/u/Divya_Mathur_ELK)\ **Post date:** [August 5, 2020, 3:11pm UTC](https://discuss.elastic.co/t/not-able-to-load-custom-data-using-csv-file-in-logstash/243891/1 "2020-08-05T15:11:30Z")
```` Hi Team- I am really new to elk and looking for the help. i am trying importing data from csv to elasticsearch using logstash but that is getting failed. Below is my CSV file. ``` Sl.No InstanceName InstanceID MAXCPU% 1 POC-LMS-Application-Server xxxxxxx 38.83333333 2 POC-LMS-DB-Server xxxxxx2.166666667 3 Zabbix_Test_Server xxxxxx 98.19672131 4 N/A xxxxx 5.901639344 5 POC-Anthos xxxxx 11.5 6 POC-BastionHost xxxxx 99.49152542 ``` Below is logstash conf file ```[root@xxx]# [root@xxxx]# cat /etc/logstash/conf.d/input.conf input { file { path => "/home/xxx/aws/result/MAXCPU.csv" path => "/home/xxx/aws/result/MINCPU.csv" start_position => "beginning" sincedb_path => "/dev/null" } } [root@xxxx conf.d]# cat /etc/logstash/conf.d/output.conf output { elasticsearch { hosts => "xxxxx:9200" index => "logstash-accesslog" user => "xxxxx" password => "xxxxx" } stdout {} } [root@xxxx conf.d]# cat /etc/logstash/conf.d/filter.conf filter { csv { separator => "," columns => ["Sl.No" , "InstanceName" , "InstanceID" , "MAXCPU%" , "MINCPU%"] } date { match => ["InstanceID" , "UNIX"] target => "EventTime" } mutate {convert => ["InstanceID", "integer"]} mutate {convert => ["InstanceName", "integer"]} } [root@xxxx conf.d]# ` PFB logs "/var/log/logstash/logstash-plain.log" ``` ``` [2020-08-05T18:16:00,547][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@xxxxxx:9200/"} [2020-08-05T18:16:00,759][INFO][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7} [2020-08-05T18:16:00,769][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [2020-08-05T18:16:00,912][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//xxxxxx:9200"]} [2020-08-05T18:16:01,021][INFO][logstash.outputs.elasticsearch][main] Using default mapping template [2020-08-05T18:16:01,128][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/conf.d/filter.conf", "/etc/logstash/conf.d/input.conf", "/etc/logstash/conf.d/output.conf"], :thread=>"#"} [2020-08-05T18:16:01,335][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [2020-08-05T18:16:03,018][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} [2020-08-05T18:16:03,118][INFO][filewatch.observingtail][main][c0ff6327f421f602f1ac30374c6dda8bd9a4bedc093fad223e95c44720012e99] START, creating Discoverer, Watch with file and sincedb collections [2020-08-05T18:16:03,116][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-08-05T18:16:03,686][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [root@xxxxxx conf.d]#``` Also PFB output from Kibana Dev tools ```GET /logstash-accesslog/_search { "took" : 1, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 0, "relation" : "eq" }, "max_score" : null, "hits" : [] } }``` Please check and help me to get rid over from this issue. ``` elasticsearch.x86_64 7.8.1-1 @elastic-7.x kibana.x86_64 7.8.1-1 @elastic-7.x logstash.noarch 1:7.8.1-1 @elastic-7.x``` machine OS : CentOS Linux release 7.8.2003 (Core) ```` ---
**Author:** ![Rom1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rom1/32/49699_2.png) [@Rom1](https://discuss.elastic.co/u/Rom1)\ **Post date:** [August 5, 2020, 3:23pm UTC](https://discuss.elastic.co/t/not-able-to-load-custom-data-using-csv-file-in-logstash/243891/2 "2020-08-05T15:23:43Z")
Hi, First could you format your message correctly because it is diffcult to read. My first remark is that you wrote 3 conf files: - input.conf - filter.conf - output.conf I suggest you to prefix their name with number: - 01-input.conf - 02-filter.conf - 03-output.conf Because Logstash loads them in memory in an alfanumeric order so your filter section is currently located before your input section. 2/ Your separator is not ",", change it to spaces or tabulation ```auto separator => "," ``` 3/ You maybe have to `skip_header` 4/ You try to transform '`InstanceID`' into a date (`EventTime`), but I'm not sure that field is a date 5/ You try to convert '`InstanceName`' to an integer, but in your input file it seems to be a string ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 5, 2020, 3:32pm UTC](https://discuss.elastic.co/t/not-able-to-load-custom-data-using-csv-file-in-logstash/243891/3 "2020-08-05T15:32:46Z")
Even if your filters do not work, the file input should be generating events and they should be written to elasticsearch as documents. I suggest you enabled log.level debug, or even trace, which should show you the events being flushed into the pipeline. ---
**Author:** ![Divya\_Mathur\_ELK](https://avatars.discourse-cdn.com/v4/letter/d/c0e974/32.png) [@Divya\_Mathur\_ELK](https://discuss.elastic.co/u/Divya_Mathur_ELK)\ **Post date:** [August 6, 2020, 1:02pm UTC](https://discuss.elastic.co/t/not-able-to-load-custom-data-using-csv-file-in-logstash/243891/4 "2020-08-06T13:02:20Z")
``` Hi, `Thanks for your reply, really appreciate your response :slightly_smiling_face:` `As per your suggestion i have renamed the input, output and filter files.` `[root@xxxx conf.d]# pwd ; ls` ` /etc/logstash/conf.d` `01-input.conf 02-filter.conf 03-output.conf` ` [root@xxxx conf.d]#` `PFB is screenshot of my sample csv file : !`[sample_CSV|490x166]`(upload://xRtOJVkzjNvLdK8sAUANZU1a7TT.png)` `Please do refer it.` `PFB 02-filter.conf file and do let me know what exactly i have to modify in it or else` `please paste correct filter file further in comment section so that I can put same` `configuration on my ELK stack.` ` [root@xxx conf.d]# cat 02-filter.conf` ` filter {` `csv {` ` separator => ","` `columns => ["Sl.No" , "InstanceName" , "InstanceID" , "MAXCPU%" , "MINCPU%"]` } ` grok {` `match => ["InstanceID" , "UNIX"]` ` target => "EventTime"` ` }` `mutate {convert => ["InstanceID", "string"]`} ` mutate {convert => ["InstanceName", "string"]}` ` }` **********Logstash plain log******** `[2020-08-06T18:06:50,419][INFO][logstash.outputs.elasticsearch][main] Using default` `mapping template` [2020-08-06T18:06:50,627][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [2020-08-06T18:06:50,827][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/02-filter.conf", "/etc/logstash/conf.d/03-output.conf"], :thread=>"#"} [2020-08-06T18:06:53,437][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} [2020-08-06T18:06:53,512][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-08-06T18:06:53,535][INFO][filewatch.observingtail][main][3c387366658701f23901c694a7dab7b8fa4b4604a48f33e0822a68e6221a18ba] START, creating Discoverer, Watch with file and sincedb collections [2020-08-06T18:06:54,100][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} ************** `Note : I am trying to take csv file from server A and move to B (using logstash) server where my complete ELK infra is running. In server A , I have installed elasticsearch , logstash and kibana and in all host settings i have provide host B Ip address.` ******************* please check all above provided data and help as I am very new in ELK technology. ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 3, 2020, 1:02pm UTC](https://discuss.elastic.co/t/not-able-to-load-custom-data-using-csv-file-in-logstash/243891/5 "2020-09-03T13:02:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.