Csv file upload in logstash but data is not showing in kibana and elasticsearch

Elastic Stack Logstash
1

I am trying to upload countrties csv file on logstash and follow output I get on windows 10. Can some one help me on this so I will upload csv file in logstash

D:\ELK\Logstash\logstash-7.1.1\bin>logstash --config.test_and_exit -f logstash_countries.conf
Sending Logstash logs to D:/ELK/Logstash/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-06-26T09:53:19,298][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-06-26T09:53:24,703][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

D:\ELK\Logstash\logstash-7.1.1\bin>logstash -f logstash_countries.conf
Sending Logstash logs to D:/ELK/Logstash/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-06-26T12:41:53,691][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-26T12:41:53,711][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-06-26T12:42:01,570][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2019-06-26T12:42:01,774][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-06-26T12:42:01,846][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-06-26T12:42:01,852][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2019-06-26T12:42:01,882][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-06-26T12:42:01,901][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-06-26T12:42:01,938][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x62c81e8a run>"}
[2019-06-26T12:42:02,114][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-06-26T12:42:02,822][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-06-26T12:42:02,913][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-06-26T12:42:02,916][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-06-26T12:42:03,452][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

2

My config file is this

input
{
file
{
path => "D:\kibanaproject\countriesdata.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => ["Country","Region","Population","Area"]
}
mutate {convert => ["Population", "integer"]}
mutate {convert => ["Area", "integer"]}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "countriesdata-%{+dd.MM.YYYY}"
}
stdout {codec => rubydebug }
}

3

Welcome!

I moved your post to #logstash.

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image.jpg764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

4

My config file is this and when i try to run this on logstash , logstash start this config but no data is upload in kibana or elasticsearch

5

Output of below logstash config file

D:\ELK\Logstash\logstash-7.1.1\bin>logstash -f logstash_countries.conf

Sending Logstash logs to D:/ELK/Logstash/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-06-26T12:41:53,691][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-26T12:41:53,711][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-06-26T12:42:01,570][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2019-06-26T12:42:01,774][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-06-26T12:42:01,846][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-06-26T12:42:01,852][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2019-06-26T12:42:01,882][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-06-26T12:42:01,901][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-06-26T12:42:01,938][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x62c81e8a run>"}
[2019-06-26T12:42:02,114][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-06-26T12:42:02,822][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-06-26T12:42:02,913][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-06-26T12:42:02,916][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-06-26T12:42:03,452][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

6 
indent preformatted text by 4 spaces

input
{
file
{
path => "D:\kibanaproject\countriesdata.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => ["Country","Region","Population","Area"]
}
mutate {convert => ["Population", "integer"]}
mutate {convert => ["Area", "integer"]}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "countriesdata-%{+dd.MM.YYYY}"
}
stdout {codec => rubydebug }
}

7

Use forward slash instead of backslash in the path option of a file input.

8

Thank you problem resolved

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.