Import CSV File as input in logstash and the output data to elasticsearch

Elastic Stack Logstash
1

Hi,
I'm new in elasticsearch, i install elasticsearch, logstash and kibana, i need to import a CSV file as input in logstash and pass the outuput data to elasticsearch, I create the index in elasticsearch , in kibana i apprciate th index but any thing from the data i can't visualize the data .
please can help me .
my code is:

input {
  file {
    path => "/Users/salma/Desktop/friends.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
     separator => ","
    columns => ["id","name","age","money"]
  }
}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "data-index"
  }
stdout {}
}

I run logstash and he reurn this message :
Could not find log4j2 configuration at path /Project/elk/logstash/config/log4j2.
properties. Using default config which logs to console
12:05:30.200 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeli
ne {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.b
atch.delay"=>5, "pipeline.max_inflight"=>500}
12:05:30.562 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main s
tarted
12:05:30.662 [Api Webserver] INFO logstash.agent - Successfully started Logstas
h API endpoint {:port=>9600}

2

Are you seeing anything in Kibana? If yes, what?

3

Hi Magnus,
I see just the index create in elasticsearch and i can't start the logstash and elasticsearch services

4

I see just the index create in elasticsearch

What do you mean?

and i can't start the logstash and elasticsearch services

Tell us more about this.

5

Hi,
I don't see any data in kibana . when i start the logstach service windows me return an error .

6

when i start the logstach service windows me return an error

Without more details about the error you're getting it's impossible to help.

7

when i run the logstash service window return this error:

8

Does the service configuration point to a file that doesn't exist?

discuss.elastic.co has a French-speaking group that might be a better fit for you.

9

yes the service configuration point to a file doesn't exist

I create test.conf
input {
file {
path => "/Users/salma/Desktop/creditcard.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Time","V1","V2","V3","V4","V5","V6","V7","V8","V9","V10","V11","V12","V13","V14","V15","V16","V17","V18","V19","V20","V21","V22","V23","V24","V25","V26","V27","V28","Amount"]
remove_field => ["class"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "dataset"
}
}
this error appreas:

C:\Project\elk\logstash\bin>logstash -f test.conf
Picked up _JAVA_OPTIONS: -Xmx512M -Xms256M
Could not find log4j2 configuration at path /Project/elk/logstash/config/log4j2.
properties. Using default config which logs to console
16:17:36.800 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - El
asticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhos
t:9200/]}}
16:17:36.810 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Ru
nning health check to see if an Elasticsearch connection is working {:healthchec
k_url=>http://localhost:9200/, :path=>"/"}
16:17:36.950 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Re
stored connection to ES instance {:url=>#<URI::HTTP:0x478f1ddb URL:http://localh
ost:9200/>}
16:17:36.960 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Us
ing mapping template from {:path=>nil}
16:17:37.240 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - At
tempting to install template {:manage_template=>{"template"=>"logstash-*", "vers
ion"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_defaul
t_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message
_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"
type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mappin
g_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keywo
rd"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "in
clude_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false},
 "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"t
ype"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"h
alf_float"}}}}}}}}
16:17:37.250 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Ne
w Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<U
RI::HTTP:0x7bf2ccce URL:http://localhost:9200>]}
16:17:37.260 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeli
ne {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.b
atch.delay"=>5, "pipeline.max_inflight"=>500}
16:17:37.735 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main s
tarted
16:17:37.863 [Api Webserver] INFO  logstash.agent - Successfully started Logstas
h API endpoint {:port=>9600}
10

Logstash is starting up just fine. Read the whole log.

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.