Indexing CSV data into elasticsearch

Elastic Stack Logstash
1

I have tried the below code to import data from CSV into elasticsearch using logstash:

input {
file {
path => ["D:\test.csv"]
type => "csv"
start_position => "beginning"
}
}

filter {
csv {
columns => ["app1", "app2", "c_strength"]
separator => ","
}
}
output {
elasticsearch {
action => "index"
hosts => "127.0.0.1"
index => "logstash-CSVTest"
}
stdout { codec => rubydebug }

Few records from the csv file is getting imported into logstash, but there is no index created to it.

Someone please suggest/correct me. Thanks in advance!

2

Moving to the Logstash forums :slight_smile:

3

It's probably this https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files

4

I did not understand how can we import csv data into elasticsearch using logstash.

5

Is there any plugin available for the same?

6

Disable the elasticsearch output for now and pay attention to the stdout { codec => rubydebug } output only. Use it to make sure Logstash is reading the input data. Please study the link Mark sent and search the archives for sincedb_path and ignore_older.

7

Hi,
I am new to elastic search and I am trying to import data from csv to elasticsearch using logstash.
I am using the below logstash conf file

input {
file {
path => ["D:/AnalyticsTool/Exe/logstash-5.2.0/logstash-5.2.0/bin/data.csv"]
type => "core2"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["Date","Open","High","Low","Close","Volume","Adj Close"]
}
mutate {convert => ["High", "float"]}
mutate {convert => ["Open", "float"]}
mutate {convert => ["Low", "float"]}
mutate {convert => ["Close", "float"]}
mutate {convert => ["Volume", "float"]}
}
output {
elasticsearch {
action => "index"
hosts => ["localhost:9200"]
index => "stock"
workers => 1
user => elastic
password => changeme
}
stdout { codec => rubydebug}
}

I am not able to visualize in kibana. Do we need to create index manually in elastic search or kibana will create above mentioned index automatically ?
Please let me know if I am missing anything

Thanks in advance

8

@Uma_Ananthu, please start a new topic for your question. I do recommend that you read and understand this topic first. My last recommendations probably apply to you too.