How to load csv file into elasticsearch for visualizaion in kibana

Elastic Stack Logstash
1

I have csv file that i attached with this.

I have already try with logstash, with this i can able to open that file but can not do the visualization in kibana. its shows some error. I m using the following code for logstash::
input {
file {
path => "/home/hdcluster/data.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","

 columns => ["DATE","TIME","LATITUDE","LONGITUDE","TEMP","VIBRATION"]

}
}
output {
elasticsearch {
action => "index"
hosts => "http://localhost:9200"
index => "data"
workers => 1
}
stdout {}
}

that tutorial provided by "elastic- team" that i have already done, its working perfectly.(that shakesper, amount and bank problem). But for my data set it will not work so please help me to load this data.
Thank you

2

its shows some error

What error?

3

At time of configuration as an index what should I take? Date or @timestamp

If i take timestamp this is what i got..

Screenshot from 2016-09-27 10:28:36.png1302×744 118 KB

why is this showing like this
message:2016-09-12,11:06:42,13.059061,80.256934,30,202@version:1@timestamp:September 27th 2016, 10:23:40.559path:/home/hdcluster/data.csvhost:namenodeDATE:September 12th 2016, 05:30:00.000TIME:11:06:42LATITUDE:13.059061LONGITUDE:80.256934TEMP:30VIBRATION:202_id:AVdp_cexVEIAnSsq2H5A_type:logs_index:sample_score:1

my dataset only have date,time,latitude,longitude,temp and vibration
but it was showing message, version, path

and if I take Date than this thing i got.

Screenshot from 2016-09-27 10:30:59.png1298×710 78.3 KB

4

I want to show line chart or area chart with using time and vibration column.

5

You will have to use date plugin in Logstash to parse the time. In your case, below should be the conf:
filter
{
csv {
columns => ["DATE","TIME","LATITUDE","LONGITUDE","TEMP","VIBRATION"]
}
mutate {
add_field => {
"Incident Date" => "%{DATE}%{TIME}"
}
}
date
{
match => [ "insert_date", "YYYY-MM-ddHH:mm:ss"]
}
}

1 Like
6

Thank u for ur replay @Meenal.Luktuke

but i have a confusion that i have to match date with my all data set?
means i have to manually enter all the date??

7

Matching using date filter is only for format. Elasticsearch, by default, will expect the value of @timestamp in format: YYYY-MM-ddTHH:mm:ss
Since you don't have that,

  1. concat DATE and TIME to create a new field
  2. Using date filter, make this new field as @timestamp
8

Okiez I ll try. Thank u ones again :slight_smile: