Hello,
I want to import a csv file to elasticsearch via logstash but later I can find the index called bitcoin-price is created but missing a lot of data, 1320 rows with 1310 rows missing. Here are some details below.
The first row of my csv data <7 columns>:
2014-02-25 173.2 173.84343 101.62872 135 29886.75324 3667985.396 122.7294704
running logstach <version is 4.2.6> , it read all data in my csv and keeps running without any error
2017-02-24T15:41:50.154Z simons-MacBook-Air.local 2010-07-20,0.08181,0.08181,0.07426,0.07474,262,20.58952,0.078585954
...
The elasticsearch part:
[2017-02-24T09:39:52,163][INFO ][o.e.n.Node ] [ZUWG2gq] started
[2017-02-24T09:39:52,494][INFO ][o.e.g.GatewayService ] [ZUWG2gq] recovered [4] indices into cluster_state
[2017-02-24T09:39:53,171][INFO ][o.e.c.r.a.AllocationService] [ZUWG2gq] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
[2017-02-24T09:41:10,214][INFO ][o.e.c.m.MetaDataDeleteIndexService] [ZUWG2gq] [bitcoin-prices/pEbdalJaSSy_UtulRW8pmg] deleting index
[2017-02-24T09:41:47,013][INFO ][o.e.c.m.MetaDataCreateIndexService] [ZUWG2gq] [bitcoin-prices] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2017-02-24T09:41:47,266][INFO ][o.e.c.m.MetaDataMappingService] [ZUWG2gq] [bitcoin-prices/isnPZWOzQfWG50WWszYZcg] create_mapping [logs]
[2017-02-24T09:41:48,107][WARN ][o.e.m.j.JvmGcMonitorService] [ZUWG2gq] [gc][119] overhead, spent [565ms] collecting in the last [1s]
my logstash.conf:
input {
file {
path => "/Users/simon/Desktop/simon.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Date","Open","High","Low","Close","Volume (BTC)", "Volume (Currency)" ,"Weighted Price"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "bitcoin-prices"
}
stdout {}
}
Watch data ingestion status from kibana console:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open bitcoin-prices isnPZWOzQfWG50WWszYZcg 5 1 1319 0 1.2mb 1.2mb
After tying GET /bitcoin-prices/_search?q=* , you can see:
{
"took": 9,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1319,
"max_score": 1,
"hits": [
{
"_index": "bitcoin-prices",
"_type": "logs",
"_id": "AVpwyQcYyBWtLM3Y0ayZ",
"_score": 1,
"_source": {
"High": "173.84343",
"Volume (BTC)": "29886.75324",
"Volume (Currency)": "3667985.396",
"Weighted Price": "122.7294704",
"message": "2014-02-25,173.2,173.84343,101.62872,135,29886.75324,3667985.396,122.7294704\r",
"Date": "2014-02-25",
"tags": [],
"Open": "173.2",
"path": "/Users/simon/Desktop/simon.csv",
"@timestamp": "2017-02-24T15:41:45.750Z",
"Low": "101.62872",
"@version": "1",
"host": "simons-MacBook-Air.local",
"Close": "135"
}
},
Thanks for all answers, I'm appreciated u can share your ideas here to help me.