Logstash 5.1.1 gets hanged while reading csv

Elastic Stack Logstash
1

when i run logstash. it starts read the csv send data to elasticsearch as well. but it hangs in the middle.

logs show no error

    input {
     file {
        path => "/opt/logstash-5.1.1/bin/01.12.2016.csv"
        start_position => "beginning"
       sincedb_path => "/dev/null"
      }
    }
    filter {
      csv {
          separator => ","
         columns => ["departureDate","flightNo","passportNo"]
      }

    if [departureDate] == "departureDate" { drop{ } }

    }
    output {
       elasticsearch {
         hosts => "http://192.168.2.26:9200"
         index => "dollar"
      }
    stdout { }
    }

Below shows the console output while reading the file

2016-12-15T12:43:10.216Z localhost.localdomain 2016-12-01,UL116,E0100000
2016-12-15T12:43:10.216Z localhost.localdomain 2016-12-01,UL116,E0000000
2016-12-15T12:43:10.216Z localhost.localdomain 2016-12-01,UL116,E00000000
2016-12-15T12:43:10.216Z localhost.localdomain 2016-12-01,UL116,NC000000
2016-12-15T12:43:10.216Z localhost.localdomain 2016-12-01,UL116,LA0000000
2016-12-15T12:43:10.217Z localhost.localdomain 2016-12-01,UL116,L00000000
2016-12-15T12:43:10.217Z localhost.localdomain 2016-12-01,UL116,E00000000

checked cpu , overload its fine,,

i am scratching my head now

2

when i run in --verbose --debug

2016-12-15T15:25:54.356Z localhost.localdomain 2010-08-14,0.0655,0.0695,0.0645,0.067,4404.2,295.3084789,0.067051559625
2016-12-15T15:25:54.356Z localhost.localdomain 2010-08-13,0.0665,0.068,0.0645,0.0645,3591.77,233.8004368,0.065093376469
2016-12-15T15:25:54.357Z localhost.localdomain 2010-08-12,0.068,0.07,0.06141,0.07,2062.31,134.4872288,0.065211936518
2016-12-15T15:25:54.357Z localhost.localdomain 2010-08-11,0.06,0.07541,0.06,0.067,14061.18,1015.6416825,0.07223018854
2016-12-15T15:25:54.357Z localhost.localdomain 2010-08-10,0.068,0.0709,0.06651,0.07,1310.39,88.8736329,0.067822276498
2016-12-15T15:25:54.370Z localhost.localdomain 2010-08-09,0.06091,0.0735,0.0593,0.071,13631.09,886.9327103,0.065066895626
2016-12-15T15:25:54.371Z localhost.localdomain 2010-08-08,0.059,0.061,0.059,0.0609,2201.0,132.5954,0.060243253067
2016-12-15T15:25:54.371Z localhost.localdomain 2010-08-07,0.0608,0.0622,0.059,0.059,2619.0,157.3408,0.060076670485
2016-12-15T15:25:54.371Z localhost.localdomain 2010-08-06,0.062,0.0624,0.0607,0.0623,1395.0,85.908,0.061582795699
2016-12-15T15:25:54.372Z localhost.localdomain 2010-08-05,0.0581,0.061,0.058,0.061,5034.07,303.609091,0.06031086
2016-12-15T15:25:54.372Z localhost.localdomain 2010-08-04,0.06229,0.06231,0.057,0.057,3494.0,210.765111,0.060322012307
2016-12-15T15:25:54.374Z localhost.localdomain 2010-08-03,0.06,0.065,0.059,0.06,9821.46,606.0456913,0.061706272927
2016-12-15T15:25:54.374Z localhost.localdomain 2010-08-02,0.0627,0.0633,0.06,0.06,3599.0,221.20058,0.061461678244
2016-12-15T15:25:54.377Z localhost.localdomain 2010-08-01,0.061,0.065,0.06,0.0611,2601.0,162.6451925,0.06253179258
2016-12-15T15:25:54.377Z localhost.localdomain 2010-07-31,0.065,0.06889,0.056,0.06785,4022.25,243.898458,0.060637319411
2016-12-15T15:25:54.378Z localhost.localdomain 2010-07-30,0.0698,0.0698,0.0582,0.0627,3020.85,198.5299149,0.065719885099
2016-12-15T15:25:54.378Z localhost.localdomain 2010-07-29,0.0597,0.0699,0.0571,0.0699,8058.49,528.3183128,0.065560460185
2016-12-15T15:25:54.379Z localhost.localdomain 2010-07-28,0.06,0.062,0.054,0.0589,4390.29,255.759194,0.058255649171
2016-12-15T15:25:54.380Z localhost.localdomain 2010-07-27,0.053,0.0605,0.053,0.06,3373.69,196.91835,0.058368833532
2016-12-15T15:25:54.380Z localhost.localdomain 2010-07-26,0.05,0.056,0.05,0.056,877.0,46.907,0.053485746864
2016-12-15T15:25:54.380Z localhost.localdomain 2010-07-25,0.05545,0.05941,0.0505,0.0505,1551.48,85.062975,0.054826987779
2016-12-15T15:25:54.381Z localhost.localdomain 2010-07-24,0.06161,0.06161,0.05049,0.05454,496.32,26.7277095,0.053851768013
2016-12-15T15:25:54.381Z localhost.localdomain 2010-07-23,0.0505,0.06767,0.0505,0.06262,2402.5,141.0650511,0.058715942185
2016-12-15T15:25:54.382Z localhost.localdomain 2010-07-22,0.07921,0.08181,0.0505,0.0505,2160.0,129.78428,0.060085314815
2016-12-15T15:25:54.382Z localhost.localdomain 2010-07-21,0.07425,0.07921,0.06634,0.07921,575.0,42.2622,0.073499478261
2016-12-15T15:25:54.383Z localhost.localdomain 2010-07-20,0.08181,0.08181,0.07426,0.07474,262.0,20.58952,0.078585954198
2016-12-15T15:25:54.383Z localhost.localdomain 2010-07-19,0.0909,0.09307,0.07723,0.0808,574.0,49.6613,0.086517944251
2016-12-15T15:25:54.383Z localhost.localdomain 2010-07-18,0.05941,0.08585,0.05941,0.08584,75.01,5.0919941,0.06788420344
2016-12-15T15:25:54.384Z localhost.localdomain 2010-07-17,0.04951,0.04951,0.04951,0.04951,20.0,0.9902,0.04951
[2016-12-15T20:25:55,364][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:25:55 +0500}
[2016-12-15T20:25:55,419][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2016-12-15T20:25:56,368][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:25:56 +0500}
[2016-12-15T20:25:57,374][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:25:57 +0500}
[2016-12-15T20:25:58,377][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:25:58 +0500}
[2016-12-15T20:25:59,380][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:25:59 +0500}
[2016-12-15T20:26:00,389][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:26:00 +0500}
[2016-12-15T20:26:00,423][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2016-12-15T20:26:01,395][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:26:01 +0500}
[2016-12-15T20:26:02,400][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:26:02 +0500}
[2016-12-15T20:26:03,402][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:26:03 +0500}
[2016-12-15T20:26:04,410][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:26:04 +0500}
[2016-12-15T20:26:05,413][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:26:05 +0500}
[2016-12-15T20:26:05,423][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2016-12-15T20:26:06,422][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:26:06 +0500}
[2016-12-15T20:26:07,426][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers {:created_at=>2016-12-15 20:26:07 +0500}
[2016-12-15T20:26:08,419][DEBUG][logstash.inputs.file ] _globbed_files: /opt/logstash-5.0.2/bin/1.csv: glob is: ["/opt/logstash-5.0.2/bin/1.csv"]
[2016-12-15T20:26:08,459][DEBUG][logstash.instrument.collector] Collector: Sending snapshot to observers

3

fixed it, there was an empty line in the csv file, by default the file exported from the mysql has an empty line in the bottom, backspace it and it fixes the issue..

took me alot of time to figure it finally it was an empty line :rage:

ELK needs to address it as if we automating csv read in a directory removing the empty line should not be a manual process,

also i found that it gives csv parse failure for some records.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.