Not able to load custom data using csv file in logstash

Hi Team- I am really new to elk and looking for the help.

i am trying importing data from csv to elasticsearch using logstash but that is getting failed. Below is my CSV file.
```          Sl.No	InstanceName	InstanceID	MAXCPU%
        1	POC-LMS-Application-Server	xxxxxxx	38.83333333
        2	POC-LMS-DB-Server	xxxxxx2.166666667
        3	Zabbix_Test_Server	xxxxxx	98.19672131
        4	N/A	xxxxx	5.901639344
        5	POC-Anthos	xxxxx	11.5
        6	POC-BastionHost xxxxx	99.49152542 ```

Below is logstash conf file 

[root@xxxx]# cat /etc/logstash/conf.d/input.conf
path => "/home/xxx/aws/result/MAXCPU.csv"
path => "/home/xxx/aws/result/MINCPU.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
[root@xxxx conf.d]# cat /etc/logstash/conf.d/output.conf
output {

elasticsearch {

hosts => "xxxxx:9200"

index => "logstash-accesslog"

user => "xxxxx"

password => "xxxxx"


stdout {}

[root@xxxx conf.d]# cat /etc/logstash/conf.d/filter.conf
filter {

csv {

separator => ","

columns => [ "Sl.No" , "InstanceName" , "InstanceID" , "MAXCPU%" , "MINCPU%" ]


date {

match => [ "InstanceID" , "UNIX" ]

target => "EventTime"


mutate {convert => ["InstanceID", "integer"]}

mutate {convert => ["InstanceName", "integer"]}

[root@xxxx conf.d]#

PFB logs "/var/log/logstash/logstash-plain.log" ```

 ```   [2020-08-05T18:16:00,547][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@xxxxxx:9200/"}
    [2020-08-05T18:16:00,759][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
    [2020-08-05T18:16:00,769][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
    [2020-08-05T18:16:00,912][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//xxxxxx:9200"]}
    [2020-08-05T18:16:01,021][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
    [2020-08-05T18:16:01,128][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/conf.d/filter.conf", "/etc/logstash/conf.d/input.conf", "/etc/logstash/conf.d/output.conf"], :thread=>"#<Thread:0x5d50340e run>"}
    [2020-08-05T18:16:01,335][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
    [2020-08-05T18:16:03,018][INFO ][logstash.javapipeline    ][main] Pipeline started {""=>"main"}
    [2020-08-05T18:16:03,118][INFO ][filewatch.observingtail  ][main][c0ff6327f421f602f1ac30374c6dda8bd9a4bedc093fad223e95c44720012e99] START, creating Discoverer, Watch with file and sincedb collections
    [2020-08-05T18:16:03,116][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
    [2020-08-05T18:16:03,686][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
    [root@xxxxxx conf.d]#```

Also PFB output from Kibana Dev tools

     ```GET /logstash-accesslog/_search

      "took" : 1,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      "hits" : {
        "total" : {
          "value" : 0,
          "relation" : "eq"
        "max_score" : null,
        "hits" : [ ]

Please check and help me to get rid over from this issue.

 ```   elasticsearch.x86_64                                              7.8.1-1                                                 @elastic-7.x
    kibana.x86_64                                                     7.8.1-1                                                 @elastic-7.x
    logstash.noarch                                                   1:7.8.1-1                                               @elastic-7.x```

machine OS : CentOS Linux release 7.8.2003 (Core)


First could you format your message correctly because it is diffcult to read.

My first remark is that you wrote 3 conf files:

  • input.conf
  • filter.conf
  • output.conf

I suggest you to prefix their name with number:

  • 01-input.conf
  • 02-filter.conf
  • 03-output.conf

Because Logstash loads them in memory in an alfanumeric order so your filter section is currently located before your input section.

2/ Your separator is not ",", change it to spaces or tabulation

separator => ","

3/ You maybe have to skip_header

4/ You try to transform 'InstanceID' into a date (EventTime), but I'm not sure that field is a date

5/ You try to convert 'InstanceName' to an integer, but in your input file it seems to be a string

Even if your filters do not work, the file input should be generating events and they should be written to elasticsearch as documents.

I suggest you enabled log.level debug, or even trace, which should show you the events being flushed into the pipeline.


    `Thanks for your reply, really  appreciate your response :slightly_smiling_face:`

    `As per your suggestion i have renamed the input, output and filter files.` 

    `[root@xxxx conf.d]# pwd ; ls`
       ` /etc/logstash/conf.d`
        `01-input.conf  02-filter.conf  03-output.conf`
   `     [root@xxxx conf.d]#`

`PFB is screenshot of my sample csv file : !`[sample_CSV|490x166]`(upload://xRtOJVkzjNvLdK8sAUANZU1a7TT.png)` 
`Please do refer it.`

`PFB 02-filter.conf file and do let me know what exactly i have to modify in it or else` `please paste correct filter file further in comment section so that I can put same` `configuration on my ELK stack.` 

`    [root@xxx conf.d]# cat 02-filter.conf`
`    filter {`

    `csv {`

`    separator => ","`

        `columns => [ "Sl.No" , "InstanceName" , "InstanceID" , "MAXCPU%" , "MINCPU%" ]`


   ` grok  {`

    `match => [ "InstanceID" , "UNIX" ]`

`    target => "EventTime"`

`    }`

    `mutate {convert => ["InstanceID", "string"]`}

`    mutate {convert => ["InstanceName", "string"]}`

 `   }`

********** Logstash plain log********
`[2020-08-06T18:06:50,419][INFO ][logstash.outputs.elasticsearch][main] Using default` `mapping template`
    [2020-08-06T18:06:50,627][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
    [2020-08-06T18:06:50,827][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/02-filter.conf", "/etc/logstash/conf.d/03-output.conf"], :thread=>"#<Thread:0x7fa94e9f run>"}
    [2020-08-06T18:06:53,437][INFO ][logstash.javapipeline    ][main] Pipeline started {""=>"main"}
    [2020-08-06T18:06:53,512][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
    [2020-08-06T18:06:53,535][INFO ][filewatch.observingtail  ][main][3c387366658701f23901c694a7dab7b8fa4b4604a48f33e0822a68e6221a18ba] START, creating Discoverer, Watch with file and sincedb collections
    [2020-08-06T18:06:54,100][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

    `Note : I am trying to take csv file from server A and move to B  (using logstash)  server where my complete ELK infra is running. In server A , I have installed elasticsearch , logstash and kibana and in all host settings i have provide host B Ip address.` 
please  check all above provided data and help as I am very new in ELK technology.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.