Hi Team- I am really new to elk and looking for the help.
i am trying importing data from csv to elasticsearch using logstash but that is getting failed. Below is my CSV file.
``` Sl.No InstanceName InstanceID MAXCPU%
1 POC-LMS-Application-Server xxxxxxx 38.83333333
2 POC-LMS-DB-Server xxxxxx2.166666667
3 Zabbix_Test_Server xxxxxx 98.19672131
4 N/A xxxxx 5.901639344
5 POC-Anthos xxxxx 11.5
6 POC-BastionHost xxxxx 99.49152542 ```
Below is logstash conf file
```[root@xxx]#
[root@xxxx]# cat /etc/logstash/conf.d/input.conf
input
{
file
{
path => "/home/xxx/aws/result/MAXCPU.csv"
path => "/home/xxx/aws/result/MINCPU.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
[root@xxxx conf.d]# cat /etc/logstash/conf.d/output.conf
output {
elasticsearch {
hosts => "xxxxx:9200"
index => "logstash-accesslog"
user => "xxxxx"
password => "xxxxx"
}
stdout {}
}
[root@xxxx conf.d]# cat /etc/logstash/conf.d/filter.conf
filter {
csv {
separator => ","
columns => [ "Sl.No" , "InstanceName" , "InstanceID" , "MAXCPU%" , "MINCPU%" ]
}
date {
match => [ "InstanceID" , "UNIX" ]
target => "EventTime"
}
mutate {convert => ["InstanceID", "integer"]}
mutate {convert => ["InstanceName", "integer"]}
}
[root@xxxx conf.d]#
`
PFB logs "/var/log/logstash/logstash-plain.log" ```
``` [2020-08-05T18:16:00,547][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@xxxxxx:9200/"}
[2020-08-05T18:16:00,759][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-08-05T18:16:00,769][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-08-05T18:16:00,912][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//xxxxxx:9200"]}
[2020-08-05T18:16:01,021][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-08-05T18:16:01,128][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/conf.d/filter.conf", "/etc/logstash/conf.d/input.conf", "/etc/logstash/conf.d/output.conf"], :thread=>"#<Thread:0x5d50340e run>"}
[2020-08-05T18:16:01,335][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-08-05T18:16:03,018][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-08-05T18:16:03,118][INFO ][filewatch.observingtail ][main][c0ff6327f421f602f1ac30374c6dda8bd9a4bedc093fad223e95c44720012e99] START, creating Discoverer, Watch with file and sincedb collections
[2020-08-05T18:16:03,116][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-08-05T18:16:03,686][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[root@xxxxxx conf.d]#```
Also PFB output from Kibana Dev tools
```GET /logstash-accesslog/_search
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}```
Please check and help me to get rid over from this issue.
``` elasticsearch.x86_64 7.8.1-1 @elastic-7.x
kibana.x86_64 7.8.1-1 @elastic-7.x
logstash.noarch 1:7.8.1-1 @elastic-7.x```
machine OS : CentOS Linux release 7.8.2003 (Core)Hi,
First could you format your message correctly because it is diffcult to read.
My first remark is that you wrote 3 conf files:
- input.conf
- filter.conf
- output.conf
I suggest you to prefix their name with number:
- 01-input.conf
- 02-filter.conf
- 03-output.conf
Because Logstash loads them in memory in an alfanumeric order so your filter section is currently located before your input section.
2/ Your separator is not ",", change it to spaces or tabulation
separator => ","
3/ You maybe have to skip_header
4/ You try to transform 'InstanceID' into a date (EventTime), but I'm not sure that field is a date
5/ You try to convert 'InstanceName' to an integer, but in your input file it seems to be a string
Even if your filters do not work, the file input should be generating events and they should be written to elasticsearch as documents.
I suggest you enabled log.level debug, or even trace, which should show you the events being flushed into the pipeline.
Hi,
`Thanks for your reply, really appreciate your response :slightly_smiling_face:`
`As per your suggestion i have renamed the input, output and filter files.`
`[root@xxxx conf.d]# pwd ; ls`
` /etc/logstash/conf.d`
`01-input.conf 02-filter.conf 03-output.conf`
` [root@xxxx conf.d]#`
`PFB is screenshot of my sample csv file : !`[sample_CSV|490x166]`(upload://xRtOJVkzjNvLdK8sAUANZU1a7TT.png)`
`Please do refer it.`
`PFB 02-filter.conf file and do let me know what exactly i have to modify in it or else` `please paste correct filter file further in comment section so that I can put same` `configuration on my ELK stack.`
` [root@xxx conf.d]# cat 02-filter.conf`
` filter {`
`csv {`
` separator => ","`
`columns => [ "Sl.No" , "InstanceName" , "InstanceID" , "MAXCPU%" , "MINCPU%" ]`
}
` grok {`
`match => [ "InstanceID" , "UNIX" ]`
` target => "EventTime"`
` }`
`mutate {convert => ["InstanceID", "string"]`}
` mutate {convert => ["InstanceName", "string"]}`
` }`
********** Logstash plain log********
`[2020-08-06T18:06:50,419][INFO ][logstash.outputs.elasticsearch][main] Using default` `mapping template`
[2020-08-06T18:06:50,627][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-08-06T18:06:50,827][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/02-filter.conf", "/etc/logstash/conf.d/03-output.conf"], :thread=>"#<Thread:0x7fa94e9f run>"}
[2020-08-06T18:06:53,437][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-08-06T18:06:53,512][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-08-06T18:06:53,535][INFO ][filewatch.observingtail ][main][3c387366658701f23901c694a7dab7b8fa4b4604a48f33e0822a68e6221a18ba] START, creating Discoverer, Watch with file and sincedb collections
[2020-08-06T18:06:54,100][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
**************
`Note : I am trying to take csv file from server A and move to B (using logstash) server where my complete ELK infra is running. In server A , I have installed elasticsearch , logstash and kibana and in all host settings i have provide host B Ip address.`
*******************
please check all above provided data and help as I am very new in ELK technology.This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.