Logstash throwing error for filter portion when trying to convert csv to elastic search

(Nithin Nk) #1

Fixed so deleting

(Nithin Nk) #3

sorry, This is my config:

input {
file {
path => "/opt/TenantAvailabilityCheckConsolidatedCustomReport.csv"
start_position => "beginning"
ignore_older => 0
sincedb_path => "/persistent/log"
sincedb_write_interval => 10
filter {
csv {
seperator => ","
columns => ["Customer Name", "Tenant ID", "Cluster", "Application ID", "Business Type", "Component Name", "Error Message", "Reported On or Before Date-Time", "Closed On or Before Date-Time", "Current Status" ]
mutate {
convert => { "Reported On or Before Date-Time" => "integer"
"Closed On or Before Date-Time" => "integer"



output {
elasticsearch {
hosts => "localhost"
index => "logs"
document_type => "error_logs"


(Christian Dahlqvist) #4

You seem to be missing a curly brace to close the csv block before the mutate block starts.

(Nithin Nk) #5

Thanks, I done that... Now the error changed to

Unknown setting 'seperator' for csv {:level=>:error}
Error: Something is wrong with your configuration. {:level=>:error}

(Christian Dahlqvist) #6

I think the error message is quite clear. It is separator, not seperator.

(Nithin Nk) #7

Thanks Christian. @Christian_Dahlqvist Now I am able to process the csv. But still I have some issue. say for example
This is my csv file

Customer Name,Tenant ID,Cluster,Application ID,Business Type,Component Name,Error Message,Reported On or Before Date-Time,Closed On or Before Date-Time,Current Status

customerxvz,e0431,GBT132,e0431tmn,Test,CXF-endpoint-IFLMAP-hcisbt,"[CXF][CXF][EXCEPTION]:java.net.SocketTimeoutException: Read timed out (local port 59858 to address (vsa3873176.od.sap.biz), remote port 443 to address",29-4-2018 00:03:54 UTC,29-4-2018 02:03:58 UTC,Completed

But with the above configuration file result I got is:

message:customerxyz,e0431,GBT132,e0431iflmap,Test,Messaging Service (JMS),State Message is returning Null,30-4-2018 07:45:30 UTC,08-5-2018 06:54:28 UTC,Completed Customer Name:customerxyz Tenant ID:e0431 @version:1 @timestamp:May 14th 2018, 18:07:06.805 path:/opt/TenantAvailabilityCheckConsolidatedCustomReport.csv host:devops Cluster:GBT132 Application ID:e0431iflmap Business Type:Test Component Name:Messaging Service (JMS) Error Message:State Message is returning Null Reported On or Before Date-Time:30 Closed On or Before Date-Time:8 Current Status:Completed _id:AWNep4WMau29r0E95zBe _type:error_logs _index:hurray _score:

We can see that these are not the fields expected. Fields mismatch each other. Some fits and some are misplaced. Is there any additional match I have to done?

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.