Please assist with CSV Filter
i tried using columns and Source nothing is happening to my event.
csv {
source => "payData.features"
}
csv {
columns=> ""
}
source => "payData.features"
Wrong syntax for nested fields, see Accessing event data and fields | Logstash Reference [8.11] | Elastic.
Thanks , i used the recommended format still no change,
}
mutate {
gsub => [
"message", "{", "",
"message", "}", "",
"message", "\t\t", "," ,
"message", ":", "=",
"message", "-", "=",
"message", "\t", ",",
"[payData][features]", " ", "" ]
}
json {
source => "response"
}
csv {
source => "[payData][features]"If you want the fields in the event to have the names from your PNG image then you have to tell the CSV filter what the columns are called using the columns setting.
In the config above you set the columns setting to an empty string but it validates for an Array so I think that you will see at least one validation error in the LS logs.
csv {
source => "[payData][features]"
columns => ["InterAccountTransferGetTargetAccounts", <all the other columns>, "AddPrepaidBeneficiary"]
}
You might want to read https://www.elastic.co/guide/en/logstash/6.0/data-deserialization.html
Images of the console output are not helpful. It does not look like csv filter is working.
Please post the console text, your full config and a sample of the raw data here inside two triple backticks (```) lines.
Logstash.conf
input {
beats {
type => beats
port => 5045
}
}
filter{
grok {
match => { "message" => "IN:%{IP:ip}\\t\\t%{SPACE}AccessAccount%{NOTSPACE:accessAccount}%{SPACE}OPCODE-%{NOTSPACE:opcode}%{SPACE}DeviceModel-%{NOTSPACE:deviceModel}%{SPACE}DeviceManufacturer-%{NOTSPACE:deviceManufacturer}%{SPACE}DeviceI$
match => { "message" => "%{DATESTAMP:timestamp} %{WORD:severity} \[WebContainer : %{DATA:webcontainer}] %{JAVACLASS:cbmg} \[%{NOTSPACE:messagingFilter}] %{GREEDYDATA:request} %{GREEDYDATA:deviceID}&%{GREEDYDATA:imei}&%{GREEDYDATA:opCde$
match => { "message" => "%{DATESTAMP:timestamp} %{WORD:severity} \[WebContainer : %{DATA:webcontainer}] %{JAVACLASS:cbmg} \[%{NOTSPACE:messagingFilter}] %{WORD:jsonHeader1} %{WORD:jsonHeader2} %{WORD:jsonHeader3}=%{GREEDYDATA:response}$
remove_field => [ "message" ]
}
mutate {
gsub => [
"message", "{", "",
"message", "}", "",
"message", "\t\t", "," ,
"message", ":", "=",
"message", "-", "=",
"message", "\t", ","
]
}
json {
source => "response"
}
csv {
#source => "[payData][features]"
separator => ","
columns => ["InterAccountTransferGetTargetAccounts","ValidateAndPayAnInterAccountTransfer","InterAccountTransferGetSourceAccounts","GetRecurringPayments","GetRecurringPaymentDetails","RemoveRecurringPayments","AddRecurringPayment","Chan$
}
kv{
source => "response"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
#index => "logstash-%{+YYYY.MM.dd}"
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
manage_template => false
}
stdout {
codec => rubydebug
}
}
{
"severity" => "DEBUG",
"jsonHeader2" => "JSON",
"webcontainer" => "10",
"jsonHeader1" => "Complete",
"jsonHeader3" => "Response",
"offset" => 1952,
"input_type" => "log",
"source" => "C:\\logstash.log",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"messagingFilter" => "MessagingFilter.java:360",
"@timestamp" => 2017-11-24T09:46:31.385Z,
"response" => "{\"payHeader\":{\"resId\":null,\"resCde\":\"0000\",\"resMsg\":\"success\",\"serVer\":\"2.0\",\"expTrace\":null},\"payData\":{\"op_gsn\":\"j.intranet.com\",\"op_status\":\"00000\",\"op_msg\":null,\"deviceOSName\":\"IOS\",\"deviceOSVersion\":\"10\",\"currentMajorVersion\":\"1\",\"currentMinorVersion\":\"0\",\"currentPatchVersion\":\"2\",\"features\":[\"InterAccountTransferGetTargetAccounts\",\"ValidateAndPayAnInterAccountTransfer\",\"InterAccountTransferGetSourceAccounts\",\"GetRecurringPayments\",\"GetRecurringPaymentDetails\",\"RemoveRecurringPayments\",\"AddRecurringPayment\",\"ChangeRecurringPayment\",\"BeneficiaryPaymentGetSourceAccounts\",\"RemoveImageGoal\",\"GetImageByGoalId\",\"UploadGoalImage\",\"GetPrepaidSourceAccounts\",\"EnquireAdultToChildBeneficiaryDetails\",\"ChangeAdultToChildCellPhoneDetails\",\"GetAccountDetails\",\"ResetCredentials\",\"CreateSavingGoal\",\"AddBeneficiary\",\"AddChildDetails\",\"EnquireChildDetailsById\",\"AssignChoreToChild\",\"GetAllMySavingGoals\",\"DeleteSavingGoal\",\"AddChildConsent\",\"BeneficiaryList\",\"ValidatePrepaid\",\"ListPrepaidBeneficiaries\",\"PrepaidServiceProvider\",\"PerformPrepaidAirtimePayment\",\"ValidatePayment\",\"InsertPushNotiification\",\"DeletePushNotiification\",\"GetSavingsAccountDetails\",\"GetChoresByStatus\",\"UpdateYouthChoreStatus\",\"GetChoresAssignedByAdult\",\"GetAllChildSavingGoals\",\"TransactionHistory\",\"GetChoresAssignedToChild\",\"UpdateSavingGoals\",\"UpdateYouthChore\",\"UpdateYouthProfileForReRegistration\",\"GetLimits\",\"GetYouthProfileCellNumber\",\"PerformPayment\",\"BalanceEnquiry\",\"ValidatePerformCashsend\",\"UpdateClientAcceptanceStatusOnBehalfOfTheChild\",\"GetPowerOfAttorneyForChild\",\"ChangeCredentials\",\"UpdateYouthProfileStatus\",\"BalanceLimits\",\"PerformPrepaidDataPayment\",\"AddPrepaidBeneficiary\"],\"configHashValue\":\"-502009116\",\"upgradeManagement\":{\"forceUpgradeRequired\":false,\"softUpgradeRequired\":false,\"message\":null}}}",
"cbmg" => "c.b.m.g.i.f.MessagingFilter",
"@version" => "1",
"beat" => {
"name" => "PC",
"hostname" => "PC",
"version" => "5.6.3"
},
"host" => "PC",
"payHeader" => {
"resCde" => "0000",
"serVer" => "2.0",
"resMsg" => "success",
"resId" => nil,
"expTrace" => nil
},
"payData" => {
"features" => [
[ 0] "InterAccountTransferGetTargetAccounts",
[ 1] "ValidateAndPayAnInterAccountTransfer",
[ 2] "InterAccountTransferGetSourceAccounts",
[ 3] "GetRecurringPayments",
[ 4] "GetRecurringPaymentDetails",
[ 5] "RemoveRecurringPayments",
[ 6] "AddRecurringPayment",
[ 7] "ChangeRecurringPayment",
[ 8] "BeneficiaryPaymentGetSourceAccounts",
[ 9] "RemoveImageGoal",
[10] "GetImageByGoalId",
[11] "UploadGoalImage",
[12] "GetPrepaidSourceAccounts",
[13] "EnquireAdultToChildBeneficiaryDetails",
[14] "ChangeAdultToChildCellPhoneDetails",
[15] "GetAccountDetails",
[16] "ResetCredentials",
[17] "CreateSavingGoal",
[18] "AddBeneficiary",
[19] "AddChildDetails",
[20] "EnquireChildDetailsById",
[21] "AssignChoreToChild",
[22] "GetAllMySavingGoals",
[23] "DeleteSavingGoal",
[24] "AddChildConsent",
[25] "BeneficiaryList",
[26] "ValidatePrepaid",
[27] "ListPrepaidBeneficiaries",
[28] "PrepaidServiceProvider",
[29] "PerformPrepaidAirtimePayment",
[30] "ValidatePayment",
[31] "InsertPushNotiification",
[32] "DeletePushNotiification",
[33] "GetSavingsAccountDetails",
[34] "GetChoresByStatus",
[35] "UpdateYouthChoreStatus",
[36] "GetChoresAssignedByAdult",
[37] "GetAllChildSavingGoals",
[38] "TransactionHistory",
[39] "GetChoresAssignedToChild",
[40] "UpdateSavingGoals",
[41] "UpdateYouthChore",
[42] "UpdateYouthProfileForReRegistration",
[43] "GetLimits",
[44] "GetYouthProfileCellNumber",
[45] "PerformPayment",
[46] "BalanceEnquiry",
[47] "ValidatePerformCashsend",
[48] "UpdateClientAcceptanceStatusOnBehalfOfTheChild",
[49] "GetPowerOfAttorneyForChild",
[50] "ChangeCredentials",
[51] "UpdateYouthProfileStatus",
[52] "BalanceLimits",
[53] "PerformPrepaidDataPayment",
[54] "AddPrepaidBeneficiary"
],
"op_msg" => nil,
"configHashValue" => "-502009116",
"op_gsn" => "j.com",
"upgradeManagement" => {
"forceUpgradeRequired" => false,
"softUpgradeRequired" => false,
"message" => nil
},
"deviceOSName" => "IOS",
"currentMinorVersion" => "0",
"op_status" => "00000",
"currentMajorVersion" => "1",
"deviceOSVersion" => "10",
"currentPatchVersion" => "2"
},
"timestamp" => "17-11-20 10:03:24,378"
}
No Available connections
[ERROR] 2017-11-24 04:46:41.361 [monitoring-license-manager] licensemanager - Unable to retrieve license information from license server {:message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError"}
indent preformatted text by 4 spaces[2017-11-22T08:04:20,635][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "xpack.security.enabled" hasn't been registered>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:32:in get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:65:inset_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in block in merge'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:133:invalidate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:280:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:232:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `'"]}
indent preformatted text by 4 spaces
Using your suggestion
input {
beats {
type => beats
port => 5045
}
}
filter{
grok {
match => { "message" => "IN:%{IP:ip}\\t\\t%{SPACE}AccessAccount%{NOTSPACE:accessAccount}%{SPACE}OPCODE-%{NOTSPACE:opcode}%{SPACE}DeviceModel-%{NOTSPACE:deviceModel}%{SPACE}DeviceManufacturer-%{NOTSPACE:deviceManufacturer}%{SPACE}DeviceI$
match => { "message" => "%{DATESTAMP:timestamp} %{WORD:severity} \[WebContainer : %{DATA:webcontainer}] %{JAVACLASS:cbmg} \[%{NOTSPACE:messagingFilter}] %{GREEDYDATA:request} %{GREEDYDATA:deviceID}&%{GREEDYDATA:imei}&%{GREEDYDATA:opCde$
match => { "message" => "%{DATESTAMP:timestamp} %{WORD:severity} \[WebContainer : %{DATA:webcontainer}] %{JAVACLASS:cbmg} \[%{NOTSPACE:messagingFilter}] %{WORD:jsonHeader1} %{WORD:jsonHeader2} %{WORD:jsonHeader3}=%{GREEDYDATA:response}$
remove_field => [ "message" ]
}
mutate {
gsub => [
"message", "{", "",
"message", "}", "",
"message", "\t\t", "," ,
"message", ":", "=",
"message", "-", "=",
"message", "\t", ","
]
}
json {
source => "response"
}
csv {
source => "[payData][features]"
columns => ["InterAccountTransferGetTargetAccounts","ValidateAndPayAnInterAccountTransfer","InterAccountTransferGetSourceAccounts","GetRecurringPayments","GetRecurringPaymentDetails","RemoveRecurringPayments","AddRecurringPayment","Chan$
}
kv{
source => "response"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
#index => "logstash-%{+YYYY.MM.dd}"
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
manage_template => false
}
stdout {
codec => rubydebug
}
} { "severity" => "DEBUG",
"jsonHeader2" => "JSON",
"webcontainer" => "10",
"jsonHeader1" => "Complete",
"jsonHeader3" => "Response",
"offset" => 5856,
"input_type" => "log",
"source" => "C:\\Users\\ABCN510\\logstash.log",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_csvparsefailure"
],
"messagingFilter" => "MessagingFilter.java:360",
"@timestamp" => 2017-11-24T10:34:01.800Z,
"cbmg" => "c.b.m.g.i.f.MessagingFilter",
"@version" => "1",
"beat" => {
"name" => "PC",
"hostname" => "PC",
"version" => "5.6.3"
} "host" => "PC",
"payHeader" => {
"resCde" => "0000",
"serVer" => "2.0",
"resMsg" => "success",
"resId" => nil,
"expTrace" => nil
},
"payData" => {
"features" => [
[ 0] "InterAccountTransferGetTargetAccounts",
[ 1] "ValidateAndPayAnInterAccountTransfer",
[ 2] "InterAccountTransferGetSourceAccounts",
[ 3] "GetRecurringPayments",
[ 4] "GetRecurringPaymentDetails",
[ 5] "RemoveRecurringPayments",
[ 6] "AddRecurringPayment",
[ 7] "ChangeRecurringPayment",
[ 8] "BeneficiaryPaymentGetSourceAccounts",
[ 9] "RemoveImageGoal",
[10] "GetImageByGoalId",
[11] "UploadGoalImage",
[12] "GetPrepaidSourceAccounts",
[13] "EnquireAdultToChildBeneficiaryDetails",
[14] "ChangeAdultToChildCellPhoneDetails",
[15] "GetAccountDetails",
[16] "ResetCredentials",
[17] "CreateSavingGoal",
[18] "AddBeneficiary",
[19] "AddChildDetails",
[20] "EnquireChildDetailsById",
[21] "AssignChoreToChild",
[22] "GetAllMySavingGoals",
[23] "DeleteSavingGoal",
[24] "AddChildConsent",
[25] "BeneficiaryList",
[26] "ValidatePrepaid",
[27] "ListPrepaidBeneficiaries",
[28] "PrepaidServiceProvider",
[29] "PerformPrepaidAirtimePayment",
[30] "ValidatePayment",
[31] "InsertPushNotiification",
[32] "DeletePushNotiification",
[33] "GetSavingsAccountDetails",
[34] "GetChoresByStatus",
[35] "UpdateYouthChoreStatus",
[36] "GetChoresAssignedByAdult",
[37] "GetAllChildSavingGoals",
[38] "TransactionHistory",
[39] "GetChoresAssignedToChild",
[40] "UpdateSavingGoals",
[41] "UpdateYouthChore",
[42] "UpdateYouthProfileForReRegistration",
[43] "GetLimits",
[44] "GetYouthProfileCellNumber",
[45] "PerformPayment",
[46] "BalanceEnquiry",
[47] "ValidatePerformCashsend",
[48] "UpdateClientAcceptanceStatusOnBehalfOfTheChild",
[49] "GetPowerOfAttorneyForChild",
[50] "ChangeCredentials",
[51] "UpdateYouthProfileStatus",
[52] "BalanceLimits",
[53] "PerformPrepaidDataPayment",
[54] "AddPrepaidBeneficiary"
],
"op_msg" => nil,
"configHashValue" => "-502009116",
"op_gsn" => "jb.intranet.com",
"upgradeManagement" => {
"forceUpgradeRequired" => false,
"softUpgradeRequired" => false,
"message" => nil
},
"deviceOSName" => "IOS",
"currentMinorVersion" => "0",
"op_status" => "00000",
"currentMajorVersion" => "1",
"deviceOSVersion" => "10",
"currentPatchVersion" => "2"
},
"timestamp" => "17-11-20 10:03:24,378"}
{
"@timestamp" => 2017-11-24T10:34:01.800Z,
"offset" => 5903,
"@version" => "1",
"input_type" => "log",
"beat" => {
"name" => "PC",
"hostname" => "PC",
"version" => "5.6.3"},
"host" => "PC",
"source" => "C:\\logstash.log",
"message" => "\"softUpgradeRequired\"=false,\"message\"=null",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_grokparsefailure"
]}Ok I see now. A CSV filter is not appropriate here. The [payData][features] is not a string that the CSV filter can parse because, from the JSON, it is an array of feature strings.
What are you trying to achieve? IOW, why do you need to move these feature strings to another part of the event?
i am trying to index them to the root, so they can be search able
Are they not searchable while still being deep in the event?
Have you asked in the Elasticsearch forum about searching for terms in a child object field value that is an array?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.