Grok csv filter

Please assist with CSV Filter

i tried using columns and Source nothing is happening to my event.

csv {

    source => "payData.features"
    }

csv {

    columns=> ""
    }

source => "payData.features"

Wrong syntax for nested fields, see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

Thanks , i used the recommended format still no change,


       }
       mutate {

          gsub => [
                "message", "{", "",
                "message", "}", "",
                "message", "\t\t", "," ,
                "message", ":", "=",
                "message", "-", "=",
                "message", "\t", ",",
                "[payData][features]", " ", ""                     ]

        }

    json {
         source => "response"
         }

   csv {

        source => "[payData][features]"

If you want the fields in the event to have the names from your PNG image then you have to tell the CSV filter what the columns are called using the columns setting.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-columns

In the config above you set the columns setting to an empty string but it validates for an Array so I think that you will see at least one validation error in the LS logs.

  csv {
    source => "[payData][features]"
    columns => ["InterAccountTransferGetTargetAccounts", <all the other columns>, "AddPrepaidBeneficiary"]
  }

You might want to read https://www.elastic.co/guide/en/logstash/6.0/data-deserialization.html

Thank you, it parses the CSV , however no output to Kibana
i can see the event in ruby debug

Images of the console output are not helpful. It does not look like csv filter is working.

Please post the console text, your full config and a sample of the raw data here inside two triple backticks (```) lines.

Logstash.conf

    input {
     beats {
        type => beats
        port => 5045

    }
}
filter{

   grok {

match => { "message" => "IN:%{IP:ip}\\t\\t%{SPACE}AccessAccount%{NOTSPACE:accessAccount}%{SPACE}OPCODE-%{NOTSPACE:opcode}%{SPACE}DeviceModel-%{NOTSPACE:deviceModel}%{SPACE}DeviceManufacturer-%{NOTSPACE:deviceManufacturer}%{SPACE}DeviceI$
match => { "message" => "%{DATESTAMP:timestamp}  %{WORD:severity} \[WebContainer : %{DATA:webcontainer}] %{JAVACLASS:cbmg} \[%{NOTSPACE:messagingFilter}] %{GREEDYDATA:request} %{GREEDYDATA:deviceID}&%{GREEDYDATA:imei}&%{GREEDYDATA:opCde$
match => { "message" => "%{DATESTAMP:timestamp}  %{WORD:severity} \[WebContainer : %{DATA:webcontainer}] %{JAVACLASS:cbmg} \[%{NOTSPACE:messagingFilter}] %{WORD:jsonHeader1} %{WORD:jsonHeader2} %{WORD:jsonHeader3}=%{GREEDYDATA:response}$

 remove_field => [ "message" ]
       }
       mutate {

          gsub => [
                "message", "{", "",
                "message", "}", "",
                "message", "\t\t", "," ,
                "message", ":", "=",
                "message", "-", "=",
                "message", "\t", ","
                    ]

        }

    json {
         source => "response"

        }

  csv {
    #source => "[payData][features]"
separator => ","
columns => ["InterAccountTransferGetTargetAccounts","ValidateAndPayAnInterAccountTransfer","InterAccountTransferGetSourceAccounts","GetRecurringPayments","GetRecurringPaymentDetails","RemoveRecurringPayments","AddRecurringPayment","Chan$

     }
    kv{
        source => "response"

        }

   }

  output {
     elasticsearch {
     hosts =>  "localhost:9200"
     #index => "logstash-%{+YYYY.MM.dd}"
     index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
     document_type => "%{[@metadata][type]}"
     manage_template => false

  }

  stdout {
       codec => rubydebug
       }
  }
{
           "severity" => "DEBUG",
        "jsonHeader2" => "JSON",
       "webcontainer" => "10",
        "jsonHeader1" => "Complete",
        "jsonHeader3" => "Response",
             "offset" => 1952,
         "input_type" => "log",
             "source" => "C:\\logstash.log",
               "type" => "log",
               "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
    "messagingFilter" => "MessagingFilter.java:360",
         "@timestamp" => 2017-11-24T09:46:31.385Z,
           "response" => "{\"payHeader\":{\"resId\":null,\"resCde\":\"0000\",\"resMsg\":\"success\",\"serVer\":\"2.0\",\"expTrace\":null},\"payData\":{\"op_gsn\":\"j.intranet.com\",\"op_status\":\"00000\",\"op_msg\":null,\"deviceOSName\":\"IOS\",\"deviceOSVersion\":\"10\",\"currentMajorVersion\":\"1\",\"currentMinorVersion\":\"0\",\"currentPatchVersion\":\"2\",\"features\":[\"InterAccountTransferGetTargetAccounts\",\"ValidateAndPayAnInterAccountTransfer\",\"InterAccountTransferGetSourceAccounts\",\"GetRecurringPayments\",\"GetRecurringPaymentDetails\",\"RemoveRecurringPayments\",\"AddRecurringPayment\",\"ChangeRecurringPayment\",\"BeneficiaryPaymentGetSourceAccounts\",\"RemoveImageGoal\",\"GetImageByGoalId\",\"UploadGoalImage\",\"GetPrepaidSourceAccounts\",\"EnquireAdultToChildBeneficiaryDetails\",\"ChangeAdultToChildCellPhoneDetails\",\"GetAccountDetails\",\"ResetCredentials\",\"CreateSavingGoal\",\"AddBeneficiary\",\"AddChildDetails\",\"EnquireChildDetailsById\",\"AssignChoreToChild\",\"GetAllMySavingGoals\",\"DeleteSavingGoal\",\"AddChildConsent\",\"BeneficiaryList\",\"ValidatePrepaid\",\"ListPrepaidBeneficiaries\",\"PrepaidServiceProvider\",\"PerformPrepaidAirtimePayment\",\"ValidatePayment\",\"InsertPushNotiification\",\"DeletePushNotiification\",\"GetSavingsAccountDetails\",\"GetChoresByStatus\",\"UpdateYouthChoreStatus\",\"GetChoresAssignedByAdult\",\"GetAllChildSavingGoals\",\"TransactionHistory\",\"GetChoresAssignedToChild\",\"UpdateSavingGoals\",\"UpdateYouthChore\",\"UpdateYouthProfileForReRegistration\",\"GetLimits\",\"GetYouthProfileCellNumber\",\"PerformPayment\",\"BalanceEnquiry\",\"ValidatePerformCashsend\",\"UpdateClientAcceptanceStatusOnBehalfOfTheChild\",\"GetPowerOfAttorneyForChild\",\"ChangeCredentials\",\"UpdateYouthProfileStatus\",\"BalanceLimits\",\"PerformPrepaidDataPayment\",\"AddPrepaidBeneficiary\"],\"configHashValue\":\"-502009116\",\"upgradeManagement\":{\"forceUpgradeRequired\":false,\"softUpgradeRequired\":false,\"message\":null}}}",
               "cbmg" => "c.b.m.g.i.f.MessagingFilter",
           "@version" => "1",
               "beat" => {
            "name" => "PC",
        "hostname" => "PC",
         "version" => "5.6.3"
    },
               "host" => "PC",
          "payHeader" => {
          "resCde" => "0000",
          "serVer" => "2.0",
          "resMsg" => "success",
           "resId" => nil,
        "expTrace" => nil
    },
            "payData" => {
                   "features" => [
            [ 0] "InterAccountTransferGetTargetAccounts",
            [ 1] "ValidateAndPayAnInterAccountTransfer",
            [ 2] "InterAccountTransferGetSourceAccounts",
            [ 3] "GetRecurringPayments",
            [ 4] "GetRecurringPaymentDetails",
            [ 5] "RemoveRecurringPayments",
            [ 6] "AddRecurringPayment",
            [ 7] "ChangeRecurringPayment",
            [ 8] "BeneficiaryPaymentGetSourceAccounts",
            [ 9] "RemoveImageGoal",
            [10] "GetImageByGoalId",
            [11] "UploadGoalImage",
            [12] "GetPrepaidSourceAccounts",
            [13] "EnquireAdultToChildBeneficiaryDetails",
            [14] "ChangeAdultToChildCellPhoneDetails",
            [15] "GetAccountDetails",
            [16] "ResetCredentials",
            [17] "CreateSavingGoal",
            [18] "AddBeneficiary",
            [19] "AddChildDetails",
            [20] "EnquireChildDetailsById",
            [21] "AssignChoreToChild",
            [22] "GetAllMySavingGoals",
            [23] "DeleteSavingGoal",
            [24] "AddChildConsent",
            [25] "BeneficiaryList",
            [26] "ValidatePrepaid",
            [27] "ListPrepaidBeneficiaries",
            [28] "PrepaidServiceProvider",
            [29] "PerformPrepaidAirtimePayment",
            [30] "ValidatePayment",
            [31] "InsertPushNotiification",
            [32] "DeletePushNotiification",
            [33] "GetSavingsAccountDetails",
            [34] "GetChoresByStatus",
            [35] "UpdateYouthChoreStatus",
            [36] "GetChoresAssignedByAdult",
            [37] "GetAllChildSavingGoals",
            [38] "TransactionHistory",
            [39] "GetChoresAssignedToChild",
            [40] "UpdateSavingGoals",
            [41] "UpdateYouthChore",
            [42] "UpdateYouthProfileForReRegistration",
            [43] "GetLimits",
            [44] "GetYouthProfileCellNumber",
            [45] "PerformPayment",
            [46] "BalanceEnquiry",
            [47] "ValidatePerformCashsend",
            [48] "UpdateClientAcceptanceStatusOnBehalfOfTheChild",
            [49] "GetPowerOfAttorneyForChild",
            [50] "ChangeCredentials",
            [51] "UpdateYouthProfileStatus",
            [52] "BalanceLimits",
            [53] "PerformPrepaidDataPayment",
            [54] "AddPrepaidBeneficiary"
        ],
                     "op_msg" => nil,
            "configHashValue" => "-502009116",
                     "op_gsn" => "j.com",
          "upgradeManagement" => {
            "forceUpgradeRequired" => false,
             "softUpgradeRequired" => false,
                         "message" => nil
        },
               "deviceOSName" => "IOS",
        "currentMinorVersion" => "0",
                  "op_status" => "00000",
        "currentMajorVersion" => "1",
            "deviceOSVersion" => "10",
        "currentPatchVersion" => "2"
    },
          "timestamp" => "17-11-20 10:03:24,378"
}
No Available connections
[ERROR] 2017-11-24 04:46:41.361 [monitoring-license-manager] licensemanager - Unable to retrieve license information from license server {:message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError"}
    indent preformatted text by 4 spaces

[2017-11-22T08:04:20,635][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "xpack.security.enabled" hasn't been registered>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:32:in get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:65:inset_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in block in merge'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:133:invalidate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:280:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:232:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `'"]}
indent preformatted text by 4 spaces

Using your suggestion

input {
     beats {
        type => beats
        port => 5045

    }
}
filter{

   grok {

match => { "message" => "IN:%{IP:ip}\\t\\t%{SPACE}AccessAccount%{NOTSPACE:accessAccount}%{SPACE}OPCODE-%{NOTSPACE:opcode}%{SPACE}DeviceModel-%{NOTSPACE:deviceModel}%{SPACE}DeviceManufacturer-%{NOTSPACE:deviceManufacturer}%{SPACE}DeviceI$
match => { "message" => "%{DATESTAMP:timestamp}  %{WORD:severity} \[WebContainer : %{DATA:webcontainer}] %{JAVACLASS:cbmg} \[%{NOTSPACE:messagingFilter}] %{GREEDYDATA:request} %{GREEDYDATA:deviceID}&%{GREEDYDATA:imei}&%{GREEDYDATA:opCde$
match => { "message" => "%{DATESTAMP:timestamp}  %{WORD:severity} \[WebContainer : %{DATA:webcontainer}] %{JAVACLASS:cbmg} \[%{NOTSPACE:messagingFilter}] %{WORD:jsonHeader1} %{WORD:jsonHeader2} %{WORD:jsonHeader3}=%{GREEDYDATA:response}$

 remove_field => [ "message" ]
       }
       mutate {

          gsub => [
                "message", "{", "",
                "message", "}", "",
                "message", "\t\t", "," ,
                "message", ":", "=",
                "message", "-", "=",
                "message", "\t", ","
                    ]

        }

    json {
         source => "response"

        }

  csv {
    source => "[payData][features]"
    columns => ["InterAccountTransferGetTargetAccounts","ValidateAndPayAnInterAccountTransfer","InterAccountTransferGetSourceAccounts","GetRecurringPayments","GetRecurringPaymentDetails","RemoveRecurringPayments","AddRecurringPayment","Chan$

     }
    kv{
        source => "response"

        }

   }

  output {
     elasticsearch {
     hosts =>  "localhost:9200"
     #index => "logstash-%{+YYYY.MM.dd}"
     index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
     document_type => "%{[@metadata][type]}"
     manage_template => false

  }

  stdout {
       codec => rubydebug
       }
  }
  {       "severity" => "DEBUG",
            "jsonHeader2" => "JSON",
           "webcontainer" => "10",
            "jsonHeader1" => "Complete",
            "jsonHeader3" => "Response",
                 "offset" => 5856,
             "input_type" => "log",
                 "source" => "C:\\Users\\ABCN510\\logstash.log",
                   "type" => "log",
                   "tags" => [
            [0] "beats_input_codec_plain_applied",
            [1] "_csvparsefailure"
        ],
        "messagingFilter" => "MessagingFilter.java:360",
             "@timestamp" => 2017-11-24T10:34:01.800Z,

                   "cbmg" => "c.b.m.g.i.f.MessagingFilter",
               "@version" => "1",
                   "beat" => {
                "name" => "PC",
            "hostname" => "PC",
             "version" => "5.6.3"
        }    "host" => "PC",
              "payHeader" => {
              "resCde" => "0000",
              "serVer" => "2.0",
              "resMsg" => "success",
               "resId" => nil,
            "expTrace" => nil
        },
                "payData" => {
                       "features" => [
                [ 0] "InterAccountTransferGetTargetAccounts",
                [ 1] "ValidateAndPayAnInterAccountTransfer",
                [ 2] "InterAccountTransferGetSourceAccounts",
                [ 3] "GetRecurringPayments",
                [ 4] "GetRecurringPaymentDetails",
                [ 5] "RemoveRecurringPayments",
                [ 6] "AddRecurringPayment",
                [ 7] "ChangeRecurringPayment",
                [ 8] "BeneficiaryPaymentGetSourceAccounts",
                [ 9] "RemoveImageGoal",
                [10] "GetImageByGoalId",
                [11] "UploadGoalImage",
                [12] "GetPrepaidSourceAccounts",
                [13] "EnquireAdultToChildBeneficiaryDetails",
                [14] "ChangeAdultToChildCellPhoneDetails",
                [15] "GetAccountDetails",
                [16] "ResetCredentials",
                [17] "CreateSavingGoal",
                [18] "AddBeneficiary",
                [19] "AddChildDetails",
                [20] "EnquireChildDetailsById",
                [21] "AssignChoreToChild",
                [22] "GetAllMySavingGoals",
                [23] "DeleteSavingGoal",
                [24] "AddChildConsent",
                [25] "BeneficiaryList",
                [26] "ValidatePrepaid",
                [27] "ListPrepaidBeneficiaries",
                [28] "PrepaidServiceProvider",
                [29] "PerformPrepaidAirtimePayment",
                [30] "ValidatePayment",
                [31] "InsertPushNotiification",
                [32] "DeletePushNotiification",
                [33] "GetSavingsAccountDetails",
                [34] "GetChoresByStatus",
                [35] "UpdateYouthChoreStatus",
                [36] "GetChoresAssignedByAdult",
                [37] "GetAllChildSavingGoals",
                [38] "TransactionHistory",
                [39] "GetChoresAssignedToChild",
                [40] "UpdateSavingGoals",
                [41] "UpdateYouthChore",
                [42] "UpdateYouthProfileForReRegistration",
                [43] "GetLimits",
                [44] "GetYouthProfileCellNumber",
                [45] "PerformPayment",
                [46] "BalanceEnquiry",
                [47] "ValidatePerformCashsend",
                [48] "UpdateClientAcceptanceStatusOnBehalfOfTheChild",
                [49] "GetPowerOfAttorneyForChild",
                [50] "ChangeCredentials",
                [51] "UpdateYouthProfileStatus",
                [52] "BalanceLimits",
                [53] "PerformPrepaidDataPayment",
                [54] "AddPrepaidBeneficiary"
            ],
                         "op_msg" => nil,
                "configHashValue" => "-502009116",
                         "op_gsn" => "jb.intranet.com",
              "upgradeManagement" => {
                "forceUpgradeRequired" => false,
                 "softUpgradeRequired" => false,
                             "message" => nil
            },
                   "deviceOSName" => "IOS",
            "currentMinorVersion" => "0",
                      "op_status" => "00000",
            "currentMajorVersion" => "1",
                "deviceOSVersion" => "10",
            "currentPatchVersion" => "2"
        },
              "timestamp" => "17-11-20 10:03:24,378"}
    {
        "@timestamp" => 2017-11-24T10:34:01.800Z,
            "offset" => 5903,
          "@version" => "1",
        "input_type" => "log",
              "beat" => {
                "name" => "PC",
            "hostname" => "PC",
             "version" => "5.6.3"},
              "host" => "PC",
            "source" => "C:\\logstash.log",
           "message" => "\"softUpgradeRequired\"=false,\"message\"=null",
              "type" => "log",
              "tags" => [
            [0] "beats_input_codec_plain_applied",
            [1] "_grokparsefailure"
        ]}

Ok I see now. A CSV filter is not appropriate here. The [payData][features] is not a string that the CSV filter can parse because, from the JSON, it is an array of feature strings.

What are you trying to achieve? IOW, why do you need to move these feature strings to another part of the event?

i am trying to index them to the root, so they can be search able

Are they not searchable while still being deep in the event?

Have you asked in the Elasticsearch forum about searching for terms in a child object field value that is an array?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.