Error parsing csv field message

Recently upgraded to LS 5.6.15 and began receiving the error below. Is this a break in this version of LS? Do you see anything wrong with my config? Thanks.

Error message:
[logstash.filters.csv ] Error parsing csv {:field=>"message"

Config file:

           if [message] =~ /^".*"/ {
                csv {
                    columns => ["attack_type","timestamp","dest_ip","dest_port","device_id","geo_location",
                                "http_class_name","ip_address_intelligence","ip_client","ip_with_route_domain",
                                "is_truncated","management_ip_address","Method","policy_apply_date","policy_name",
                                "Protocol","query_string","Request","request_status","response_code",
                                "route_domain","session_id","Severity","sig_ids","sig_names","sig_set_names",
                                "src_port","sub_violations","support_id","unit_hostname","Uri","Username",
                                "violation_details","violation_rating","Violations","virus_name","websocket_direction",
                                "websocket_message_type","x_forwarded_for_header_value","Response"]
                    add_tag => ["waf", "asm"]
                    remove_field => ["message"]
                    # convert => { "dest_port" => "integer", "response_code" => "integer", "src_port" => "integer" }
                }
                grok {
                    #match => [ "timestamp", 'ASM:"%{timestamp}"']
                    # remove_field => ["message" ]
                    remove_tag => [ '_grokparsefailure' ]
                }
            }
            else
            {
                mutate {
                    add_tag => ["malformed"]
                }
            }

    }
    else {
        grok {
            break_on_match => true
            match => [
                "message", "%{SYSLOG5424LINE}",
                "message", "%{SYSLOGLINE}"
            ]
        }

What is the complete error message?

Here you go Badger:
indent preformatted text by 4 spaces
[2019-03-07T10:48:15,976][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"<134>Mar 7 10:48:14 10.84.0.59 hostname.esri.com ASM:"","2019-03-07 10:48:14","10.36.129.8","80","N/A","US","/ASG/esri_ColdFusion_WebHelp_PRD_http.app/esri_ColdFusion_WebHelp_PRD_http_vs","N/A","64.62.252.174","64.62.252.174%0","","10.249.212.138","GET","2018-09-07 10:31:15","/ASG/esri_ColdFusion_WebHelp_PRD_http.app/esri_ColdFusion_WebHelp_PRD_http_vs","HTTP","itemID=6779","GET /arcgisdesktop/9.3/toc.cfm?itemID=6779 HTTP/1.1\nFrom: the.knowledge.ai@gmail.com\nHost: webhelp.xxxx.com\nConnection: Keep-Alive\nUser-Agent: The Knowledge AI\nAccept-Encoding: gzip,deflate\nX-Forwarded-For: 64.62.252.174\n\n","passed","200","0","28ed72ecb67bc5a1","Informational","","","","59804","","17802897188554385068","hostname.xxxx.com","/arcgisdesktop/9.3/toc.cfm","N/A","","0","","N/A","N/A","N/A","64.62.252.174","Only illegal requests are logged"", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
[2019-03-07T10:48:15,977][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"<134>Mar 7 10:48:14 10.84.0.59 hostname.xxxx.com ASM:"","2019-03-07 10:48:14","10.36.129.8","80","N/A","US","/ASG/xxxx_ColdFusion_WebHelp_PRD_http.app/xxxxx_ColdFusion_WebHelp_PRD_http_vs","N/A","64.62.252.174","64.62.252.174%0","","10.249.212.138","GET","2018-09-07 10:31:15","/ASG/esri_ColdFusion_WebHelp_PRD_http.app/esri_ColdFusion_WebHelp_PRD_http_vs","HTTP","ID=6779&TopicName=Address%20Locator%20Properties%20dialog%20box","GET /arcgisdesktop/9.3/head.cfm?ID=6779&TopicName=Address%20Locator%20Properties%20dialog%20box HTTP/1.1\nFrom: the.knowledge.ai@gmail.com\nHost: webhelp.esri.com\nConnection: Keep-Alive\nUser-Agent: The Knowledge AI\nAccept-Encoding: gzip,deflate\nX-Forwarded-For: 64.62.252.174\n\n","passed","200","0","9ab2e74dd3b0cc68","Informational","","","","59804","","17802897188554385060","hostname.esri.com","/arcgisdesktop/9.3/head.cfm","N/A","<?xml version='1.0' encoding='UTF-8'?><BAD_MSG><violation_masks>0000000000000000-00000000000000005cf7d3eb6b0c2fdb-40000000000000005cf7d3eb6b0c2fdb-40000000000000000000008000000000-0000000000000000</violation_masks><viol_index>24</viol_index><viol_name>VIOL_PARAMETER_VALUE_METACHAR</viol_name><parameter_data><value_error/><enforcement_level>global</enforcement_level>VG9waWNOYW1lQWRkcmVzcyBMb2NhdG9yIFByb3BlcnRpZXMgZGlhbG9nIGJveA==</parameter_data><wildcard_entity>*</wildcard_entity>1<language_type>4</language_type><metachar_index>32</metachar_index></BAD_MSG>","1","","N/A","N/A","N/A","64.62.252.174","Only illegal requests are logged"", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
indent preformatted text by 4 spaces

So the first column of the CSV is

<134>Mar 7 10:48:14 10.84.0.59 [hostname.esri.com](http://hostname.esri.com) ASM:""

That's what is causing the CSV::MalformedCSVError: Illegal quoting. Each column should start and end with " if it is quoted. You cannot start quoting a field part way through. I suggest something like

dissect { mapping => { "message" => "%{syslogHeader} ASM:%{csvData}" } }

Thanks Badger, I see that you are recommending using dissect. How could I fix it for now using the CSV filter?

Are you telling me that

<134>Mar 7 10:48:14 10.84.0.59 hostname.esri.com ASM:""

is the attack type? Surely you need to remove that. I was suggesting that you do that using dissect and then feed the csvData field to your existing csv filter.

If that really is the attack type then you can remove the double quotes using

mutate { gsub => [ "message" ' ASM:"",', " ASM:," ] }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.