[ Error parsing CSV ] : :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>

Elastic Stack Logstash
1

Hello,

I am trying to parse some csv logs using logstash and I am getting this error:

[WARN ] 2021-01-27 14:05:31.217 [[main]>worker2] csv - Error parsing csv {:field=>"message", :source=>"\"2020-12-24 00:01:23\", \"293f23a665294a306dc11dcf4924f11eaf888f03e1c7832b6e21158c426fa845\", \"aeb998262d89a558bd758415046bc97a\", \"6df6346dbf5434c7d1f5029ce15e30cb8ee1072d\", \"Cryptolaemus1\", \"emotet_exe_e1_293f23a665294a306dc11dcf4924f11eaf888f03e1c7832b6e21158c426fa845_2020-12-24__000119.exe\", \"dll\", \"application/x-dosexec\", \"Heodo\", \"n/a\", \"18.57\", \"9fea91b52e7c5a38a1d438872edc6c0f\", \"6144:YlB9q3D33LZpU8f7712LIrkmQ8byjBraRQnoI:YlaLZpU8fH12MYr8yraKoI\", \"F934CF01F181C0B2D5AE653E4466D6721B7AB861CF786AC77BE036AE4F216D3DF24342\"\r", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}

An example of my logs:

"2020-11-17 15:20:13", "8d5df0806d59f8ea4e00a5e31de656eea1faa01d5788a93e883f3df8e421e4cd", "371251b366583c76edad12069c5f0e94", "9c297c81debd1c35e44653a3dce76dd714697e7c", "Seifreed", "afcce4c5e6199dfe759d688f637d50e2", "exe", "application/x-dosexec", "n/a", "n/a", "n/a", "87bed5a7cba00c7e1f4015f1bdae2183", "768:ud5u7mNGtyVf/wQGPL4vzZq2o9W7G1xzUw:ud5z/fzGCq2iW7a", "0FC2C072CE8080FFC0CB3472208521CB9B575A72657A6867A750981E7DBCDE0DA7A753"

My logstash configuration:

 csv {
      columns => ["first_seen", "sha256_hash", "md5_hash", "sha1_hash", "reporter", "file_name", "file_type_guess", "mime_type", "signature", "clamav", "vtpercent", "imphash", "ssdeep", "tlsh"]
      separator => ","
    }
    fingerprint {
      concatenate_sources => true
      method => "SHA256"
      source => [ "sha1_hash" ] # And possibly other fields
      target => "[@metadata][fingerprint]"
    }
    mutate {
      remove_field => ["message"]
    }
  }

Could you please help me to figure out how can I solve this error !

Best regards

2

If a field is quoted then the entire field has to be enclosed in double quotes. You cannot have any characters (e.g. a space) before the opening quote. You can use mutate+gsub to replace '", "' with '","'.

1 Like
3

Thank you very much @Badger,

I used gsub and now it's working :star_struck:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.