Malformed Error


(Robert) #1

Can't figure out these errors I've been seeing. Been seeing a lot of these kind errors in my logstash.log file:

"caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2017-01-10T12:36:22-08:00\" is malformed at \"17-01-10T12:36:22-08:00\""}}

Here's my config:

filter {
  if [source] == "/var/log/test.log" {
    multiline {
        pattern => "^(%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp})"
        what => "previous"
        negate=> true
    }

    grok {
      break_on_match => false
      match => [ "message", "(%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp})%{SPACE}%{WORD:severity} \(%{INT:severity_level}\)\:\s(?<vx_tid>[0-9a-zA-Z-]*)\s(HAR = )(?:(?<har_log>[^\t]*))" ]
    }

    date {
      match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ssZZ" ]
      timezone => "America/Los_Angeles"
    }
  }
}

Currently using logstash 2.3.4.


(Magnus Bäck) #2

Please show the full log message. I suspect the error message comes from ES rather than Logstash.


(Robert) #3

{:timestamp=>"2017-01-10T21:19:19.084000+0000", :message=>"Failed action. ", :status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-smith-nginx-2017.01", :_type=>"log", :_routing=>nil}, #<LogStash::Event:0x7e5e344d @metadata_accessors=#<LogStash::Util::Accessors:0x5f6c2c26 @store={"beat"=>"filebeat-smith-nginx", "type"=>"log"}, @lut={"[type]"=>[{"beat"=>"filebeat-smith-nginx", "type"=>"log"}, "type"], "[beat]"=>[{"beat"=>"filebeat-smith-nginx", "type"=>"log"}, "beat"]}>, @cancelled=false, @data={"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}", "offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f", "har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, @metadata={"beat"=>"filebeat-smith-nginx", "type"=>"log"}, @accessors=#<LogStash::Util::Accessors:0x4643fbe0 @store={"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}", "offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f",


(Robert) #4

"har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, @lut={"message"=>[{"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}", "offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f", "har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, "message"], "timestamp"=>[{"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}", "offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f", "har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, "timestamp"], "severity"=>[{"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}", "offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f", "har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, "severity"], "severity_level"=>[{"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}", "offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f", "har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, "severity_level"], "tid"=>[{"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}",


(Robert) #5

"offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f", "har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, "tid"], "har_log"=>[{"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}", "offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f", "har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, "har_log"], "@timestamp"=>[{"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}", "offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f", "har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, "@timestamp"], "[source]"=>[{"@timestamp"=>"2017-01-10T21:19:10.000Z", "beat"=>{"hostname"=>"HOST-123", "name"=>"HOST-123"}, "count"=>1, "input_type"=>"log", "message"=>"2017-01-10T13:19:10-08:00 HEAVYINFO (8): tId-e3789f2da5313c1847d560f7013b3e0f HAR = {\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}", "offset"=>224089479, "source"=>"/var/log/request.log", "type"=>"log", "@version"=>"1", "mid"=>"smith-nginx.stage", "timestamp"=>"2017-01-10T13:19:10-08:00", "severity"=>"HEAVYINFO", "severity_level"=>"8", "tid"=>"tId-e3789f2da5313c1847d560f7013b3e0f", "har_log"=>"{\n "log": {\n "version": "1.2",\n "creator": {\n "name": "Sample Harchive",\n "version": "0.1",\n "comment": "SERVER_NAME: api.sample.local"\n },\n "entries": [\n\n ]\n }\n}"}, "source"]}>>], :response=>{"create"=>{"_index"=>"filebeat-smith-nginx-2017.01", "_type"=>"log", "_id"=>"AVmKP-Ai3MMKPhA1Iw0h", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "2017-01-10T13:19:10-08:00" is malformed at "17-01-10T13:19:10-08:00""}}}}, :level=>:warn}


(Robert) #6

Sorry for the multiple posts, that was one single entry.


(Magnus Bäck) #7

Okay, it's ES that's complaining. What's the mapping of the timestamp field? Use ES's get mapping API.


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.