Issue with message duplication

[mikewitters]

Using the ELK stack (6.2.2) to aggregate our application logs and almost everything is going well after a few bumps and bruises. The only issue that I see now is that the actual log statement the app writes is being duplicated in the message field along with the entire message that is placed in the log file. I've found a bunch of similar posts, but everything I try results in error or a stranger outcome than what I've already gotten.

This filter works, but combines the message with the whole log statement:

grok {
match => { "message" =>  "(?m)\[%{TIMESTAMP_ISO8601:timestamp}\] \[%{HOSTNAME:host}\] \[%{DATA:thread}\] %{LOGLEVEL:logLevel} %{DATA:class}@%{DATA:method}:%{DATA:line} \- %{GREEDYDATA:message}"}
}

resulting in a 'message' field that looks like this:

[2018-06-27 09:32:37.643] [webservicedev.ourplace.com] [http-nio-8581-exec-10] DEBUG c.w.c.s.CachedCompanyDataService@getCompanyByNumber:66 - Got company from DAO - com.win.coserv.model.Company@7aa59c11, Got company from DAO - com.win.coserv.model.Company@7aa59c11

rather than what I want which is just this:

Got company from DAO - com.win.coserv.model.Company@7aa59c11

This doesn't even work:

grok {
match => [ "message", "(?m)\[%{TIMESTAMP_ISO8601:timestamp}\] \[%{HOSTNAME:host}\] \[%{DATA:thread}\] %{LOGLEVEL:logLevel} %{DATA:class}@%{DATA:method}:%{DATA:line} \- %{GREEDYDATA:message}"]
overwrite => ["message"]
}

It fails with:

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after ",

Any assistance appreciated.

Thanks.
Mike

[magnusbaeck (Magnus Bäck)]

You need the overwrite option and the second example you posted looks correct. The syntax error at the very beginning of the file is probably a byte-order mark or some other garbage.

[mikewitters]

As usual, you were correct. I started fresh and used the same config as the second example and it's working now so there must've been some garbage in there. It's weird because I've been using vi the whole time. Anyway, thanks a bunch.

Mike W.