Multiline not removing \n


(Lookingcloudy) #1

My log looks like this:

User Name,Transaction,Target,Transaction Time,Total Time
PCGennes,POLICY LEVEL To INTEREST,PCG - TBA,01/09/2015 16:13:02:316 - 01/09/2015 16:13:04:100,1.78
,,,Request Time:,0.39
,,,Server Time:,0.92
,,,Response Time:,0.48
PCGennes,GENERIC To POLICY LEVEL,PCG - TBA,01/09/2015 16:12:33:725 - 01/09/2015 16:12:41:842,8.117
,,,Request Time:,6.518
,,,Server Time:,1.36
,,,Response Time:,0.239
PCGennes,HOME To GENERIC,PCG,01/09/2015 16:11:47:801 - 01/09/2015 16:11:50:723,2.922
,,,Request Time:,0.416
,,,Server Time:,2.141
,,,Response Time:,0.365

Config looks like this:

input {
    stdin {
        codec => multiline {
            pattern => "^,"
            what => "previous"
        }
    }
}
filter {
    csv {
        separator => ","
    }
    if ([column1] == "User Name") {
        drop {}
    }
}
output {
    stdout { codec => json }
}

Here is the output. Notice the \n newline characters are still there, which means the CSV filter does not pick up the columns correctly:

{"@timestamp":"2015-09-04T08:21:06.045Z","message":["PCGennes,POLICY LEVEL To INTEREST,PCG - TBA,01/09/2015 16:13:02:316 - 01/09/2015 16:13:04:100,1.78\n,,,Request Time:,0.39\n,,,Server Time:,0.92\n,,,Response Time:,0.48"],"@version":"1","tags":["multiline"],"host":"bradschulzMBR.local","column1":"PCGennes","column2":"POLICY LEVEL To INTEREST","column3":"PCG - TBA","column4":"01/09/2015 16:13:02:316 - 01/09/2015 16:13:04:100","column5":"1.78"}{"@timestamp":"2015-09-04T08:21:06.046Z","message":["PCGennes,GENERIC To POLICY LEVEL,PCG - TBA,01/09/2015 16:12:33:725 - 01/09/2015 16:12:41:842,8.117\n,,,Request Time:,6.518\n,,,Server Time:,1.36\n,,,Response Time:,0.239"],"@version":"1","tags":["multiline"],"host":"bradschulzMBR.local","column1":"PCGennes","column2":"GENERIC To POLICY LEVEL","column3":"PCG - TBA","column4":"01/09/2015 16:12:33:725 - 01/09/2015 16:12:41:842","column5":"8.117"}

(Lookingcloudy) #2

Curious to know if this is expected behavior (multiline keeping the linefeed). Anyone have experience with this?


(system) #3