Parsing multiline Tomcat logs

Elastic Stack Logstash
1

Hi,
I'm trying to parse some Tomcat logs with logstash, here is an example of data :

Summary 
25-04-19 10:16:00 WARN  [BasicResourcePool] com.mchange.v2.resourcepool.BasicResourcePool@74f26dc -- Acquisition Attempt Failedclear! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30).
26-04-19 11:38:00 WARN  [BasicResourcePool] com.mchange.v2.resourcepool.BasicResourcePool@74f26dc -- Acquisition Attempt Failedclear! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: TEST STACKTRACE BIS
org.postgresql.util.PSQLException: Connection to ii refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:265)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
        at org.postgresql.Driver.makeConnection(Driver.java:431)
        at org.postgresql.Driver.connect(Driver.java:247)
        at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
        at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
        at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
        at com.mchange.v2.resourcepool.BasicResourcePool.access00(BasicResourcePool.java:44)
        at com.mchange.v2.resourcepool.BasicResourcePool.run(BasicResourcePool.java:1870)
        at com.mchange.v2.async.ThreadPoolAsynchronousRunner .run(ThreadPoolAsynchronousRunner.java:696)
Caused by: java.net.ConnectException: Network is unreachable (connect failed)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.postgresql.core.PGStream.<init>(PGStream.java:62)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
        ... 13 more

I have seen on others topics that multilines plugin and patterns are usefull :

Pattern file :

Summary

JAVA_TRACE (^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)

Logstash config file :

Summary 
input {
        tcp{
                port => 12201
        }
        udp{
                port => 12201
        }

stdin{
codec => multiline {
pattern => "^%{DATE_EU}"
negate => "true"
what => "previous"
}
}

}

filter{
        if [type]== "tomcat" {
            grok{
                    patterns_dir => ["/conf/patterns"]
                    match => {message => ["(^%{DATE_EU} %{TIME} %{LOGLEVEL} %{GREEDYDATA}) | (^%{JAVA_TRACE}) "]}
                    tag_on_failure => ["failed"]
            }

    }

}

output{
        elasticsearch{
                hosts => ["elasticsearch:9200"]
                index => "test"
        }
}

I created a grok pattern with a multiline codec, my logs seem to be parsed by the grok, but the multiIine is not working. I suppose that I missed something but i don't know what ... does anyone have an idea ?

Thx for your help.

2

Nobody ? :frowning:

3

I would expect that multiline codec to result in

   "message" => "25-04-19 10:16:00 WARN  [BasicResourcePool] com.mchange.v2.resourcepool.BasicResourcePool@74f26dc -- Acquisition Attempt Failedclear! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30)."

for that input. The remainder of the file is being accumulated in another event and waiting for another line that matches %{DATE_EU} to trigger it being flushed.

You could use the auto_flush_interval to flush the event if no such line occurs after a certain number of seconds.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.