Display complete multiline message

Elastic Stack Kibana
1

I have several multiline messages created by filebeat that are sent directly to Elasticsearch. When I do a discovery in Kibana, it is not showing the complete message, which the users would like to see on the Kibana dashboard search window. Is there a way to increase the size of the message being displayed? These are primarily Java stack trace messages using the multiline filter

multiline:
pattern: '^[[:space:]]+|^Caused by:'
negate: false
match: after

2

Hi Gary,

Would you mind adding a screenshot so I can get a better idea of what the problem looks like to your users?

Thanks,
CJ

3

CJ,

It seems my multiline filter in Filebeat is not working I was looking at the log file for the multiline messages and found that they were not being transferred to Elastisearch. My config file is

filebeat:

List of prospectors to fetch data.

prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
paths:
- /local/gstapps/development/jboss/domain/servers/axeda68dev/log/server.log
- /local/gstapps/development/jboss/domain/servers/axeda68dev/log/customobjects.log
input_type: log
exclude_lines: ["^$"]
multiline:

    pattern: '^[[:space:]]+|^Caused by:'
    negate: false
    match: after

  close_older: 15m

output:

Elasticsearch as output

elasticsearch:
hosts: ["3.28.180.76:9200"]
index: "filebeat-jboss-dev"

A sample from the server.log file is, where the message at 15:20 is tranferred but the message at 15:22 is not.

15:20:30,233 ERROR [com.axeda.drm.config.validation.AbstractCacheClusterValidator] (EHCACHE ClusterValidator Thread) Found cache key 3.20.244.80-e555b44:157c4a58579:-8000:3.20.244.82--8bae67c:157c4a880e3:-8000 when it should have been deleted (invalidated) for validatorType=EHCACHE nodeId='3.20.244.80-e555b44:157c4a58579:-8000'
15:22:47,141 ERROR [com.axeda.drm.services.eventsubscription.AMQServer] (ESS-Periodic-Reaper-%d thread-1) Error acquiring MBean server connection, username: GEHC, password: gehcManager, jmxUri: service:jmx:rmi:///jndi/rmi://usmkersvpvmd2l.am.health.ge.com:3099/jmxrmi,service:jmx:rmi:///jndi/rmi://usmkersvpvmd3l.am.health.ge.com:3099/jmxrmi,service:jmx:rmi:///jndi/rmi://usmkersvpvmd4l.am.health.ge.com:3099/jmxrmi: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi,service:jmx:rmi:///jndi/rmi://usmkersvpvmd3l.am.health.ge.com:3099/jmxrmi,service:jmx:rmi:///jndi/rmi://usmkersvpvmd4l.am.health.ge.com:3099/jmxrmi
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) [rt.jar:1.7.0_75]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:249) [rt.jar:1.7.0_75]
at com.axeda.drm.services.eventsubscription.AMQServer.getMBeanServerConnection(AMQServer.java:164)
at com.axeda.drm.services.eventsubscription.AMQHelper.getBrokerViewMbean(AMQHelper.java:98)
at com.axeda.drm.services.eventsubscription.QueueManagerImpl.getSamePrefixQueues(QueueManagerImpl.java:222)
at com.axeda.drm.services.eventsubscription.AMQConnectionMonitor.reapQueues(AMQConnectionMonitor.java:159)
at com.axeda.drm.services.eventsubscription.PeriodicReaperRunnable.run(PeriodicReaperRunnable.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_75]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [rt.jar:1.7.0_75]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [rt.jar:1.7.0_75]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
Caused by: javax.naming.NameNotFoundException: jmxrmi,service:jmx:rmi:///jndi/rmi://usmkersvpvmd3l.am.health.ge.com:3099/jmxrmi,service:jmx:rmi:///jndi/rmi://usmkersvpvmd4l.am.health.ge.com:3099/jmxrmi
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:116) [rt.jar:1.7.0_75]
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203) [rt.jar:1.7.0_75]
at javax.naming.InitialContext.lookup(InitialContext.java:411) [rt.jar:1.7.0_75]
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929) [rt.jar:1.7.0_75]
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896) [rt.jar:1.7.0_75]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:286) [rt.jar:1.7.0_75]
... 13 more

4

Hi Gary, I'm afraid I'm not very familiar with Beats. Would you mind re-posting your problem in the Beats forum? https://discuss.elastic.co/c/beats

Thanks,
CJ

5

CJ,

Will gladly do that. Thanks for offering to help.

6

CJ, Now that I have the filebeat creating the proper message, can you tell me if there is a way to display more of the message is the dashboard that has the search window in it? Here is a sample of what is currently being displayed.

JBoss_Server_log.JPG1220×539 101 KB

7

CJ, I found the way to increase the size of the message being displayed by modifying the truncate:maxHeight and setting it to 0 to be able to display the entire message.

1 Like
8

Fantastic! I'm glad you found the solution to your problem. Thank you for posting the solution here, too.

Best,
CJ