Possible to convert exceptions from multiple logs into one single log in elastic?

We are planning to stream Pivotal Cloud Foundry logs to Logstash, however looking into some log examples when an exception is logged, instead of sending one single log, it is sending each line of the exception as one log. Example (content changed for compliance reasons):

2019-02-27T12:02:57.022-05:00 [APP/PROC/WEB/0] [OUT] Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled."}
2019-02-27T12:02:57.024-05:00 [APP/PROC/WEB/0] [OUT] {"timestamp":"2019-02-27 17:02:57.024","thread":"main","level":"ERROR","logger":"org.springframework.boot.SpringApplication:678","app":"references","version":"3.4.1","msg":"run failed"}
2019-02-27T12:02:57.024-05:00 [APP/PROC/WEB/0] [OUT] org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'specialAwesomeFactory' defined in class path resource [config/myConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.sleep.factory]: Factory method 'specialAwesomeFactory' threw exception; nested exception is java.lang.RuntimeException: Error building Factory
2019-02-27T12:02:57.024-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:753)
2019-02-27T12:02:57.024-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(theResolver.java:241)
2019-02-27T12:02:57.024-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractFactory.java:13275)

Since those logs are sent to Logstash one at time, it will (obviously), log multiple logs in Elastic.

Is there a way to consolidate all those exception logs to a single log in Elastic? Honestly, I don't see how, but doesn't hurt asking. We never know...

Thanks,
Rob

Logstash used to have a multiline filter (or was it input) module but you could not use with with more than one CPU core.

Where I work, we do all the multiline parsing pre-Logstash. I have no experience with Pivotal Cloud Foundry... Is there a way to output logs in JSON (single line per event, even stack traces)? That is what we do with e.g. GCP logs and stack traces are handled correctly.

Hi @A_B,

Thanks for your response. Yes, I usually do the same here. I mean, always try having logs pre-formatted before logstash, however in this case, it is a syslog, so PCF is streaming those logs directly to Logstash. Since the exceptions from the example that I provided are sent one at a time to Logstash, it is handled as completely different logs, so I think that a multiline wouldn't help here.

I'm not from the PCF team, but it seems that they are already consolidating logs in JSON format (looking at the other logs coming), but this particular one is an exception in PCF, so it is logged differently (it seems), but it is a good point. I'll check with them if possible.

Thanks!
Rob

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.