Logstash cut line why?

Elastic Stack Logstash
1

i have this secotion of logs

2022-09-28 16:19:49,383 ERROR [it.unidoc.cdr.core.route.Iti41MllpSource] (TcpSocketConsumerRunnable[mllp://0.0.0.0:9010] - /192.168.0.7:56372 => /192.168.0.32:9010) Conversion error: ca.uhn.hl7v2.HL7Exception: java.lang.Exception: Invalid PDF header
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.ProvideAndRegisterBuilder.toDocuments(ProvideAndRegisterBuilder.java:194)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.ProvideAndRegisterBuilder.build(ProvideAndRegisterBuilder.java:70)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.route.Iti41MllpSource.lambda$configure$2(Iti41MllpSource.java:118)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.hl7v2.HL7InternalProcessor.process(HL7InternalProcessor.java:16)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.hl7v2.HL7State.process(HL7State.java:100)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.route.Iti41MllpSource.lambda$configure$3(Iti41MllpSource.java:104)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.support.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:66)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.management.DefaultInstrumentationProcessor.process(DefaultInstrumentationProcessor.java:90)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.doRun(RedeliveryErrorHandler.java:714)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.run(RedeliveryErrorHandler.java:623)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:312)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:41)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.component.mllp.MllpTcpServerConsumer.processMessage(MllpTcpServerConsumer.java:273)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.component.mllp.internal.TcpSocketConsumerRunnable.run(TcpSocketConsumerRunnable.java:141)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.Exception: Invalid PDF header
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.decoder.PdfDecoder.<init>(PdfDecoder.java:21)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.decoder.DecoderFactory.fromRaw(DecoderFactory.java:20)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.hl7v2.MdmDocumentConverter.convert(MdmDocumentConverter.java:173)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.ProvideAndRegisterBuilder.toDocuments(ProvideAndRegisterBuilder.java:192)
	... 22 more

but when discover is cut by line

image
image932×241 34.9 KB

this is cut

at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.ProvideAndRegisterBuilder.toDocuments(ProvideAndRegisterBuilder.java:194)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.ProvideAndRegisterBuilder.build(ProvideAndRegisterBuilder.java:70)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.route.Iti41MllpSource.lambda$configure$2(Iti41MllpSource.java:118)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.hl7v2.HL7InternalProcessor.process(HL7InternalProcessor.java:16)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.hl7v2.HL7State.process(HL7State.java:100)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.route.Iti41MllpSource.lambda$configure$3(Iti41MllpSource.java:104)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.support.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:66)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.management.DefaultInstrumentationProcessor.process(DefaultInstrumentationProcessor.java:90)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.doRun(RedeliveryErrorHandler.java:714)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.run(RedeliveryErrorHandler.java:623)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:312)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:41)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.component.mllp.MllpTcpServerConsumer.processMessage(MllpTcpServerConsumer.java:273)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//org.apache.camel.component.mllp.internal.TcpSocketConsumerRunnable.run(TcpSocketConsumerRunnable.java:141)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.Exception: Invalid PDF header
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.decoder.PdfDecoder.<init>(PdfDecoder.java:21)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.decoder.DecoderFactory.fromRaw(DecoderFactory.java:20)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.hl7v2.MdmDocumentConverter.convert(MdmDocumentConverter.java:173)
	at deployment.cdr-source-1.0.0.18-SNAPSHOT.war//it.unidoc.cdr.core.mllp.ProvideAndRegisterBuilder.toDocuments(ProvideAndRegisterBuilder.java:192)
	... 22 more
2

This is the default behavior, Logstash read the file line by line, if your log is a multline log, which is your case, you need to configure your input to use the multiline codec so your multiline logs you be sent as one log message.

How are you sending your data to logstash? You need to share your configuration.

If you are using the file input you need to configure it to use the multiline codec

Something like this should work:

    file {
        path => "/path/to/your/log.log"
        codec => multiline {
          pattern => "^%{TIMESTAMP_ISO8601} "
          negate => true
          what => "previous"
        }

If you are using filebeat you need to configure it to read multiline files.

3

hello i use this

input {
  file {

    path => [ "/logstash_dir/federico2/p1/*.*" ]
    
    
    start_position => "beginning"

    

     sincedb_path => "/dev/null"

  }
}


output {
 		elasticsearch {
			hosts => ["elkserver:9200"]
			index => "p1"
		}
   
		stdout { codec => rubydebug }
}
4

You need to configure the multiline codec as the example you the previous answer.

Configure it and see if it solves your issue.

5

hello now is ok ?

input {
  file {

    path => [ "/logstash_dir/federico2/FSEBroker/*.*" ]
    
    start_position => "beginning"

    sincedb_path => "/dev/null"

      codec => multiline {
          pattern => "^%{TIMESTAMP_ISO8601} "
          negate => true
          what => "previous"
        }


  }
}


output {
 		elasticsearch {
			hosts => ["elkserver:9200"]
			index => "fssmulti2"
		}
   
		stdout { codec => rubydebug }
}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.