Multiline Codec: Message mismatch between ES and ruby plugin logger

Hi all,
I'm processing some NMAP XML output using multiline input plugin and XML filter plugin.
And either I'm doing something wrong or an strange behavior is taking place.
In order to check if multiline codec were splinting messages correctly I put

            ruby {
                    code =>  "
                            logger.info('new_msg')
                            logger.info(event.get('message'))
                    "
            }

in my LS configuration.
As a result I get the message not printed completed. Instead of starting as <host starttime="1551275965" endtime="1551275986"><status state="up" reason="reset" reason_ttl="63"/>) I got

but when I go the the index, message is ok.


What I'm doing wrong?
Below my entire LS configuration
Thank you!
Regards
Ana

input {
  file {
    path => "/data/xml_test/xml_sample_3.xml"
    sincedb_path => "/dev/null"
    start_position => "beginning"
        codec => multiline {
      pattern => "^<host starttime="
      auto_flush_interval => 1
      negate => "true"
      what => "previous"
      max_lines => 1000000000
     max_bytes => "50 MiB"

    }
      tags => "xml_test_nmap"
      type => "xml_test_nmap"
  }
}

filter {

                ruby {
                        code =>  "
                                logger.info('new_msg')
                                logger.info(event.get('message'))
                        "
                }

                xml { source => "message"
                        target => poc
                        xpath => ["host/@name", hostname]

                }
                split {
                        field => "[poc][ports][0][port]"
                }

}



output {
         elasticsearch {
        hosts => ["localhost:9200"]
        index => "xml_test_nmap_2"
      }
      stdout { codec => rubydebug }
}

It looks like the default log4j.properties configuration for Logstash only outputs the last 1000 bytes of the given message; IIRC, this was a safety measure to prevent accidental flooding of disk:

appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %-.10000m%n

-- https://github.com/elastic/logstash/blob/master/config/log4j2.properties#L24

1 Like

Hi Ry,
Many thanks! It was that setting
Regards
Ana

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.