Need to send complete xml file from filebeat to logstash in message field

Elastic Stack Beats
1

Hi ,

I am using below filebeat config yaml to send below xml to logstash for filteration. but it is not sending complete xml file. and sometimes the xml being sent is not in correct format and tags are getting closed earlier which is failing logstash parsing
Can you please help what can be corrected here

 filebeat.inputs:
    - type: log
      paths:
      - /var/jenkins_home/jobs/*/branches/*/builds/*/build.xml
    
      multiline.pattern: '^[\S]*<flow-build'
      multiline.negate: true
      multiline.match: after
      multiline.timeout: 30m
      multiline.max_lines: 10000000000
      close_inactive: 8m
    output.logstash.hosts: ["logstash:31100"]

Below is my xml file :

<?xml version="1.1" encoding="UTF-8"?>
<flow-build plugin="workflow-job@1145.v7f2433caa07f">
  <actions>
    <hudson.model.CauseAction>
      <causeBag class="linked-hash-map">
        <entry>
          <hudson.model.Cause_-UserIdCause>
            <userId>503273614</userId>
          </hudson.model.Cause_-UserIdCause>
          <int>1</int>
        </entry>
      </causeBag>
    </hudson.model.CauseAction>
    <jenkins.metrics.impl.TimeInQueueAction plugin="metrics@4.0.2.8">
      <queuingDurationMillis>4</queuingDurationMillis>
      <blockedDurationMillis>0</blockedDurationMillis>
      <buildableDurationMillis>0</buildableDurationMillis>
      <waitingDurationMillis>1</waitingDurationMillis>
    </jenkins.metrics.impl.TimeInQueueAction>
    <jenkins.scm.api.SCMRevisionAction plugin="scm-api@2.6.5">
      <sourceId>ssh-Jenkins-s02</sourceId>
      <revision class="jenkins.plugins.git.GitBranchSCMRevision" plugin="git@4.10.1">
        <head class="jenkins.plugins.git.GitBranchSCMHead">
          <name>master</name>
        </head>
        <hash>84f9c4b7a6c22d59ead4eb4306f4e8def7313e26</hash>
      </revision>
    </jenkins.scm.api.SCMRevisionAction>
    <org.jenkinsci.plugins.workflow.libs.LibrariesAction plugin="workflow-cps-global-lib@552.vd9cc05b8a2e1">
      <libraries>
        <org.jenkinsci.plugins.workflow.libs.LibraryRecord>
          <name>dsl-lib</name>
          <version>v2.3.7</version>
          <variables class="sorted-set">
            <string>jaas_artifactory</string>
            <string>jaas_data</string>
            <string>jaas_email</string>
            <string>jaas_flowdock</string>
          </variables>
          <trusted>true</trusted>
          <changelog>false</changelog>
        </org.jenkinsci.plugins.workflow.libs.LibraryRecord>
      </libraries>
    </org.jenkinsci.plugins.workflow.libs.LibrariesAction>
    <hudson.plugins.git.util.BuildData plugin="git@4.10.1">
      <buildsByBranchName>
        <entry>
          <string>feature.missingSettings</string>
          <hudson.plugins.git.util.Build>
            <marked plugin="git-client@3.10.1">
              <sha1>1870b0637a0937014bd18c205ac821c28b351125</sha1>
              <branches class="singleton-set">
                <hudson.plugins.git.Branch>
                  <sha1 reference="../../../sha1"/>
                  <name>feature.missingSettings</name>
                </hudson.plugins.git.Branch>
              </branches>
            </marked>
            <revision reference="../marked"/>
            <hudsonBuildNumber>31</hudsonBuildNumber>
          </hudson.plugins.git.util.Build>
        </entry>
      </buildsByBranchName>
      <lastBuild reference="../buildsByBranchName/entry/hudson.plugins.git.util.Build"/>
      <remoteUrls>
        <string>git@name.com:ctt-jaas/dsl-lib.git</string>
      </remoteUrls>
    </hudson.plugins.git.util.BuildData>
    <org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState plugin="workflow-scm-step@2.13">
      <revisionStates>
        <entry>
          <string>git git@name.com:ctt-jaas/dsl-lib.git</string>
          <hudson.scm.SCMRevisionState_-None/>
        </entry>
        <entry>
          <string>git git@name.com:CTSD/apis/eatviewer.git</string>
          <hudson.scm.SCMRevisionState_-None reference="../../entry/hudson.scm.SCMRevisionState_-None"/>
        </entry>
      </revisionStates>
    </org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState>
    <hudson.plugins.git.util.BuildData plugin="git@4.10.1">
      <buildsByBranchName>
        <entry>
          <string>post-processing-ehl</string>
          <hudson.plugins.git.util.Build>
            <marked plugin="git-client@3.10.1">
              <sha1>8155bc44c1a0fc63030cda1bce5f13de5fe54660</sha1>
              <branches class="list">
                <hudson.plugins.git.Branch>
                  <sha1 reference="../../../sha1"/>
                  <name>post-processing-ehl</name>
                </hudson.plugins.git.Branch>
              </branches>
            </marked>
            <revision plugin="git-client@3.10.1">
              <sha1 reference="../../marked/sha1"/>
              <branches class="list">
                <hudson.plugins.git.Branch reference="../../../marked/branches/hudson.plugins.git.Branch"/>
              </branches>
            </revision>
            <hudsonBuildNumber>2</hudsonBuildNumber>
          </hudson.plugins.git.util.Build>
        </entry>
        <entry>
          <string>eyal-sonarqube</string>
          <hudson.plugins.git.util.Build>
            <marked plugin="git-client@3.10.1">
              <sha1>8155bc44c1a0fc63030cda1bce5f13de5fe54660</sha1>
              <branches class="singleton-set">
                <hudson.plugins.git.Branch>
                  <sha1 reference="../../../sha1"/>
                  <name>eyal-sonarqube</name>
                </hudson.plugins.git.Branch>
              </branches>
            </marked>
            <revision reference="../marked"/>
            <hudsonBuildNumber>31</hudsonBuildNumber>
          </hudson.plugins.git.util.Build>
        </entry>
      </buildsByBranchName>
      <lastBuild reference="../buildsByBranchName/entry[2]/hudson.plugins.git.util.Build"/>
      <remoteUrls>
        <string>git@name.com:CTDEVOPS/devops-lib.git</string>
      </remoteUrls>
      <index>2</index>
    </hudson.plugins.git.util.BuildData>
    <org.jenkinsci.plugins.pipeline.modeldefinition.actions.ExecutionModelAction plugin="pipeline-model-definition@1.9.3">
      <stagesUUID>1e44a1a1-ba4d-48c0-ab52-8965ec222440</stagesUUID>
      <pipelineDefs>
        <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTPipelineDef plugin="pipeline-model-api@1.9.3">
          <stages>
            <stages>
              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStage>
                <name>Prep</name>
                <branches>
                  <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBranch>
                    <name>default</name>
                    <steps>
                      <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTScriptBlock>
                        <name>script</name>
                        <args class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTNamedArgumentList">
                          <arguments class="linked-hash-map">
                            <entry>
                              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                                <key>scriptBlock</key>
                              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-ConstantValue>
                                <value class="string">echo &quot;====================&gt; Setup stage on ${env.hostname}&quot;
                        updateGitlabCommitStatus(name: STAGE_NAME, state: &apos;running&apos;)
                        // buildMgr.initialize()
                        updateGitlabCommitStatus(name: STAGE_NAME, state: &apos;success&apos;)</value>
                              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-ConstantValue>
                            </entry>
                          </arguments>
                        </args>
                      </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTScriptBlock>
                    </steps>
                  </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBranch>
                </branches>
              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStage>
              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStage>
                <options>
                  <options>
                    <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTOption>
                      <name>skipDefaultCheckout</name>
                      <args/>
                    </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTOption>
                  </options>
                  <inStage>true</inStage>
                </options>
                <name>deploy</name>
                <branches>
                  <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBranch>
                    <name>default</name>
                    <steps>
                      <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTScriptBlock>
                        <name>script</name>
                        <args class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTNamedArgumentList">
                          <arguments class="linked-hash-map">
                            <entry>
                              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                                <key>scriptBlock</key>
                              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-ConstantValue>
                                <value class="string">isRelease = utilities.isReleaseBranch(env.BRANCH_NAME)
                        def artServer = Artifactory.server &apos;hc-us-east-aws-artifactory&apos;
                        artServer.setBypassProxy(true)

                        def buildInfo = Artifactory.newBuildInfo()
                        buildInfo.retention(maxBuilds: 10, maxDays: 14, deleteBuildArtifacts: true, async: true)
                        buildInfo.env.collect()
                        buildInfo.name = &quot;${config.component}-apis&quot;

                        // *********** Upload Artifacts to Artifactory ***************
                        echo &quot;====================&gt; Deploy for ${buildName} on ${env.hostname} &lt;====================&quot;
                        updateGitlabCommitStatus(name: STAGE_NAME, state: &apos;running&apos;)

                        def specs = new java.util.ArrayList();
                        echo &quot;config component: ${config.component}&quot;
                        def fileAndVersion = sh(script:&quot;find ./META-INF/ -type f -name &apos;*.yaml&apos; -print0 -exec sed -n &apos;s/version: //p&apos; &apos;{}&apos; \\; &quot;, returnStdout:true)
                        def apiList = fileAndVersion.split(&quot;\n&quot;)
                        for(apiItem in apiList  ){
                            def apiItemDetails = apiItem.split(&quot;  &quot;)
                            file = apiItemDetails[0]
                            file = file.trim().replaceAll(&quot;\\u0000&quot;,&quot;&quot;)
                            version = apiItemDetails[1]
                            version = version.trim().replaceAll(&quot;\&quot;&quot;, &quot;&quot;)
                            version = version.trim().replaceAll(&quot;\&apos;&quot;, &quot;&quot;)
                            def releaseTarget = &apos;&apos;
                            if(!isRelease){
                                version=env.BRANCH_NAME+&quot;/&quot;+version
                                releaseTarget = &apos;generic-ctprem/revolution/staging/microservices/apis/&apos; + config.component + &apos;/&apos; + version + &apos;/&apos;
                            }else{
                                releaseTarget = &apos;generic-ctprem/revolution/microservices/apis/&apos; + config.component + &apos;/&apos; + version + &apos;/&apos;
                            }
                            specs.add([pattern:  file , target:  releaseTarget, recursive :  &apos;true&apos; , flat:  &apos;true&apos; ])
                        }

                        if(specs.size() &gt; 0) {
                            def uploadSpec = groovy.json.JsonOutput.toJson(specs)
                            uploadSpec = &quot;{ \&quot;files\&quot; : &quot; + uploadSpec + &quot; } &quot;
                            echo &quot;Artifacts Upload Specification : ${uploadSpec}&quot;

                            artServer.upload(uploadSpec, buildInfo)
                            echo &quot;buildInfo: ${buildInfo}&quot;
                            artServer.publishBuildInfo(buildInfo)

                            // upload artifacts to ct build result
                            if(isRelease) {
                                updateGitlabCommitStatus(name: STAGE_NAME, state: &apos;success&apos;)
                            }
                        }</value>
                              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-ConstantValue>
                            </entry>
                          </arguments>
                        </args>
                      </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTScriptBlock>
                    </steps>
                  </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBranch>
                </branches>
              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStage>
            </stages>
            <uuid>1e44a1a1-ba4d-48c0-ab52-8965ec222440</uuid>
          </stages>
          <postBuild>
            <conditions>
              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBuildCondition>
                <condition>always</condition>
                <branch>
                  <name>default</name>
                  <steps>
                    <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStep>
                      <name>echo</name>
                      <args class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTNamedArgumentList">
                        <arguments class="linked-hash-map">
                          <entry>
                            <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                              <key>message</key>
                            </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                            <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-GStringValue>
                              <value class="string">&quot;Cleaning workspace ${env.WORKSPACE} on ${env.hostname}&quot;</value>
                            </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-GStringValue>
                          </entry>
                        </arguments>
                      </args>
                    </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStep>
                    <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStep>
                      <name>cleanWs</name>
                      <args class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTNamedArgumentList">
                        <arguments class="linked-hash-map"/>
                      </args>
                    </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStep>
                  </steps>
                </branch>
              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBuildCondition>
              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBuildCondition>
                <condition>success</condition>
                <branch>
                  <name>default</name>
                  <steps>
                    <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTScriptBlock>
                      <name>script</name>
                      <args class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTNamedArgumentList">
                        <arguments class="linked-hash-map">
                          <entry>
                            <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                              <key>scriptBlock</key>
                            </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                            <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-ConstantValue>
                              <value class="string">echo &apos;====================&gt; POST success!&apos;
                    //jaas_sensor_postjob{ settings = hcddSettings }

                    //buildMgr.postSuccess(config.email)
                    updateGitlabCommitStatus(name: STAGE_NAME, state: &apos;success&apos;)</value>
                            </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-ConstantValue>
                          </entry>
                        </arguments>
                      </args>
                    </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTScriptBlock>
                  </steps>
                </branch>
              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBuildCondition>
              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBuildCondition>
                <condition>failure</condition>
                <branch>
                  <name>default</name>
                  <steps>
                    <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTScriptBlock>
                      <name>script</name>
                      <args class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTNamedArgumentList">
                        <arguments class="linked-hash-map">
                          <entry>
                            <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                              <key>scriptBlock</key>
                            </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKey>
                            <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-ConstantValue>
                              <value class="string">echo &apos;====================&gt; POST failure!&apos;
                    //jaas_sensor_postjob{ settings = hcddSettings }

                    //buildMgr.postFailure(config.email)
                    updateGitlabCommitStatus(name: STAGE_NAME, state: &apos;failed&apos;)</value>
                            </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue_-ConstantValue>
                          </entry>
                        </arguments>
                      </args>
                    </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTScriptBlock>
                  </steps>
                </branch>
              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTBuildCondition>
            </conditions>
          </postBuild>
          <agent>
            <variables class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue$GStringValue">
              <value class="string">${agent_node}</value>
            </variables>
            <agentType>
              <key>label</key>
            </agentType>
          </agent>
          <options>
            <options>
              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTOption>
                <name>timeout</name>
                <args>
                  <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKeyValueOrMethodCallPair>
                    <key>
                      <key>time</key>
                    </key>
                    <value class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue$GStringValue">
                      <value class="string">${pipelineTimeout}</value>
                    </value>
                  </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKeyValueOrMethodCallPair>
                  <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKeyValueOrMethodCallPair>
                    <key>
                      <key>unit</key>
                    </key>
                    <value class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue$ConstantValue">
                      <value class="string">MINUTES</value>
                    </value>
                  </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKeyValueOrMethodCallPair>
                </args>
              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTOption>
              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTOption>
                <name>buildDiscarder</name>
                <args>
                  <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTMethodCall>
                    <name>logRotator</name>
                    <args>
                      <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKeyValueOrMethodCallPair>
                        <key>
                          <key>numToKeepStr</key>
                        </key>
                        <value class="org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue$ConstantValue">
                          <value class="string">3</value>
                        </value>
                      </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTKeyValueOrMethodCallPair>
                    </args>
                  </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTMethodCall>
                </args>
              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTOption>
              <org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTOption>
                <name>timestamps</name>
                <args/>
              </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTOption>
            </options>
            <inStage>false</inStage>
          </options>
        </org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTPipelineDef>
      </pipelineDefs>
    </org.jenkinsci.plugins.pipeline.modeldefinition.actions.ExecutionModelAction>
    <hudson.plugins.git.util.BuildData plugin="git@4.10.1">
      <buildsByBranchName>
        <entry>
          <string>master</string>
          <hudson.plugins.git.util.Build>
            <marked plugin="git-client@3.10.1">
              <sha1>84f9c4b7a6c22d59ead4eb4306f4e8def7313e26</sha1>
              <branches class="singleton-set">
                <hudson.plugins.git.Branch>
                  <sha1 reference="../../../sha1"/>
                  <name>master</name>
                </hudson.plugins.git.Branch>
              </branches>
            </marked>
            <revision reference="../marked"/>
            <hudsonBuildNumber>31</hudsonBuildNumber>
          </hudson.plugins.git.util.Build>
        </entry>
      </buildsByBranchName>
      <lastBuild reference="../buildsByBranchName/entry/hudson.plugins.git.util.Build"/>
      <remoteUrls>
        <string>git@name.com:CTSD/apis/eatviewer.git</string>
      </remoteUrls>
      <index>3</index>
    </hudson.plugins.git.util.BuildData>
    <org.jenkinsci.plugins.workflow.cps.EnvActionImpl plugin="workflow-cps@2648.va9433432b33c">
      <env class="tree-map"/>
    </org.jenkinsci.plugins.workflow.cps.EnvActionImpl>
    <org.jfrog.hudson.BuildInfoResultAction plugin="artifactory@3.14.2">
      <publishedBuildsDetails class="java.util.concurrent.CopyOnWriteArrayList">
        <org.jfrog.hudson.PublishedBuildDetails>
          <artifactoryUrl>https://name.com/artifactory</artifactoryUrl>
          <buildName>eatviewer-service-apis</buildName>
          <buildNumber>31</buildNumber>
          <platformUrl>https://name.com</platformUrl>
          <startedTimeStamp>1649848538977</startedTimeStamp>
        </org.jfrog.hudson.PublishedBuildDetails>
      </publishedBuildsDetails>
      <build class="flow-build" reference="../../.."/>
    </org.jfrog.hudson.BuildInfoResultAction>
    <jenkins.metrics.impl.SubTaskTimeInQueueAction plugin="metrics@4.0.2.8">
      <queuingDurationMillis>23334</queuingDurationMillis>
      <blockedDurationMillis>0</blockedDurationMillis>
      <buildableDurationMillis>23332</buildableDurationMillis>
      <waitingDurationMillis>0</waitingDurationMillis>
      <executingDurationMillis>23930</executingDurationMillis>
      <workUnitCount>1</workUnitCount>
    </jenkins.metrics.impl.SubTaskTimeInQueueAction>
    <com.sonyericsson.rebuild.RebuildAction plugin="rebuild@1.32"/>
  </actions>
  <queueId>105</queueId>
  <timestamp>1649848461367</timestamp>
  <startTime>1649848461397</startTime>
  <result>SUCCESS</result>
  <duration>89440</duration>
  <charset>UTF-8</charset>
  <keepLog>false</keepLog>
  <execution class="org.jenkinsci.plugins.workflow.cps.CpsFlowExecution">
    <result>SUCCESS</result>
    <script>@Library([&apos;dsl-lib&apos;, &apos;devops-lib&apos;]) _
properties([gitLabConnection(&apos;GitSwarmWithAPIToken&apos;)])

echo &quot;executing eatviewer-service apis pipeline..&quot;

apis {
    component   = &apos;eatviewer-service&apos;
    mailingList = &apos;ct.component.od.plat@name.com&apos;
}
</script>
    <loadedScripts class="map"/>
    <durabilityHint>PERFORMANCE_OPTIMIZED</durabilityHint>
    <timings class="map">
      <entry>
        <string>flowNode</string>
        <long>32918812</long>
      </entry>
      <entry>
        <string>classLoad</string>
        <long>1638530250</long>
      </entry>
      <entry>
        <string>run</string>
        <long>6742688989</long>
      </entry>
      <entry>
        <string>parse</string>
        <long>32562919535</long>
      </entry>
    </timings>
    <sandbox>true</sandbox>
    <iota>69</iota>
    <head>1:69</head>
    <done>true</done>
    <resumeBlocked>false</resumeBlocked>
  </execution>
  <completed>true</completed>
  <checkouts class="hudson.util.PersistedList">
    <org.jenkinsci.plugins.workflow.job.WorkflowRun_-SCMCheckout>
      <scm class="hudson.plugins.git.GitSCM" plugin="git@4.10.1">
        <configVersion>2</configVersion>
        <userRemoteConfigs>
          <hudson.plugins.git.UserRemoteConfig>
            <name>origin</name>
            <refspec>+refs/heads/*:refs/remotes/origin/*</refspec>
            <url>git@name.com:ctt-jaas/dsl-lib.git</url>
            <credentialsId>STO</credentialsId>
          </hudson.plugins.git.UserRemoteConfig>
        </userRemoteConfigs>
        <branches class="singleton-list">
          <hudson.plugins.git.BranchSpec>
            <name>feature.missingSettings</name>
          </hudson.plugins.git.BranchSpec>
        </branches>
        <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
        <submoduleCfg class="empty-list"/>
        <extensions>
          <jenkins.plugins.git.GitSCMSourceDefaults>
            <includeTags>false</includeTags>
          </jenkins.plugins.git.GitSCMSourceDefaults>
          <hudson.plugins.git.extensions.impl.BuildChooserSetting>
            <buildChooser class="jenkins.plugins.git.AbstractGitSCMSource$SpecificRevisionBuildChooser">
              <revision reference="../../../../../../../actions/hudson.plugins.git.util.BuildData/buildsByBranchName/entry/hudson.plugins.git.util.Build/marked"/>
            </buildChooser>
          </hudson.plugins.git.extensions.impl.BuildChooserSetting>
        </extensions>
      </scm>
      <node></node>
      <workspace>/var/jenkins_home/workspace/CTSD_apis_eatviewer_master@libs/dsl-lib</workspace>
      <pollingBaseline class="hudson.scm.SCMRevisionState$None" reference="../../../actions/org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState/revisionStates/entry/hudson.scm.SCMRevisionState_-None"/>
    </org.jenkinsci.plugins.workflow.job.WorkflowRun_-SCMCheckout>
    <org.jenkinsci.plugins.workflow.job.WorkflowRun_-SCMCheckout>
      <scm class="hudson.plugins.git.GitSCM" plugin="git@4.10.1">
        <configVersion>2</configVersion>
        <userRemoteConfigs>
          <hudson.plugins.git.UserRemoteConfig>
            <name>origin</name>
            <refspec>+refs/heads/*:refs/remotes/origin/*</refspec>
            <url>git@name.com:CTDEVOPS/devops-lib.git</url>
            <credentialsId>VM</credentialsId>
          </hudson.plugins.git.UserRemoteConfig>
        </userRemoteConfigs>
        <branches class="singleton-list">
          <hudson.plugins.git.BranchSpec>
            <name>eyal-sonarqube</name>
          </hudson.plugins.git.BranchSpec>
        </branches>
        <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
        <submoduleCfg class="empty-list"/>
        <extensions>
          <jenkins.plugins.git.GitSCMSourceDefaults>
            <includeTags>false</includeTags>
          </jenkins.plugins.git.GitSCMSourceDefaults>
          <hudson.plugins.git.extensions.impl.BuildChooserSetting>
            <buildChooser class="jenkins.plugins.git.AbstractGitSCMSource$SpecificRevisionBuildChooser">
              <revision reference="../../../../../../../actions/hudson.plugins.git.util.BuildData[2]/buildsByBranchName/entry[2]/hudson.plugins.git.util.Build/marked"/>
            </buildChooser>
          </hudson.plugins.git.extensions.impl.BuildChooserSetting>
        </extensions>
      </scm>
      <node></node>
      <workspace>/var/jenkins_home/workspace/CTSD_apis_eatviewer_master@libs/devops-lib</workspace>
    </org.jenkinsci.plugins.workflow.job.WorkflowRun_-SCMCheckout>
    <org.jenkinsci.plugins.workflow.job.WorkflowRun_-SCMCheckout>
      <scm class="hudson.plugins.git.GitSCM" plugin="git@4.10.1">
        <configVersion>2</configVersion>
        <userRemoteConfigs>
          <hudson.plugins.git.UserRemoteConfig>
            <name>origin</name>
            <refspec>+refs/heads/*:refs/remotes/origin/*</refspec>
            <url>git@name.com:CTSD/apis/eatviewer.git</url>
            <credentialsId>Jenkins</credentialsId>
          </hudson.plugins.git.UserRemoteConfig>
        </userRemoteConfigs>
        <branches class="singleton-list">
          <hudson.plugins.git.BranchSpec>
            <name>master</name>
          </hudson.plugins.git.BranchSpec>
        </branches>
        <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
        <submoduleCfg class="empty-list"/>
        <extensions>
          <jenkins.plugins.git.GitSCMSourceDefaults>
            <includeTags>false</includeTags>
          </jenkins.plugins.git.GitSCMSourceDefaults>
          <hudson.plugins.git.extensions.impl.BuildChooserSetting>
            <buildChooser class="jenkins.plugins.git.AbstractGitSCMSource$SpecificRevisionBuildChooser">
              <revision reference="../../../../../../../actions/hudson.plugins.git.util.BuildData[3]/buildsByBranchName/entry/hudson.plugins.git.util.Build/marked"/>
            </buildChooser>
          </hudson.plugins.git.extensions.impl.BuildChooserSetting>
        </extensions>
      </scm>
      <node>build-agent-n8m6b</node>
      <workspace>/home/jenkins/agent/workspace/CTSD_apis_eatviewer_master</workspace>
      <pollingBaseline class="hudson.scm.SCMRevisionState$None" reference="../../../actions/org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState/revisionStates/entry/hudson.scm.SCMRevisionState_-None"/>
    </org.jenkinsci.plugins.workflow.job.WorkflowRun_-SCMCheckout>
  </checkouts>
</flow-build>
3

Hi @Rohit_Goel1

That's a rather unusual use of Filebeat, so it might be very hard to get it working reliably. To be honest, I'm not even sure if it is possible to have a whole XML file matched with Regexp.

There are also some other configurations you might need to tune in order to get it fully working, take a look at things like multiline.max_lines and multiline.timeout. Look carefully at all the options listed in our documentation: Manage multiline messages | Filebeat Reference [8.11] | Elastic you might need to try out quite a few of them. At the end of the page there are some tips on how to test your regexp.

Based on your configurations of negate and match, the expected result is:

Consecutive lines that don’t match the pattern are appended to the previous line that does match.

That means you need another <flow-build to indicate the end of the multiline. If you don't have one you can try to set multiline.flush_pattern or just play with the other parameters I mentioned above.

4

Hi @TiagoQueiroz ,

Thanks, yes I tried multiline multiple parameters in filebeat but still xml which is being sent by it is not in proper format and thus failing logstash xml filter to parse the xml in "message" field.

I want to extract only below part of xml as a single event in xml format so that i can fetch different values from it and create seperate fields

Currently filebeat is sending mutiple events when provided below patterns, it is sending all xml lines in mutiple events

multiline.patterns => '^[/s]*<queueId>'
flush pattern => '^[/s]*<KeepLog>'

 <queueId>105</queueId>
  <timestamp>1649848461367</timestamp>
  <startTime>1649848461397</startTime>
  <result>SUCCESS</result>
  <duration>89440</duration>
  <charset>UTF-8</charset>
  <keepLog>false</keepLog>
5

What do you mean by "proper format"? A valid XML?

I am a bit confused here. Are those settings from Logstatsh or Filebeat?

As I mentioned before it seems your multiline configuration does not know when to finish the multiline, using the XML you provided the following configuration worked for me:

filebeat.inputs:
    - type: log
      paths:
        - /home/tiago/sandbox/CommunityRotation/302844/input/*
      multiline.pattern: '^[\S]*<flow-build'
      multiline.flush_pattern: '^[\S]*</flow-build>'
      multiline.negate: true
      multiline.match: after
      multiline.timeout: 50s 
      multiline.max_lines: 10000000000
      close_inactive: 1m

I used a very short multiline.timeout and close_inactive because I was just testing locally on my development environment. Tune them for your environment.

The difference from your initial configuration is the flush_pattern that tells Filebeat when to finish the multiline. Be aware that those regexps do not accept any blank space at the beginning of the lines.

Also bear in mind that the first line of the file:

<?xml version=\"1.1\" encoding=\"UTF-8\"?>

will be sent as a single event because when the first <flow-build is encoutered, everything that was read before and did not start/match a multiline event is sent to the output and then the multiline starts parsing the new event.

What you want to achieve, even though unusual, seems to work fine, you just need to make sure your regexps are correctly configured.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.