# Need to send complete xml file from filebeat to logstash in message field
**URL:** https://discuss.elastic.co/t/need-to-send-complete-xml-file-from-filebeat-to-logstash-in-message-field/302844
**Category:** Beats
**Tags:** filebeat
**Created:** [April 20, 2022, 5:08pm UTC](https://discuss.elastic.co/t/need-to-send-complete-xml-file-from-filebeat-to-logstash-in-message-field/302844 "2022-04-20T17:08:07Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)
#### Post date: [April 20, 2022, 5:08pm UTC](https://discuss.elastic.co/t/need-to-send-complete-xml-file-from-filebeat-to-logstash-in-message-field/302844/1 "2022-04-20T17:08:07Z")
Hi ,
I am using below filebeat config yaml to send below xml to logstash for filteration. but it is not sending complete xml file. and sometimes the xml being sent is not in correct format and tags are getting closed earlier which is failing logstash parsing
Can you please help what can be corrected here
```auto
filebeat.inputs:
- type: log
paths:
- /var/jenkins_home/jobs/*/branches/*/builds/*/build.xml
multiline.pattern: '^[\S]*
503273614
1
4
0
0
1
ssh-Jenkins-s02
master
84f9c4b7a6c22d59ead4eb4306f4e8def7313e26
dsl-lib
v2.3.7
jaas_artifactory
jaas_data
jaas_email
jaas_flowdock
true
false
feature.missingSettings
1870b0637a0937014bd18c205ac821c28b351125
feature.missingSettings
31
git@name.com:ctt-jaas/dsl-lib.git
git git@name.com:ctt-jaas/dsl-lib.git
git git@name.com:CTSD/apis/eatviewer.git
post-processing-ehl
8155bc44c1a0fc63030cda1bce5f13de5fe54660
post-processing-ehl
2
eyal-sonarqube
8155bc44c1a0fc63030cda1bce5f13de5fe54660
eyal-sonarqube
31
git@name.com:CTDEVOPS/devops-lib.git
2
1e44a1a1-ba4d-48c0-ab52-8965ec222440
Prep
default
script
scriptBlock
echo "====================> Setup stage on ${env.hostname}"
updateGitlabCommitStatus(name: STAGE_NAME, state: 'running')
// buildMgr.initialize()
updateGitlabCommitStatus(name: STAGE_NAME, state: 'success')
skipDefaultCheckout
true
deploy
default
script
scriptBlock
isRelease = utilities.isReleaseBranch(env.BRANCH_NAME)
def artServer = Artifactory.server 'hc-us-east-aws-artifactory'
artServer.setBypassProxy(true)
def buildInfo = Artifactory.newBuildInfo()
buildInfo.retention(maxBuilds: 10, maxDays: 14, deleteBuildArtifacts: true, async: true)
buildInfo.env.collect()
buildInfo.name = "${config.component}-apis"
// ***********Upload Artifacts to Artifactory***************
echo "====================> Deploy for ${buildName} on ${env.hostname} <===================="
updateGitlabCommitStatus(name: STAGE_NAME, state: 'running')
def specs = new java.util.ArrayList();
echo "config component: ${config.component}"
def fileAndVersion = sh(script:"find ./META-INF/ -type f -name '*.yaml' -print0 -exec sed -n 's/version: //p' '{}' \\; ", returnStdout:true)
def apiList = fileAndVersion.split("\n")
for(apiItem in apiList ){
def apiItemDetails = apiItem.split(" ")
file = apiItemDetails[0]
file = file.trim().replaceAll("\\u0000","")
version = apiItemDetails[1]
version = version.trim().replaceAll("\"", "")
version = version.trim().replaceAll("\'", "")
def releaseTarget = ''
if(!isRelease){
version=env.BRANCH_NAME+"/"+version
releaseTarget = 'generic-ctprem/revolution/staging/microservices/apis/' + config.component + '/' + version + '/'
}else{
releaseTarget = 'generic-ctprem/revolution/microservices/apis/' + config.component + '/' + version + '/'
}
specs.add([pattern: file , target: releaseTarget, recursive : 'true' , flat: 'true'])
}
if(specs.size() > 0) {
def uploadSpec = groovy.json.JsonOutput.toJson(specs)
uploadSpec = "{ \"files\" : " + uploadSpec + " } "
echo "Artifacts Upload Specification : ${uploadSpec}"
artServer.upload(uploadSpec, buildInfo)
echo "buildInfo: ${buildInfo}"
artServer.publishBuildInfo(buildInfo)
// upload artifacts to ct build result
if(isRelease) {
updateGitlabCommitStatus(name: STAGE_NAME, state: 'success')
}
}
1e44a1a1-ba4d-48c0-ab52-8965ec222440
always
default
echo
message
"Cleaning workspace ${env.WORKSPACE} on ${env.hostname}"
cleanWs
success
default
script
scriptBlock
echo '====================> POST success!'
//jaas_sensor_postjob{ settings = hcddSettings }
//buildMgr.postSuccess(config.email)
updateGitlabCommitStatus(name: STAGE_NAME, state: 'success')
failure
default
script
scriptBlock
echo '====================> POST failure!'
//jaas_sensor_postjob{ settings = hcddSettings }
//buildMgr.postFailure(config.email)
updateGitlabCommitStatus(name: STAGE_NAME, state: 'failed')
${agent_node}
label
timeout
time
${pipelineTimeout}
unit
MINUTES
buildDiscarder
logRotator
numToKeepStr
3
timestamps
false
master
84f9c4b7a6c22d59ead4eb4306f4e8def7313e26
master
31
git@name.com:CTSD/apis/eatviewer.git
3
https://name.com/artifactory
eatviewer-service-apis
31
https://name.com
1649848538977
23334
0
23332
0
23930
1
105
1649848461367
1649848461397
SUCCESS
89440
UTF-8
false
SUCCESS
PERFORMANCE_OPTIMIZED
flowNode
32918812
classLoad
1638530250
run
6742688989
parse
32562919535
true
69
1:69
true
false
true
2
origin
+refs/heads/*:refs/remotes/origin/*
git@name.com:ctt-jaas/dsl-lib.git
STO
feature.missingSettings
false
false
/var/jenkins_home/workspace/CTSD_apis_eatviewer_master@libs/dsl-lib
2
origin
+refs/heads/*:refs/remotes/origin/*
git@name.com:CTDEVOPS/devops-lib.git
VM
eyal-sonarqube
false
false
/var/jenkins_home/workspace/CTSD_apis_eatviewer_master@libs/devops-lib
2
origin
+refs/heads/*:refs/remotes/origin/*
git@name.com:CTSD/apis/eatviewer.git
Jenkins
master
false
false
build-agent-n8m6b
/home/jenkins/agent/workspace/CTSD_apis_eatviewer_master
```
---
### Author:  [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)
#### Post date: [April 21, 2022, 3:35pm UTC](https://discuss.elastic.co/t/need-to-send-complete-xml-file-from-filebeat-to-logstash-in-message-field/302844/3 "2022-04-21T15:35:26Z")
Hi @Rohit_Goel1
That's a rather unusual use of Filebeat, so it might be very hard to get it working reliably. To be honest, I'm not even sure if it is possible to have a whole XML file matched with Regexp.
There are also some other configurations you might need to tune in order to get it fully working, take a look at things like `multiline.max_lines` and `multiline.timeout`. Look carefully at all the options listed in our documentation: [Manage multiline messages | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html) you might need to try out quite a few of them. At the end of the page there are some tips on how to test your regexp.
Based on your configurations of `negate` and `match`, the expected result is:
> Consecutive lines that don’t match the pattern are appended to the previous line that does match.
That means you need another `
### Author:  [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)
#### Post date: [April 22, 2022, 11:17am UTC](https://discuss.elastic.co/t/need-to-send-complete-xml-file-from-filebeat-to-logstash-in-message-field/302844/4 "2022-04-22T11:17:32Z")
Hi @TiagoQueiroz ,
Thanks, yes I tried multiline multiple parameters in filebeat but still xml which is being sent by it is not in proper format and thus failing logstash xml filter to parse the xml in "message" field.
I want to extract only below part of xml as a single event in xml format so that i can fetch different values from it and create seperate fields
Currently filebeat is sending mutiple events when provided below patterns, it is sending all xml lines in mutiple events
```auto
multiline.patterns => '^[/s]*'
flush pattern => '^[/s]*'
```
```auto
105
1649848461367
1649848461397
SUCCESS
89440
UTF-8
false
```
---
### Author:  [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)
#### Post date: [April 25, 2022, 11:55am UTC](https://discuss.elastic.co/t/need-to-send-complete-xml-file-from-filebeat-to-logstash-in-message-field/302844/5 "2022-04-25T11:55:45Z")
> [@Rohit\_Goel1](#):
>
> Thanks, yes I tried multiline multiple parameters in filebeat but still xml which is being sent by it is not in proper format and thus failing logstash xml filter to parse the xml in "message" field.
What do you mean by "proper format"? A valid XML?
> [@Rohit\_Goel1](#):
>
> ```auto
> multiline.patterns => '^[/s]*'
> flush pattern => '^[/s]*'
>
> ```
I am a bit confused here. Are those settings from Logstatsh or Filebeat?
As I mentioned before it seems your multiline configuration does not know when to finish the multiline, using the XML you provided the following configuration worked for me:
```yaml
filebeat.inputs:
- type: log
paths:
- /home/tiago/sandbox/CommunityRotation/302844/input/*
multiline.pattern: '^[\S]*'
multiline.negate: true
multiline.match: after
multiline.timeout: 50s
multiline.max_lines: 10000000000
close_inactive: 1m
```
I used a very short `multiline.timeout` and `close_inactive` because I was just testing locally on my development environment. Tune them for your environment.
The difference from your initial configuration is the `flush_pattern` that tells Filebeat when to finish the multiline. Be aware that those regexps do not accept any blank space at the beginning of the lines.
Also bear in mind that the first line of the file:
```xml
```
will be sent as a single event because when the first `
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 23, 2022, 1:56pm UTC](https://discuss.elastic.co/t/need-to-send-complete-xml-file-from-filebeat-to-logstash-in-message-field/302844/6 "2022-05-23T13:56:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.