# Filebeat behavior

**URL:** <https://discuss.elastic.co/t/filebeat-behavior/55141>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 10, 2016, 11:10pm UTC](https://discuss.elastic.co/t/filebeat-behavior/55141 "2016-07-10T23:10:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jemli\_Fathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jemli_fathi/32/10714_2.png) [@Jemli\_Fathi](https://discuss.elastic.co/u/Jemli_Fathi)\
**Post date:** [July 10, 2016, 11:10pm UTC](https://discuss.elastic.co/t/filebeat-behavior/55141/1 "2016-07-10T23:10:48Z")

</div>

I have an XML file which is gonna be updated arbitrary by another program by appending new documents each time...  
This file also will be initialized every day by depopulating it.

I configured filebeat to catch every XML document inside this file matching this format `<H_Ticket>...</H_Ticket>` using this configuration:  
\> filebeat:

> ```
> # List of prospectors to fetch data.
> prospectors:
> paths:
> - C:\busesdata\*.xml
> input_type: log
> exclude_lines: ["^.*xml"]
> #ignore_older: 10s
> #close_older: 1h
> document_type: ticket
> scan_frequency: 15s
> multiline:
> pattern: '<H_Ticket'
> negate: true
> match: after
> output:
> ### Logstash as output
> logstash:
> hosts: ["localhost:5044"]
> index: filebeat
> 
> ```

It works very well when adding many XML docs at the end of the file, but it sends an empty event when adding a single document, for example:

`<H_Ticket>ticket1</H_Ticket> <H_Ticket>ticket2</H_Ticket>`

_=\> it works properly_

`<H_Ticket>ticket</H_Ticket>`

_=\> empty event_

- First, Is this behavior is due to a wrong multiline or other miss configuration or what?
- Second, in my case, do I have to use ignore\_older and close\_older params to guarantee a smooth pipeline process or not? if yes how it might be set in my case?

Thank you in advance

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 12, 2016, 8:30am UTC](https://discuss.elastic.co/t/filebeat-behavior/55141/2 "2016-07-12T08:30:25Z")

</div>

Do you have a new line at the end of the single event? I'm somehow surprised that an empty event is sent. Be aware that `multiline.timeout: 5s` will apply for the last event in a file as long as no new event is added.

Are the events appended to the file identical for single or combined events?

What do you mean by "initialized"? Is the same file truncated or deleted and a new one with the same name is created?

---

<div class="post-metadata">

**Author:** ![Jemli\_Fathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jemli_fathi/32/10714_2.png) [@Jemli\_Fathi](https://discuss.elastic.co/u/Jemli_Fathi)\
**Post date:** [July 12, 2016, 5:48pm UTC](https://discuss.elastic.co/t/filebeat-behavior/55141/3 "2016-07-12T17:48:11Z")

</div>

Thank you

I mean by empty event an event generated by filebeat like this: `{}`  
I don't know about `multiline.timeout` option, is this a new configuration option?

Yes, the events are identical, but always the first appended event is parsed as an empty event by filebeat when adding 1 or more events to the file.

I mean by initialized, that the same file will be totally depopulated.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 13, 2016, 6:33pm UTC](https://discuss.elastic.co/t/filebeat-behavior/55141/4 "2016-07-13T18:33:56Z")

</div>

Is the full event just {} or the message? Because what should be always sent is for example the timestamp and some basic beat info.

Here you find all the docs for multiline and also timeout: [https://www.elastic.co/guide/en/beats/filebeat/1.2/configuration-filebeat-options.html#multiline](https://www.elastic.co/guide/en/beats/filebeat/1.2/configuration-filebeat-options.html#multiline) I thought timeout exists since multiline was introduced.

Sorry to ask again, but depopulated = truncated the file = remove all content inside the file?

Can you share 2 full events? That will make it easier to see if there is perhaps something wrong with multiline or exclude\_lines. Did you ever remove exlude\_lines and check if everything works as expected?

---

<div class="post-metadata">

**Author:** ![Jemli\_Fathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jemli_fathi/32/10714_2.png) [@Jemli\_Fathi](https://discuss.elastic.co/u/Jemli_Fathi)\
**Post date:** [July 15, 2016, 10:55pm UTC](https://discuss.elastic.co/t/filebeat-behavior/55141/5 "2016-07-15T22:55:47Z")

</div>

Sorry, I'm an ES newbie, I mean by depopulated, that the program will remove all content inside the XML file.

Following is a sample content from the XML file:

```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?xml-stylesheet href="ticket.xsl" type="text/xsl"?>
<HF_DOCUMENT>
	<H_Ticket>
		<IDH_Ticket>31</IDH_Ticket>
		<CodeBus>186</CodeBus>
		<CodeCh>5531</CodeCh>
		<CodeConv>5531</CodeConv>
		<Codeligne>12</Codeligne>
		<Date>20151217</Date>
		<Heur>1214</Heur>
		<NomFR1>SOUK AHAD</NomFR1>
		<NomFR2>CHOTT MERIEM </NomFR2>
		<Prix>0.8</Prix>
		<IDTicket>31</IDTicket>
		<CodeRoute>107</CodeRoute>
		<origine>01</origine>
		<Distination>09</Distination>
		<Num>1</Num>
		<Ligne>107</Ligne>
		<requisition> </requisition>
		<voyage>0</voyage>
		<faveur> </faveur>
	</H_Ticket>
	<H_Ticket>
		<IDH_Ticket>32</IDH_Ticket>
		<CodeBus>186</CodeBus>
		<CodeCh>5531</CodeCh>
		<CodeConv>5531</CodeConv>
		<Codeligne>12</Codeligne>
		<Date>20151217</Date>
		<Heur>1214</Heur>
		<NomFR1>SOUK AHAD</NomFR1>
		<NomFR2>SOVIVA </NomFR2>
		<Prix>0.66</Prix>
		<IDTicket>32</IDTicket>
		<CodeRoute>107</CodeRoute>
		<origine>01</origine>
		<Distination>07</Distination>
		<Num>2</Num>
		<Ligne>107</Ligne>
		<requisition> </requisition>
		<voyage>0</voyage>
		<faveur> </faveur>
	</H_Ticket>
	<H_Ticket>
		<IDH_Ticket>33</IDH_Ticket>
		<CodeBus>186</CodeBus>
		<CodeCh>5531</CodeCh>
		<CodeConv>5531</CodeConv>
		<Codeligne>12</Codeligne>
		<Date>20151217</Date>
		<Heur>1215</Heur>
		<NomFR1>SOUK AHAD</NomFR1>
		<NomFR2>KANTAOUI </NomFR2>
		<Prix>0.66</Prix>
		<IDTicket>33</IDTicket>
		<CodeRoute>107</CodeRoute>
		<origine>01</origine>
		<Distination>06</Distination>
		<Num>1</Num>
		<Ligne>107</Ligne>
		<requisition> </requisition>
		<voyage>0</voyage>
		<faveur> </faveur>
	</H_Ticket>
</HF_DOCUMENT>
```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 20, 2016, 2:27pm UTC](https://discuss.elastic.co/t/filebeat-behavior/55141/6 "2016-07-20T14:27:46Z")

</div>

Sorry for the late reply. Just to be sure. You don't want the full event in one document which is between `<HF_DOCUMENT` but each sub entry in `<H_Ticket>...`. I assume every even starts like this, so the first three lines and last line should never be sent?

---

<div class="post-metadata">

**Author:** ![DavidL](https://avatars.discourse-cdn.com/v4/letter/d/ecc23a/32.png) [@DavidL](https://discuss.elastic.co/u/DavidL)\
**Post date:** [July 22, 2016, 4:36pm UTC](https://discuss.elastic.co/t/filebeat-behavior/55141/7 "2016-07-22T16:36:18Z")

</div>

I hope your problem is solved, just a suggestion, can we have a more detailed subject line for question topic in the future? so people have a better chance finding what they need when searching, and they don't create duplicate topics.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 31, 2016, 11:11pm UTC](https://discuss.elastic.co/t/filebeat-behavior/55141/8 "2016-07-31T23:11:07Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.