# Gap in basic understanding of message processing of Logstash

**URL:** <https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980>\
**Category:** Logstash\
**Created:** [April 9, 2019, 8:09am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980 "2019-04-09T08:09:57Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [April 9, 2019, 8:09am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/1 "2019-04-09T08:09:57Z")

</div>

Hi Guys,

I did a quick training about Logstash on Udemy and I think i forgot about one important thing and didn't took a note. So I could use a brief explanation on what do I do wrong.  
Based on the message content, I wanted to add a field called fingerprint, where value depends on whether message starts with _\<BATCH_. My config looks like this:

```
input {
	file {
		path => "/location/*.xml"
		start_position => "beginning"
	}
}
filter {
	if [message] =~ "<BATCH.*$" {
		mutate {
			add_field => {"fingerprint" => "hoho"}
			}
		} else {
		mutate {
			add_field => {"fingerprint2" => "hihi"}
			}
		}
}
output {
	stdout {
	}
}

```

Here's a sample .xml file:

```
<?xml version="1.0" encoding="UTF-8"?>
<BATCH TIMESTAMP="2019-02-28T12:27:07.798+01:00" SYNTAX_REV="1.3">DATA</BATCH>

```

And the outcome in stdout looks like this:

```
{
"@version" => "1",
"message" => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r",
"@timestamp" => 2019-04-08T13:55:10.975Z,
"path" => "/location/1.xml",
"host" => "host_info"
}
{
"@version" => "1",
"message" => "<BATCH TIMESTAMP="2019-02-28T12:27:07.798+01:00" SYNTAX_REV="1.3">DATA</BATCH>",
"@timestamp" => 2019-04-08T13:55:11.975Z,
"path" => "/location/1.xml",
"host" => "host_info"
}

```

So no new field is added. How should i fix that? Should I first add something to message filed in filter? The output contains field message, so I'm not sure whether this should be done in the first step. Appreciate the help!

---

<div class="post-metadata">

**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\
**Post date:** [April 9, 2019, 8:49am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/2 "2019-04-09T08:49:12Z")

</div>

Try using forward slashes instead of quotes around your regex

`/<BATCH.*$/`

Also, I would avoid using the term _fingerprint_ as there is a logstash plugin called [fingerprint](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html).

---

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [April 9, 2019, 9:01am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/3 "2019-04-09T09:01:19Z")

</div>

Thanks for the tips. I changed the config to:  
`if [message] =~ "/<BATCH.*$/" {`  
but after starting the pipeline, i got an error:

> [ERROR] 2019-04-09 10:59:51.114 [Converge PipelineAction::Create] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"SyntaxError", :message=\>"(eval):80: syntax error, unexpected tGVAR\n if (((event.get("[message]") =~ //\<BATCH._$//))) # if [message] =~ "/\<BATCH._$/"\n", :backtrace=\>["org/jruby/RubyKernel.java:994:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in `block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in `exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:327:in `block in converge\_state'"]}

---

<div class="post-metadata">

**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\
**Post date:** [April 9, 2019, 9:02am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/4 "2019-04-09T09:02:08Z")

</div>

Yes, remove the " entirely

---

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [April 9, 2019, 9:08am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/5 "2019-04-09T09:08:49Z")

</div>

Worked like a charm. Thank you very much.

---

<div class="post-metadata">

**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\
**Post date:** [April 9, 2019, 9:26am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/6 "2019-04-09T09:26:12Z")

</div>

You're welcome!

---

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [April 9, 2019, 9:31am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/7 "2019-04-09T09:31:05Z")

</div>

One more question. If I'm trying to check whether a field contains specific word, I should use =~ "BATCH" (with quotes) but I want want to validate a regex like the one above, I should not use them?

---

<div class="post-metadata">

**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\
**Post date:** [April 9, 2019, 9:55am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/8 "2019-04-09T09:55:52Z")

</div>

`=~` is used when you want to check with regular expression.  
`==` is used when you want to compare something like int, string etc. Strings definitely need quotes around them.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 9, 2019, 11:24am UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/9 "2019-04-09T11:24:48Z")

</div>

> [@saif3r](#):
>
> value depends on whether message starts with _\<BATCH_

There is no implicit anchoring in a regep. So having a trailing .\*$ in it makes no sense. If you care whether it **starts** with that you need to anchor it yourself.

```
if [message] =~ /^<BATCH/ ...

```

---

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [April 9, 2019, 12:36pm UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/10 "2019-04-09T12:36:35Z")

</div>

Thanks @BKG & @Badger.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2019, 12:36pm UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980/11 "2019-05-07T12:36:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.