# When processing stdin, any equivalents to awk's BEGIN and END?

**URL:** <https://discuss.elastic.co/t/when-processing-stdin-any-equivalents-to-awks-begin-and-end/93623>\
**Category:** Logstash\
**Created:** [July 18, 2017, 4:20pm UTC](https://discuss.elastic.co/t/when-processing-stdin-any-equivalents-to-awks-begin-and-end/93623 "2017-07-18T16:20:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![UnitedMarsupials](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/unitedmarsupials/32/19429_2.png) [@UnitedMarsupials](https://discuss.elastic.co/u/UnitedMarsupials)\
**Post date:** [July 18, 2017, 4:20pm UTC](https://discuss.elastic.co/t/when-processing-stdin-any-equivalents-to-awks-begin-and-end/93623/1 "2017-07-18T16:20:52Z")

</div>

`awk` has the special actions triggered by the `BEGIN`ning and `END`ing the processing of `stdin` (or each individual input file).

(There are also variables automatically maintained by the tool (like `NR`).)

Is there any equivalent in Logstash? For example, when working on filters, it is very useful to see the throughput -- events per second -- but using external tools (like `time`(1)) is imprecise, because logstash is very slow to startup.

Can I somehow start a counter, when the first event is read from `stdin` and finish it, when the processing ends?

I suppose, I can use `ruby`-filter's `init`-clause to capture the _beginning_, but how would I report the results at the _end_ -- without reporting them for _every_ event?

---

<div class="post-metadata">

**Author:** ![Andrew\_Cholakian1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_cholakian1/32/3612_2.png) [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)\
**Post date:** [July 21, 2017, 3:56pm UTC](https://discuss.elastic.co/t/when-processing-stdin-any-equivalents-to-awks-begin-and-end/93623/2 "2017-07-21T15:56:07Z")

</div>

This is a great question! I think the best way is to use a config like:

```auto
 bin/logstash -b 500 -w 1 -e 'input { tcp { port => 1234 } } filter { my_filter {} }' > /dev/null

```

then, to benchmark it, after logstash is booted, in another terminal run:

```auto
time nc localhost 1234 < lots_of_test_data.log

```

The idea here is that by using the in-memory queue there won't be much buffering, so for a reasonably sized amount of test data the output of `time` will only be off by whatever logstash has buffered in memory, which is only the value of `-b`. Alternatively you could do use `pv` to measure the rate, though keep in mind you need its `-a` flag to average the rate over the full run, by default its rate is a sliding window IIRC.

---

<div class="post-metadata">

**Author:** ![UnitedMarsupials](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/unitedmarsupials/32/19429_2.png) [@UnitedMarsupials](https://discuss.elastic.co/u/UnitedMarsupials)\
**Post date:** [July 21, 2017, 4:14pm UTC](https://discuss.elastic.co/t/when-processing-stdin-any-equivalents-to-awks-begin-and-end/93623/3 "2017-07-21T16:14:19Z")

</div>

Yes, that would work too, I suppose... But back to my original question, a `ruby`-filter's `init`-clause is the equivalent to awk's `BEGIN`, right? And there is no equivalent to awk's `END`, is there?

BTW, my current performance-meter is thus:

```
filter {
	ruby {
		init => '
			$lines = 0
			StartedAt = Time.now.to_f
		'
		code => '
			$lines += 1
			if $lines % 100 == 0
				spent = Time.now.to_f - StartedAt
				if spent > 0	# Just in case, avoid dividing by zero
					printf("\r%d lines at %.2f lines per second\t", $lines, $lines / spent)
				end
			end
		'
	}
}

```

This is neat and pretty, when running Logstash interactively, but it can not output the final summary -- because there is no `END`-equivalent... Perhaps, there should be?

---

<div class="post-metadata">

**Author:** ![jordansissel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jordansissel/32/44957_2.png) [@jordansissel](https://discuss.elastic.co/u/jordansissel)\
**Post date:** [July 21, 2017, 8:28pm UTC](https://discuss.elastic.co/t/when-processing-stdin-any-equivalents-to-awks-begin-and-end/93623/4 "2017-07-21T20:28:09Z")

</div>

Logstash is intended to be a processing system for an endless stream of data, so in many ways there is no "beginning" and there is no "end".

We made a special case for a few plugins (stdin, for example) that can have an "end" but there is no "end" event from stdin. You could fork or patch the plugin to add this capability if you need it.

---

<div class="post-metadata">

**Author:** ![UnitedMarsupials](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/unitedmarsupials/32/19429_2.png) [@UnitedMarsupials](https://discuss.elastic.co/u/UnitedMarsupials)\
**Post date:** [July 24, 2017, 3:03pm UTC](https://discuss.elastic.co/t/when-processing-stdin-any-equivalents-to-awks-begin-and-end/93623/5 "2017-07-24T15:03:50Z")

</div>

I'd say, `stdin` and `file` inputs could generate a `BEGIN` and `END` events every time they begin and end processing... Maybe, I will create a pull-request... Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 21, 2017, 3:04pm UTC](https://discuss.elastic.co/t/when-processing-stdin-any-equivalents-to-awks-begin-and-end/93623/6 "2017-08-21T15:04:23Z")

</div>

Closed on August 21, 2017, 3:04pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.