I'm fairly new to the elastic stack but I've successfully created various sandbox environments (using docker compose) chaining FileBeat, WinLogBeat and PacketBeat to LogStash for example. I've worked a lot with the provided filter plugins in LogStash and have written my own plugin as well.
Anyway, now I have a better feel for the stack I want to understand how testing works, specifically if I have a log which contains data I want to use to exercise my stack (whether it be a HTTP log, a PCAP file or a windows event log for example), is there a way to run the same log/scenario in over and over? In effect I want to test the end to end behaviour of the stack so I can be sure it is doing what I need it to.
With PacketBeat I can startup with a specific PCAP file, so that seems easy on the face of it, however with FileBeat and WinLogBeat I am less sure, because I've had issues with the way the components keep track of the position in the log files. Also in terms of WinLogBeat it seems less obvious how to point the component at a fixed 'test log' as opposed to the standard logs in Windows O/S (I suppose I could point FileBeat at a Windows event log, but that feels slightly wrong, and of course means I'm not testing WinLogBeat).
I'd be grateful if anyone could point me in the right direction,
Filebeat keeps track of file positions in registry folder/file (depending on Filebeat version under data/registry. You can delete this file before starting your test.
In case of Winlogbeat positions are stored in the file set in the option winlogbeat.registry_file. By default it's named .winlogbeat.yml. You can also delete this, so Winlogbeat starts processing events from the beginning.
So in effect you're saying I can reset FileBeat/WinLogBeat so that the log is re-processed from the beginning every time I run my test. I guess that makes sense, even if it is a bit clunky (I had been wondering whether file mode would do the same for FileBeat).
Then I just need some way to verify the outcome matches my expectations. Its a shame there is not something a little more specific, like being able to provide a single 'test' log at startup.
What we do in integration tests of Filebeat and Winlogbeat is that we create a new folder which contains all files required (config, data folder, etc.) and set path.config, path.data and path.home. So we get a clean "environment" everytime a test is running.
May I ask why are you testing such things? Have you looked at our intergation tests yet?
I haven't looked at your integration tests no, in fact I didn't know they existed, so that would be a perfect place to start (it is really what I have been looking for without much luck). Can you tell me where they are please? You're probably going to tell me they're in the github repo I guess
I'm not trying to test the beats component themselves of course, I'm trying to test the logic in those components end to end such as filters and aggregation etc (mainly in LogStash i admit). I should really write specific unit tests for the LogStash config for example but I'm thinking in the short term it is a lot easier to use existing test data we have (logs and pcap files) to test the whole setup end-to-end (packetbeat to logstash to kafka ... might drop the kafka bit for the moment). When I have more time I will try and write more specific tests for the different pipelines/configs I have setup etc.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
