Is there a faster way to test logstash configs with stdin?

Elastic Stack Logstash
1

I found this post from Dec 2018, but the automatic reload doesn't work with stdin (at least up to 6.8, which is my limit at the moment)... Any tips on how to test a developing configuration faster? Or we really have to suffer with starting logstash back every time?

2

echo "2019-04-11 14:57:59.11" | /usr/share/logstash/bin/logstash -e 'input { stdin {} } filter { date { match => [ "message", "yyyy-MM-dd HH:mm:ss.SS"] } }'

{
"host" => "localhost",
"@version" => "1",
"@timestamp" => 2019-04-11T19:57:59.110Z,
"message" => "2019-04-11 14:57:59.11"
}

3

I'm running with a config file (Get-Content X | .\logstash -f ../config/logstash.conf), but takes 62 seconds for [2019-09-25T19:49:43,334][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified to show up and logstash start doing it's thing.

4

Why not read data from file as described in this blog post? That allows auto reload to work. With stdin I think there is no way to avoid restarting every time. Another option could be to switch to a TCP input or use a generator input.

1 Like
5

my scenario, at the moment, is to create a pipeline to ingest old IAS/RRAS logs, but I'm still struggling with the filters - so I need to test it over and over and over again... but each time I try, Logstash takes over a minute to start up...

6

The ways I suggested avoids that, assuming you are on a recent version.

1 Like
7

my limit version is 6.8

I've read the article you linked, but didn't understand where does it talk about a different way to ingest the content. File input won't be feasible while I'm still working on the filters, AFAIK it will read files once, and be done with them, and I need it to keep re-reading the same line to check for the desired output

8

Using TCP input and when I've made a change to the configuration I send
an HUP-signal to Logstash which reloads the config within seconds and
then send the log via TCP again is how I do it (which I guess is sort of
what Christian suggested).

I'm unsure how you'd HUP on Windows (which I assume you're using) but
you can make Logstash poll for changes in the config, see
https://www.elastic.co/guide/en/logstash/6.8/reloading-config.html.

BR,

  • Simon
1 Like
9

I think I could manage to find something. But that'll have to wait for the next project (dunno when), since this one is, thankfully, over.

In case anyone want to check out, I needed to ingest some old IAS logs (not NPS format), and this is the end result =D

So, for now, no more suffering waiting 62 seconds for Logstash to boot up =p

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.