Logstash pipeline not starting

[g.le]

Hello,
I have created a rather complex Logstash config using various plugins.

I have noticed that Logstash normally produces the following entries in /var/log/logstash/logstash-plain.log:

[2017-03-24T20:01:05,639][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-03-24T20:01:05,643][INFO ][logstash.pipeline ] Pipeline main started
[2017-03-24T20:01:05,713][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Nevertheless, there are cases where Logstash seems not to be able to start a pipeline and keeps retrying (i.e. logs are starting over).
In those cases the first two entries mentioned above (in bold) are completely missing.

No errors are logged, therefore I cannot understand what the issue mught be.
Any help?

Thank you!

[suyograo]

You can try running Logstash with --debug. Also, check if there is a stderr. What OS/platform and version is this?

[raorashmi (Rashmi Rao)]

Hi! I am using Ubuntu 14.04 version vm and have installed Elasticsearch 5.2.2 and Logstash 5.2.2.
I used https://www.elastic.co/guide/en/logstash/current/installing-logstash.html to install logstash using apt-get.
I have started elasticsearch and logstash as service but when I try to run ./logstash "Pipeline main started" isnt appearing in the console. When I check /var/logs/logstash/logstash.err I am not able to see any logs. My default Logstash directory is not /opt/logstash. It is /usr/share/logstash. Not able to find what am I doing wrong. Sorry but I am new to ELK and I might be missing something obvious. Please help.

[laangarita (Luis Angarita)]

Hey Rashmi,

You can verify your config as such:
--config.debug
Show the fully compiled configuration as a debug log message (you must also have --log.level=debug enabled). WARNING: The log message will include any password options passed to plugin configs as plaintext, and may result in plaintext passwords appearing in your logs!

Also, your bolded entry shows "pipeline.max_inflight" => 250. I verified your settings at this link and that arguments doesn't seem to be an option. When you specify batch size, that sets the maximum for you : https://www.elastic.co/guide/en/logstash/5.2/running-logstash-command-line.html

Try running:
# <logstash_home>/bin/logstash --debug -f <logstash_home>/conf/ -w 2 -b 125 -u 5

... and see what it tells you. You can add -l <log_file> to capture the output in a log file and just "tail -f" the log file.

Post your output if you continue to have issues or if you remove the pipeline.max_inflight argument from your startup script and it resolves the issue, feel free to post as well

[raorashmi (Rashmi Rao)]

Hey Luis,
That code in bold is not mine. My problem is "pipeline main started" isnt even appearing in console when I try to run the basic logstash pipeline given in https://www.elastic.co/guide/en/logstash/current/first-event.html

[g.le]

@suyograo : I'm using Logstash 5.2.1 under CentOS 7. There is no *.err log file generated though; all output is logged under /var/log/logstash/logstash-plain.log

@laangarita : I am worried about the pipeline.max_inflight mentioned in the logs as well. Nevertheless, I am using the default logstash.yml and there is no explicit specification of this flag.

Anyway, I have enabled the debug parameter and logs are getting temporarily stuck at:

[2017-03-27T10:55:29,822][DEBUG][logstash.filters.aggregate] Aggregate register call {:code=>"map['EventMap'] = event.get('Event')"}
[2017-03-27T10:55:29,824][DEBUG][logstash.filters.aggregate] Aggregate register call {:code=>"event.set('Event', (map['EventMap'])"}
[2017-03-27T10:55:29,830][DEBUG][logstash.agent ] Starting puma
[2017-03-27T10:55:29,831][DEBUG][logstash.agent ] Trying to start WebServer {:port=>9600}
[2017-03-27T10:55:29,831][DEBUG][logstash.api.service ] [api-service] start
[2017-03-27T10:55:29,849][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Then, the following error appears and entries are starting over:

[2017-03-27T11:01:59,373][DEBUG][logstash.agent ] Error in reactor loop escaped: Bad file descriptor - Bad file descriptor (Errno::EBADF)
[2017-03-27T11:02:53,728][DEBUG][logstash.runner ] -------- Logstash Settings ( means modified) ---------*
[2017-03-27T11:02:53,729][DEBUG][logstash.runner ] node.name: "some.host.name"
[2017-03-27T11:02:53,729][DEBUG][logstash.runner ] *path.config: "/etc/logstash/conf.d"
[2017-03-27T11:02:53,730][DEBUG][logstash.runner ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")

Finally, note how path.config and path.data are marked as modified during the "start-over".

Any ideas?

[laangarita (Luis Angarita)]

Hi @suyograo

All I can find for this is a bug reported and getting worked on:
https://github.com/elastic/logstash/issues/5822.

Anyone else have any information on this?

[laangarita (Luis Angarita)]

Hi @raorashmi,

This pipeline from the link assumes that you have data being sent to your logstash instance to get some sort of output. Have you tried adding the --debg flag to your command? If so, do you get anything?

[g.le]

Hi @laangarita

The issue was due to an extra parenthesis in the code part of the Aggregate filter plugin.
I am wondering why I never got a syntax error during startup.

I suggest @raorashmi to also perform a syntax check in any ruby code she might be using.

[laangarita (Luis Angarita)]

Thanks for the update @g.le. I'll keep this in mind for those who face similar issues in the future.

[g.le]

In the meantime, anyone having access could file a report for investigating why the Aggregate filter plugin either does not perform Ruby syntax check or suppresses the error