Logstash pipeline not starting

Elastic Stack Logstash
1

Hello,
I have created a rather complex Logstash config using various plugins.

I have noticed that Logstash normally produces the following entries in /var/log/logstash/logstash-plain.log:

[2017-03-24T20:01:05,639][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-03-24T20:01:05,643][INFO ][logstash.pipeline ] Pipeline main started
[2017-03-24T20:01:05,713][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Nevertheless, there are cases where Logstash seems not to be able to start a pipeline and keeps retrying (i.e. logs are starting over).
In those cases the first two entries mentioned above (in bold) are completely missing.

No errors are logged, therefore I cannot understand what the issue mught be.
Any help?

Thank you!

2

You can try running Logstash with --debug. Also, check if there is a stderr. What OS/platform and version is this?

3

Hi! I am using Ubuntu 14.04 version vm and have installed Elasticsearch 5.2.2 and Logstash 5.2.2.
I used https://www.elastic.co/guide/en/logstash/current/installing-logstash.html to install logstash using apt-get.
I have started elasticsearch and logstash as service but when I try to run ./logstash "Pipeline main started" isnt appearing in the console. When I check /var/logs/logstash/logstash.err I am not able to see any logs. My default Logstash directory is not /opt/logstash. It is /usr/share/logstash. Not able to find what am I doing wrong. Sorry but I am new to ELK and I might be missing something obvious. Please help.

4

Hey Rashmi,

You can verify your config as such:
--config.debug
Show the fully compiled configuration as a debug log message (you must also have --log.level=debug enabled). WARNING: The log message will include any password options passed to plugin configs as plaintext, and may result in plaintext passwords appearing in your logs!

Also, your bolded entry shows "pipeline.max_inflight" => 250. I verified your settings at this link and that arguments doesn't seem to be an option. When you specify batch size, that sets the maximum for you : https://www.elastic.co/guide/en/logstash/5.2/running-logstash-command-line.html

Try running:
# <logstash_home>/bin/logstash --debug -f <logstash_home>/conf/ -w 2 -b 125 -u 5

... and see what it tells you. You can add -l <log_file> to capture the output in a log file and just "tail -f" the log file.

Post your output if you continue to have issues or if you remove the pipeline.max_inflight argument from your startup script and it resolves the issue, feel free to post as well :wink:

5

Hey Luis,
That code in bold is not mine. My problem is "pipeline main started" isnt even appearing in console when I try to run the basic logstash pipeline given in https://www.elastic.co/guide/en/logstash/current/first-event.html

6

@suyograo : I'm using Logstash 5.2.1 under CentOS 7. There is no *.err log file generated though; all output is logged under /var/log/logstash/logstash-plain.log

@laangarita : I am worried about the pipeline.max_inflight mentioned in the logs as well. Nevertheless, I am using the default logstash.yml and there is no explicit specification of this flag.

Anyway, I have enabled the debug parameter and logs are getting temporarily stuck at:

[2017-03-27T10:55:29,822][DEBUG][logstash.filters.aggregate] Aggregate register call {:code=>"map['EventMap'] = event.get('Event')"}
[2017-03-27T10:55:29,824][DEBUG][logstash.filters.aggregate] Aggregate register call {:code=>"event.set('Event', (map['EventMap'])"}
[2017-03-27T10:55:29,830][DEBUG][logstash.agent ] Starting puma
[2017-03-27T10:55:29,831][DEBUG][logstash.agent ] Trying to start WebServer {:port=>9600}
[2017-03-27T10:55:29,831][DEBUG][logstash.api.service ] [api-service] start
[2017-03-27T10:55:29,849][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Then, the following error appears and entries are starting over:

[2017-03-27T11:01:59,373][DEBUG][logstash.agent ] Error in reactor loop escaped: Bad file descriptor - Bad file descriptor (Errno::EBADF)
[2017-03-27T11:02:53,728][DEBUG][logstash.runner ] -------- Logstash Settings ( means modified) ---------*
[2017-03-27T11:02:53,729][DEBUG][logstash.runner ] node.name: "some.host.name"
[2017-03-27T11:02:53,729][DEBUG][logstash.runner ] *path.config: "/etc/logstash/conf.d"
[2017-03-27T11:02:53,730][DEBUG][logstash.runner ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")

Finally, note how path.config and path.data are marked as modified during the "start-over".

Any ideas?

7

Hi @suyograo

All I can find for this is a bug reported and getting worked on:
https://github.com/elastic/logstash/issues/5822.

Anyone else have any information on this?

8

Hi @raorashmi,

This pipeline from the link assumes that you have data being sent to your logstash instance to get some sort of output. Have you tried adding the --debg flag to your command? If so, do you get anything?

9

Hi @laangarita

The issue was due to an extra parenthesis in the code part of the Aggregate filter plugin.
I am wondering why I never got a syntax error during startup.

I suggest @raorashmi to also perform a syntax check in any ruby code she might be using.

10

Thanks for the update @g.le. I'll keep this in mind for those who face similar issues in the future.

11

In the meantime, anyone having access could file a report for investigating why the Aggregate filter plugin either does not perform Ruby syntax check or suppresses the error

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.