Topbeat and logstash error

Elastic Stack Beats
25

I will start with dev-36 and look to get that solved first. Good point about the configuration file formatting, exactly how sensitive are these files to indentation?

YAML files are sensitive to indentation.

  • Opened another terminal to view all processes (ps -A) and could not locate Logstash

That's because they'll show up as "java". Try ps aux | grep logstash instead.

/var/log/logstash/logstash.log did not appear to be updated after the above commands.

Is this the only location for the log file?

The logfile location depends on how Logstash is invoked. Nothing else. If you start Logstash via ./logstash agent -f logstash.conf Logstash won't send anything to that logfile. Everything will go to the terminal.

logstash.conf does not have 81 lines

Do you get this message when starting Logstash via service logstash start? If so, check that you don't have multiple files in /etc/logstash/conf.d. Logstash will read all files and it's possible that the line number counter isn't zeroed for each file, i.e. line 81 could mean the first line of a file that's read right after an 80-line file has been read.

26

Configuration file formats of beats and logstash are very different. Beats uses YAML (which is very sensitive to spaces vs. tabs and indentation) and logstash uses custom file format somewhat inspired by ruby syntax I think.

I have been copying and pasting in-between local editors and web pages as building out our documentation set, entirely possibly bad characters could have got in.

Yeah, for YAML files it's kind of a bad idea. Some editors can display whitespace characters for you to cleanup (some editors can help cleaning up the mess). for example emacs or vim (I think ATOM can do so too).

/etc/init.d/logstash

this file is not the logstash binary, but an init script to initialize and startup logstash

/opt/logstash/bin/logstash

This one is the actual logstash 'binary' (check init script for using /opt/logstash/bin/logstash). Actually it's a shell script itself initialising and starting logstash in JVM.

this command starts logstash in foreground (not returning to command line is to be expected). By pressing '^Z' you didn't kill logstash, but kinda paused it. Doing $ jobs in your shell you will see 'logstash' being stopped, but still active (ports are not closed). Using 'fg' or 'bg' you can continue the paused logstash process. Some bash job control foo. Doing '^C' kills the process and '^Z' just pauses the process.

To be expected. Logstash is still running(paused) in background. The port is still bound by the paused logstash instance.

Do $ jobs to list jobs known to local shell. See @magnusbaeck tip to find your running logstash instance via ps.

See tip by @magnusbaeck

1 Like
27

Gents you've been awesome - that helps clear up a whole bunch of concepts and sysadmin operations.

Think I have found the problem.

dev-36: - had multiple configuration files
/etc/logstash/conf.d/logstash.conf
/etc/logstash/conf.d/logstash.conf.original
/etc/logstash/conf.d/logstash-topbeats.conf
/etc/logstash/conf.d/logstash.syslog.conf

dev-37: - had one configuration file
/etc/logstash/conf.d/logstash.conf

dev-36 - made the configuration files (logstash.conf and topbeat.yml) identical to those on dev-37
dev-36 - logstash still starting and stopping after a few seconds (problem persisting)
dev-36 - removed all other .conf files from /etc/logstash/conf.d
dev-36 - logstash now started and still running after 10 minutes (problem solved, I think)

It would be great to do some sanity tests for the topbeat data collection and storage in Elasticsearch.

The ELK stacks are running on our worker servers (E5v3 16 Core, 1024 GB RAM) which is part of the reason for collecting the performance metrics (next syslogs, and eventually app logs).

We are developing our data processing pipelines and need to evaluate the performance of our internal application and its algorithms (also written in Java).

I guess there are some obvious questions about optimisation of the ELK stack processes for a balance somewhere between performance and reliability.

I noted in some of the other vendor stacks (Influx, TICK, collectd, rsyslog) forums there can be issues with metric data being dropped under certain circumstances (heavy cpu or memory loads).

Welcome and suggestions and guidance on these topics..

But so far it looks like the logstash issue on dev-36 has been solved..

Configuration files:

dev-36:/etc/logstash/conf.d/logstash.conf
input { beats { port => 5044 } } output { elasticsearch { hosts => "localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } }

dev-36/etc/topbeat/topbeat.yml
input: period: 10 procs: [".*"] stats: system: true proc: true filesystem: true cpu_per_core: true output: logstash: hosts: ["127.0.0.1:5044"]

Does the logging to files help address that issue or concern about potential data loss?
Not currently in topbeat.yml
logging: to_files: true files: path: /var/log/topbeat name: topbeat.log rotateeverybytes: 10485760 keepfiles: 7 level: info

Topbeat now starting on both dev-38 and dev-39.

Replaced the contents of the topbeat.yml files on both servers and the service started and seems to running normally. This looks like it may have been a hidden characters issue, was having the same issue on both servers, same changes appears to have fixed the problem on both servers..

Alight, next on the list is syslog (rsyslog) collection by logstash, have searched and read a few posts about the syslog plug-in.

I am keen to get this operational as soon as possible on our servers, the documentation did not seem very detailed, are you able to help with this here, or start a new thread?

28

Better start a new thread for your syslog questions.