Topbeat and logstash error

Elastic Stack Beats
1

HI There,

First time using these products, working through the installation procedure and all was going well until the config file /etc/topbeat/topbeat.yml all looks good but getting a strange error about a key in line 118?

It's all locally configured, did not think any keys were needed?

https://www.elastic.co/guide/en/beats/topbeat/current/topbeat-configuration.html

sudo /etc/init.d/topbeat start

Loading config file error: YAML config parsing failed on /etc/topbeat/topbeat.yml: yaml: line 118: did not find expected key. Exiting.
...fail!

2

It would help if you posted your /etc/topbeat/topbeat.yml. Make sure you format it as code with the </> button.

4

topbeat.yml file is 8,910 bytes, attempt to paste but exceeded the character limit, attempted to attach but not accepted file type

5

can you remove the commented out lines + empty lines before posting? Makes config file much easier to read for trouble shooting.

7

Removed the additional hosts and service started, thank you.

The main thing wanting to achieve is:

  • Collect metrics for all cores and for all processes
  • Topbeat to pass to logstash
  • Logstash to store in Elasticsearch and InfluxDB

Not sure about inputing, parsing and outputing files yet. Get basic metrics working first.

input:
  period: 10
  procs: [".*"]
  stats:
    system: true
    proc: true
    filesystem: true
    cpu_per_core: true
output:
   logstash:
    hosts: ["127.0.0.1:5044"]
    # Number of workers per Logstash host.
    #worker: 1
    # Set gzip compression level.
    #compression_level: 3
    # Optional load balance the events between the Logstash hosts
    #loadbalance: true
    # Optional index name. The default index name depends on the each beat.
    # For Packetbeat, the default is set to packetbeat, for Topbeat
    # top topbeat and for Filebeat to filebeat.
    #index: topbeat
  ### File as output
  #file:
    # Path to the directory where to save the generated files. The option is mandatory.
    #path: "/tmp/topbeat"
    # Name of the generated files. The default is `topbeat` and it generates files: `topbeat`, `topbeat.1`, `topbeat.2`, etc.
    #filename: topbeat
    # Maximum size in kilobytes of each file. When this size is reached, the files are
    # rotated. The default value is 10 MB.
    #rotate_every_kb: 10000
    # Maximum number of files under path. When this number of files is reached, the
    # oldest file is deleted and the rest are shifted from last to first. The default
    # is 7 files.
    #number_of_files: 7
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 10MB
8

Try starting with this minimal config. Put only this in your /etc/topbeat/topbeat.yml.

input:
  period: 10
  procs: [".*"]
  stats:
    system: true
    proc: true
    filesystem: true
    cpu_per_core: true
output:
  logstash:
    hosts: ["127.0.0.1:5044"]
logging:
  to_files: true
  files:
    path: /var/log/topbeat
    name: topbeat.log
    rotateeverybytes: 10485760
    keepfiles: 7
  level: info

Then start Topbeat in the foreground. Check if there are any errors communicating to Logstash. This will also print all of the JSON being sent to logstash, if you don't want this then remove the -d "publish".

sudo topbeat -c /etc/topbeat/topbeat.yml -e -v -d "publish"

If there are no errors sending to Logstash, then stop the process and start it up as a service.

sudo service topbeat start (command varies by OS)

10

Just tested topbeat with above config file, not sure if this normal output or not

$ sudo topbeat -c /etc/topbeat/topbeat.yml -e -v -d "publish"

2016/03/04 03:10:54.634112 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths 2016/03/04 03:10:54.634369 logstash.go:106: INFO Max Retries set to: 3 2016/03/04 03:10:54.634585 outputs.go:119: INFO Activated logstash as output plugin. 2016/03/04 03:10:54.634605 publish.go:232: DBG Create output worker 2016/03/04 03:10:54.634644 publish.go:274: DBG No output is defined to store the topology. The server fields might not be filled. 2016/03/04 03:10:54.634669 publish.go:288: INFO Publisher name: heliase-dev-36 2016/03/04 03:10:54.634957 async.go:78: INFO Flush Interval set to: 1s 2016/03/04 03:10:54.634967 async.go:84: INFO Max Bulk Size set to: 2048 2016/03/04 03:10:54.634973 async.go:92: DBG create bulk processing worker (interval=1s, bulk size=2048) 2016/03/04 03:10:54.635016 beat.go:147: INFO Init Beat: topbeat; Version: 1.1.1 2016/03/04 03:10:54.635286 beat.go:173: INFO topbeat sucessfully setup. Start running.

Then after a few seconds the JSON output appears and then some errors:

2016/03/04 03:13:48.117228 output.go:87: DBG output worker: publish 302 events 2016/03/04 03:13:48.118043 single.go:126: INFO Connecting error publishing events (retrying): dial tcp 127.0.0.1:5044: getsockopt: connection refused 2016/03/04 03:13:48.118075 single.go:152: INFO send fail 2016/03/04 03:13:48.118085 single.go:159: INFO backoff retry: 1s 2016/03/04 03:13:49.118532 single.go:126: INFO Connecting error publishing events (retrying): dial tcp 127.0.0.1:5044: getsockopt: connection refused 2016/03/04 03:13:49.118564 single.go:152: INFO send fail 2016/03/04 03:13:49.118573 single.go:159: INFO backoff retry: 2s

11

Is Logstash running and listening on localhost:5044?

12

It seems to be:
sudo service logstash start
logstash started

/opt/logstash/bin/logstash.conf

`input {
beats {
port => 5044
}
}

output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

`

13

Okay, but can you connect to localhost:5044 with e.g. telnet? You are running Topbeat and Logstash on the same machine, right?

15

Yes, topbeat, logstash, elasticsearch and influx are all running on the same server(s)

Could establish connection to localhost 22 and 9200
Could not establish connection to localhost 5044 - connection refused

See below

`
$ telnet localhost 22 Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6
^C
Connection closed by foreign host.

$ telnet localhost 9200 Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^C
Connection closed by foreign host.

$ telnet localhost 5044 Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
`

16

Okay, looks like problem is with logstash

`
$ sudo /etc/init.d/logstash start
logstash started.

$ sudo /etc/init.d/logstash status
logstash is running

$ sudo /etc/init.d/logstash status
logstash is not running
`

17

Check the Logstash logs for clues.

18

Thank you for pushing me through the process, it looks like logstash is up and running..

/var/log/logstash$ sudo nano ./logstash.log.1

{:timestamp=>"2016-03-04T04:43:57.846000+0000", :message=>"Error: No config files found: /etc/logstash/conf.d/*\nCan you make sure this path is a logstash config file?"} {:timestamp=>"2016-03-04T04:43:57.852000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto resta$

/opt/logstash/bin/logstash/logstash.conf file in bin directory
copied to /etc/logstash/conf.d/ directory

$ sudo /etc/init.d/logstash start
$ sudo /etc/init.d/logstash status

$ telnet localhost 5044
Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. ^C Connection closed by foreign host

19

Ran topbeat in the foreground and looks like topbeat it is connecting to logstash

sudo topbeat -c /etc/topbeat/topbeat.yml -e -v -d "publish"

After JSON

2016/03/04 21:49:53.556105 output.go:87: DBG output worker: publish 310 events

20

I guess now the next step is to get back to the topbeat installation guide and make sure it is connecting and storing data in Elasticsearch.

Then to configure and additional output for Logstash to store data in Influx.

21

Backed out syslog input, reverted back to topbeat input and now logstash is stopping.

Don't know what is going on.

sudo service logstash start

/var/opt/logstash/logstash.log

{:timestamp=>"2016-03-05T03:59:35.952000+0000", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-03-05T05:12:10.864000+0000", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-03-05T05:12:27.845000+0000", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}
{:timestamp=>"2016-03-05T05:28:39.272000+0000", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}
{:timestamp=>"2016-03-05T05:32:33.025000+0000", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}

/etc/logstash/conf.d/logstash.conf

input {
beats {
port => 5044
}
}

output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Need to get this working again, before moving forward..

22

Okay, have worked through the installation procedure on 3 servers, with three different results, a bit surprising and a little fustrating, greatful for assistance with the diagnostics

dev-36 elasticsearch working, logstash stops working, topbeat working
dev-37 elasticsearch working, logstash working, topbeat working
dev-38 elasticsearch working, logstash working, topbeat not working

dev-36:~$ sudo nano /var/log/logstash/logstash.log
message=>"Error: Expected one of #, input, filter, output at line 82, column 1 (byte 1455) after "}

The topbeat service start commands produce three different results

dev-36:~$ sudo service topbeat start
2016/03/05 19:03:55.183995 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/03/05 19:03:55.184097 logstash.go:106: INFO Max Retries set to: 3
2016/03/05 19:03:55.184242 outputs.go:119: INFO Activated logstash as output plugin.
2016/03/05 19:03:55.184280 publish.go:288: INFO Publisher name: heliase-dev-36
2016/03/05 19:03:55.184441 async.go:78: INFO Flush Interval set to: 1s
2016/03/05 19:03:55.184459 async.go:84: INFO Max Bulk Size set to: 2048
2016/03/05 19:03:55.184485 beat.go:147: INFO Init Beat: topbeat; Version: 1.1.1
/usr/bin/topbeat-god already running.

dev-37:~$ sudo service topbeat start
/usr/bin/topbeat-god already running.

dev-38:~$ sudo service topbeat start
Loading config file error: YAML config parsing failed on /etc/topbeat/topbeat.yml: yaml: unmarshal errors:
line 11: cannot unmarshal !!seq into outputs.MothershipConfig. Exiting.
...fail!

23

Very hard to follow your steps. Seems like you're doing everything at once.

I'd start with topbeat->logstash config and check it works correctly. In logstash do:

input {
    beats {
        port => 5044
    }
}

output {
    stdout {
        codec => rubydebug
    }
}
  • minimal topbeat:
input:
  period: 10
  procs: [".*"]
  stats:
    system: true
    proc: true
    filesystem: true
    cpu_per_core: false
output:
  logstash:
    hosts: ["localhost:5044"]

Do not use init scripts to start logstash, but run in foreground via:

/path/to/logstash agent -f <config file>

use ps tool checking processes are indeed not running.

some issues you have had so far:

(retrying): dial tcp 127.0.0.1:5044: getsockopt: connection refused
port 5044 not open. Either logstash misconfigured or not running

telnet: Unable to connect to remote host: Connection refused
Same, port 5044 not open. Either logstash misconfigured or not running

{:timestamp=>"2016-03-04T04:43:57.846000+0000", :message=>"Error: No config files found: /etc/logstash/conf.d/*\nCan you make sure this path is a logstash config file?"}

Message is very clear, no config files found. Where did you put your configs?

Ran topbeat in the foreground and looks like topbeat it is connecting to logstash
sudo topbeat -c /etc/topbeat/topbeat.yml -e -v -d "publish"
After JSON
2016/03/04 21:49:53.556105 output.go:87: DBG output worker: publish 310 events

The log message posted is incomplete. message 'publish 310 events' is telling topbeat created 310 events to be published. Any message from logstash output plugin? Can you confirm events being actually processed by logstash and index in influx?

After re-enabling topbeat you got:

{:timestamp=>"2016-03-05T05:12:27.845000+0000", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}

Message is clear, the port (whatever port it is), is still in use. An old Logstash instance still running?

Another one:

message=>"Error: Expected one of #, input, filter, output at line 82, column 1 (byte 1455) after "

Logstash config file error at line 82 ?

And in topbeat:

Loading config file error: YAML config parsing failed on /etc/topbeat/topbeat.yml: yaml: unmarshal errors:
line 11

topbeat config file incorrect. Output configuration is faulty. Check your YAML-foo and be carefull about indentation (don't use tabs, but spaces to be safe).

Looks like main reasons for logstash/topbeat not running are errors in config files.

2 Likes
24

Thank you that's very helpful.

I will start with dev-36 and look to get that solved first. Good point about the configuration file formatting, exactly how sensitive are these files to indentation?

For instance, your example code, nice indentation, assuming that's not tabs but spaces?

I am new to these formats - is more or less spaces in the indentation an issue, surely it is the parenthesis that open and close the statements?

I have been copying and pasting in-between local editors and web pages as building out our documentation set, entirely possibly bad characters could have got in.

I might have this wrong, my understanding is the paths below are the binary and configuration file locations for when running logstash as a service in the background which I have been starting with the command "sudo service logstash start'

/etc/init.d/logstash
/etc/logstash/conf.d/logstash.conf

When it comes to foreground testing, this is best done in the path below for both binary and config files

/opt/logstash/bin/logstash

eg

/opt/logstash/bin/logstash agent -f logstash-minimal.conf

Testing steps:

- 1st execution of Logstash

dev-36:/opt/logstash/bin$ ./logstash agent -f logstash.conf
Settings: Default pipeline workers: 16
Logstash startup completed

- Logstash did not return to the command line
- Opened another terminal to view all processes (ps -A) and could not locate Logstash
- ^Z to return to the command line

^Z
[3]+ Stopped ./logstash agent -f logstash.conf

- 2nd execution of logstash

dev-36:/opt/logstash/bin$ ./logstash agent -f logstash.conf

Settings: Default pipeline workers: 16
The error reported is:
Address already in use - bind - Address already in use

- Other terminal to view all processes (ps -A) and could not locate Logstash

/var/log/logstash/logstash.log did not appear to be updated after the above commands.

Is this the only location for the log file?

Is this log file used regardless of which Logstash binary and configuration file are executed?

What is/was the expected result from the foreground execution of Logstash?

ps on dev-36 (logstash not running)

ps -lu logstash
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD

ps on dev-37 (logstash is running)

ps -lu logstash

F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 995 32497 1 1 99 19 - 1761281 - ? 00:07:02 java

ps -u logstash l

F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
4 995 32497 1 39 19 7045124 629028 - SNl ? 7:02 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+

dev-36 sudo service logstash start

logstash starts and stops

dev-36 /var/logstash/conf.d/logstash.log

{:timestamp=>"2016-03-06T05:21:37.805000+0000", :message=>"Error: Expected one of #, input, filter, output at line 81, column 1 (byte 1426) after "}

{:timestamp=>"2016-03-06T05:21:37.811000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}

I don't know what this is referring to?

..at line 81, column 1 (byte 1426) after "}

logstash.conf does not have 81 lines