# Topbeat and logstash error
**URL:** https://discuss.elastic.co/t/topbeat-and-logstash-error/43213
**Category:** Beats
**Created:** [March 2, 2016, 8:49am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213 "2016-03-02T08:49:59Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 2, 2016, 8:49am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/1 "2016-03-02T08:49:59Z")
HI There,
First time using these products, working through the installation procedure and all was going well until the config file /etc/topbeat/topbeat.yml all looks good but getting a strange error about a key in line 118?
It's all locally configured, did not think any keys were needed?
[https://www.elastic.co/guide/en/beats/topbeat/current/topbeat-configuration.html](https://www.elastic.co/guide/en/beats/topbeat/current/topbeat-configuration.html)
sudo /etc/init.d/topbeat start
Loading config file error: YAML config parsing failed on /etc/topbeat/topbeat.yml: yaml: line 118: did not find expected key. Exiting.
...fail!
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 2, 2016, 9:12am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/2 "2016-03-02T09:12:59Z")
It would help if you posted your /etc/topbeat/topbeat.yml. Make sure you format it as code with the `` button.
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 2, 2016, 11:48pm UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/4 "2016-03-02T23:48:08Z")
topbeat.yml file is 8,910 bytes, attempt to paste but exceeded the character limit, attempted to attach but not accepted file type
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [March 3, 2016, 12:17am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/5 "2016-03-03T00:17:00Z")
can you remove the commented out lines + empty lines before posting? Makes config file much easier to read for trouble shooting.
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 3, 2016, 12:51am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/7 "2016-03-03T00:51:23Z")
Removed the additional hosts and service started, thank you.
The main thing wanting to achieve is:
- Collect metrics for all cores and for all processes
- Topbeat to pass to logstash
- Logstash to store in Elasticsearch and InfluxDB
Not sure about inputing, parsing and outputing files yet. Get basic metrics working first.
```auto
input:
period: 10
procs: [".*"]
stats:
system: true
proc: true
filesystem: true
cpu_per_core: true
output:
logstash:
hosts: ["127.0.0.1:5044"]
# Number of workers per Logstash host.
#worker: 1
# Set gzip compression level.
#compression_level: 3
# Optional load balance the events between the Logstash hosts
#loadbalance: true
# Optional index name. The default index name depends on the each beat.
# For Packetbeat, the default is set to packetbeat, for Topbeat
# top topbeat and for Filebeat to filebeat.
#index: topbeat
### File as output
#file:
# Path to the directory where to save the generated files. The option is mandatory.
#path: "/tmp/topbeat"
# Name of the generated files. The default is `topbeat` and it generates files: `topbeat`, `topbeat.1`, `topbeat.2`, etc.
#filename: topbeat
# Maximum size in kilobytes of each file. When this size is reached, the files are
# rotated. The default value is 10 MB.
#rotate_every_kb: 10000
# Maximum number of files under path. When this number of files is reached, the
# oldest file is deleted and the rest are shifted from last to first. The default
# is 7 files.
#number_of_files: 7
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB
```
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [March 3, 2016, 1:11am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/8 "2016-03-03T01:11:19Z")
Try starting with this minimal config. Put only this in your `/etc/topbeat/topbeat.yml`.
```auto
input:
period: 10
procs: [".*"]
stats:
system: true
proc: true
filesystem: true
cpu_per_core: true
output:
logstash:
hosts: ["127.0.0.1:5044"]
logging:
to_files: true
files:
path: /var/log/topbeat
name: topbeat.log
rotateeverybytes: 10485760
keepfiles: 7
level: info
```
Then start Topbeat in the foreground. Check if there are any errors communicating to Logstash. This will also print all of the JSON being sent to logstash, if you don't want this then remove the `-d "publish"`.
`sudo topbeat -c /etc/topbeat/topbeat.yml -e -v -d "publish"`
If there are no errors sending to Logstash, then stop the process and start it up as a service.
`sudo service topbeat start` (command varies by OS)
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 4, 2016, 3:14am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/10 "2016-03-04T03:14:34Z")
Just tested topbeat with above config file, not sure if this normal output or not
$ sudo topbeat -c /etc/topbeat/topbeat.yml -e -v -d "publish"
`2016/03/04 03:10:54.634112 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths 2016/03/04 03:10:54.634369 logstash.go:106: INFO Max Retries set to: 3 2016/03/04 03:10:54.634585 outputs.go:119: INFO Activated logstash as output plugin. 2016/03/04 03:10:54.634605 publish.go:232: DBG Create output worker 2016/03/04 03:10:54.634644 publish.go:274: DBG No output is defined to store the topology. The server fields might not be filled. 2016/03/04 03:10:54.634669 publish.go:288: INFO Publisher name: heliase-dev-36 2016/03/04 03:10:54.634957 async.go:78: INFO Flush Interval set to: 1s 2016/03/04 03:10:54.634967 async.go:84: INFO Max Bulk Size set to: 2048 2016/03/04 03:10:54.634973 async.go:92: DBG create bulk processing worker (interval=1s, bulk size=2048) 2016/03/04 03:10:54.635016 beat.go:147: INFO Init Beat: topbeat; Version: 1.1.1 2016/03/04 03:10:54.635286 beat.go:173: INFO topbeat sucessfully setup. Start running.`
Then after a few seconds the JSON output appears and then some errors:
`2016/03/04 03:13:48.117228 output.go:87: DBG output worker: publish 302 events 2016/03/04 03:13:48.118043 single.go:126: INFO Connecting error publishing events (retrying): dial tcp 127.0.0.1:5044: getsockopt: connection refused 2016/03/04 03:13:48.118075 single.go:152: INFO send fail 2016/03/04 03:13:48.118085 single.go:159: INFO backoff retry: 1s 2016/03/04 03:13:49.118532 single.go:126: INFO Connecting error publishing events (retrying): dial tcp 127.0.0.1:5044: getsockopt: connection refused 2016/03/04 03:13:49.118564 single.go:152: INFO send fail 2016/03/04 03:13:49.118573 single.go:159: INFO backoff retry: 2s`
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [March 4, 2016, 4:42am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/11 "2016-03-04T04:42:58Z")
> [@gabe](#):
>
> dial tcp 127.0.0.1:5044: getsockopt: connection refused
Is Logstash running and listening on `localhost:5044`?
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 4, 2016, 5:10am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/12 "2016-03-04T05:10:03Z")
It seems to be:
sudo service logstash start
logstash started
/opt/logstash/bin/logstash.conf
`input {
beats {
port =\> 5044
}
}
output {
elasticsearch {
hosts =\> "localhost:9200"
manage\_template =\> false
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document\_type =\> "%{[@metadata][type]}"
}
}
`
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 4, 2016, 6:40am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/13 "2016-03-04T06:40:14Z")
Okay, but can you connect to localhost:5044 with e.g. telnet? You _are_ running Topbeat and Logstash on the same machine, right?
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 4, 2016, 4:28pm UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/15 "2016-03-04T16:28:53Z")
Yes, topbeat, logstash, elasticsearch and influx are all running on the same server(s)
Could establish connection to localhost 22 and 9200
Could not establish connection to localhost 5044 - connection refused
See below
`
$ telnet localhost 22 Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH\_6.6.1p1 Ubuntu-2ubuntu2.6
^C
Connection closed by foreign host.
$ telnet localhost 9200 Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^C
Connection closed by foreign host.
$ telnet localhost 5044 Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
`
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 4, 2016, 5:35pm UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/16 "2016-03-04T17:35:38Z")
Okay, looks like problem is with logstash
`
$ sudo /etc/init.d/logstash start
logstash started.
$ sudo /etc/init.d/logstash status
logstash is running
$ sudo /etc/init.d/logstash status
logstash is not running
`
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 4, 2016, 5:46pm UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/17 "2016-03-04T17:46:01Z")
Check the Logstash logs for clues.
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 4, 2016, 9:48pm UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/18 "2016-03-04T21:48:32Z")
Thank you for pushing me through the process, it looks like logstash is up and running..
/var/log/logstash$ sudo nano ./logstash.log.1
`{:timestamp=>"2016-03-04T04:43:57.846000+0000", :message=>"Error: No config files found: /etc/logstash/conf.d/*\nCan you make sure this path is a logstash config file?"} {:timestamp=>"2016-03-04T04:43:57.852000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto resta$`
/opt/logstash/bin/logstash/logstash.conf file in bin directory
copied to /etc/logstash/conf.d/ directory
$ sudo /etc/init.d/logstash start
$ sudo /etc/init.d/logstash status
$ telnet localhost 5044
`Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. ^C Connection closed by foreign host`
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 4, 2016, 9:52pm UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/19 "2016-03-04T21:52:17Z")
Ran topbeat in the foreground and looks like topbeat it is connecting to logstash
sudo topbeat -c /etc/topbeat/topbeat.yml -e -v -d "publish"
After JSON
2016/03/04 21:49:53.556105 output.go:87: DBG output worker: publish 310 events
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 4, 2016, 9:55pm UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/20 "2016-03-04T21:55:06Z")
I guess now the next step is to get back to the topbeat installation guide and make sure it is connecting and storing data in Elasticsearch.
Then to configure and additional output for Logstash to store data in Influx.
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 5, 2016, 5:41am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/21 "2016-03-05T05:41:16Z")
Backed out syslog input, reverted back to topbeat input and now logstash is stopping.
Don't know what is going on.
sudo service logstash start
**/var/opt/logstash/logstash.log**
{:timestamp=\>"2016-03-05T03:59:35.952000+0000", :message=\>"SIGTERM received. Shutting down the pipeline.", :level=\>:warn}
{:timestamp=\>"2016-03-05T05:12:10.864000+0000", :message=\>"SIGTERM received. Shutting down the pipeline.", :level=\>:warn}
{:timestamp=\>"2016-03-05T05:12:27.845000+0000", :message=\>"The error reported is: \n Address already in use - bind - Address already in use"}
{:timestamp=\>"2016-03-05T05:28:39.272000+0000", :message=\>"The error reported is: \n Address already in use - bind - Address already in use"}
{:timestamp=\>"2016-03-05T05:32:33.025000+0000", :message=\>"The error reported is: \n Address already in use - bind - Address already in use"}
**/etc/logstash/conf.d/logstash.conf**
input {
beats {
port =\> 5044
}
}
output {
elasticsearch {
hosts =\> "localhost:9200"
manage\_template =\> false
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document\_type =\> "%{[@metadata][type]}"
}
}
Need to get this working again, before moving forward..
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 5, 2016, 7:09pm UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/22 "2016-03-05T19:09:16Z")
Okay, have worked through the installation procedure on 3 servers, with three different results, a bit surprising and a little fustrating, greatful for assistance with the diagnostics
dev-36 elasticsearch working, **logstash stops working** , topbeat working
dev-37 elasticsearch working, logstash working, topbeat working
dev-38 elasticsearch working, logstash working, **topbeat not working**
dev-36:~$ sudo nano /var/log/logstash/logstash.log
message=\>"Error: Expected one of #, input, filter, output at line 82, column 1 (byte 1455) after "}
**The topbeat service start commands produce three different results**
dev-36:~$ sudo service topbeat start
2016/03/05 19:03:55.183995 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/03/05 19:03:55.184097 logstash.go:106: INFO Max Retries set to: 3
2016/03/05 19:03:55.184242 outputs.go:119: INFO Activated logstash as output plugin.
2016/03/05 19:03:55.184280 publish.go:288: INFO Publisher name: heliase-dev-36
2016/03/05 19:03:55.184441 async.go:78: INFO Flush Interval set to: 1s
2016/03/05 19:03:55.184459 async.go:84: INFO Max Bulk Size set to: 2048
2016/03/05 19:03:55.184485 beat.go:147: INFO Init Beat: topbeat; Version: 1.1.1
/usr/bin/topbeat-god already running.
dev-37:~$ sudo service topbeat start
/usr/bin/topbeat-god already running.
dev-38:~$ sudo service topbeat start
Loading config file error: YAML config parsing failed on /etc/topbeat/topbeat.yml: yaml: unmarshal errors:
line 11: cannot unmarshal !!seq into outputs.MothershipConfig. Exiting.
...fail!
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [March 6, 2016, 12:13am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/23 "2016-03-06T00:13:08Z")
Very hard to follow your steps. Seems like you're doing everything at once.
I'd start with topbeat-\>logstash config and check it works correctly. In logstash do:
```auto
input {
beats {
port => 5044
}
}
output {
stdout {
codec => rubydebug
}
}
```
- minimal topbeat:
```auto
input:
period: 10
procs: [".*"]
stats:
system: true
proc: true
filesystem: true
cpu_per_core: false
output:
logstash:
hosts: ["localhost:5044"]
```
Do not use init scripts to start logstash, but run in foreground via:
```
/path/to/logstash agent -f
```
use ps tool checking processes are indeed not running.
some issues you have had so far:
`(retrying): dial tcp 127.0.0.1:5044: getsockopt: connection refused`
port 5044 not open. Either logstash misconfigured or not running
`telnet: Unable to connect to remote host: Connection refused`
Same, port 5044 not open. Either logstash misconfigured or not running
```auto
{:timestamp=>"2016-03-04T04:43:57.846000+0000", :message=>"Error: No config files found: /etc/logstash/conf.d/*\nCan you make sure this path is a logstash config file?"}
```
Message is very clear, no config files found. Where did you put your configs?
> Ran topbeat in the foreground and looks like topbeat it is connecting to logstash
> sudo topbeat -c /etc/topbeat/topbeat.yml -e -v -d "publish"
> After JSON
> 2016/03/04 21:49:53.556105 output.go:87: DBG output worker: publish 310 events
The log message posted is incomplete. message 'publish 310 events' is telling topbeat created 310 events to be published. Any message from logstash output plugin? Can you confirm events being actually processed by logstash and index in influx?
After re-enabling topbeat you got:
```auto
{:timestamp=>"2016-03-05T05:12:27.845000+0000", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}
```
Message is clear, the port (whatever port it is), is still in use. An old Logstash instance still running?
Another one:
```auto
message=>"Error: Expected one of #, input, filter, output at line 82, column 1 (byte 1455) after "
```
Logstash config file error at line 82 ?
And in topbeat:
```auto
Loading config file error: YAML config parsing failed on /etc/topbeat/topbeat.yml: yaml: unmarshal errors:
line 11
```
topbeat config file incorrect. Output configuration is faulty. Check your YAML-foo and be carefull about indentation (don't use tabs, but spaces to be safe).
Looks like main reasons for logstash/topbeat not running are errors in config files.
---
### Author:  [@gabe](https://discuss.elastic.co/u/gabe)
#### Post date: [March 6, 2016, 3:59am UTC](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213/24 "2016-03-06T03:59:10Z")
Thank you that's very helpful.
I will start with dev-36 and look to get that solved first. Good point about the configuration file formatting, exactly how sensitive are these files to indentation?
For instance, your example code, nice indentation, assuming that's not tabs but spaces?
I am new to these formats - is more or less spaces in the indentation an issue, surely it is the parenthesis that open and close the statements?
I have been copying and pasting in-between local editors and web pages as building out our documentation set, entirely possibly bad characters could have got in.
> [@](#):
>
> /path/to/logstash agent -f \
I might have this wrong, my understanding is the paths below are the binary and configuration file locations for when running logstash as a service in the background which I have been starting with the command "sudo service logstash start'
/etc/init.d/logstash
/etc/logstash/conf.d/logstash.conf
When it comes to foreground testing, this is best done in the path below for both binary and config files
/opt/logstash/bin/logstash
eg
/opt/logstash/bin/logstash agent -f logstash-minimal.conf
Testing steps:
**- 1st execution of Logstash**
dev-36:/opt/logstash/bin$ ./logstash agent -f logstash.conf
Settings: Default pipeline workers: 16
Logstash startup completed
**- Logstash did not return to the command line**
**- Opened another terminal to view all processes (ps -A) and could not locate Logstash**
**- ^Z to return to the command line**
^Z
[3]+ Stopped ./logstash agent -f logstash.conf
**- 2nd execution of logstash**
dev-36:/opt/logstash/bin$ ./logstash agent -f logstash.conf
Settings: Default pipeline workers: 16
The error reported is:
Address already in use - bind - Address already in use
**- Other terminal to view all processes (ps -A) and could not locate Logstash**
/var/log/logstash/logstash.log did not appear to be updated after the above commands.
Is this the only location for the log file?
Is this log file used regardless of which Logstash binary and configuration file are executed?
What is/was the expected result from the foreground execution of Logstash?
**ps on dev-36 (logstash not running)**
ps -lu logstash
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
**ps on dev-37 (logstash is running)**
ps -lu logstash
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 995 32497 1 1 99 19 - 1761281 - ? 00:07:02 java
ps -u logstash l
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
4 995 32497 1 39 19 7045124 629028 - SNl ? 7:02 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+
dev-36 sudo service logstash start
logstash starts and stops
dev-36 /var/logstash/conf.d/logstash.log
{:timestamp=\>"2016-03-06T05:21:37.805000+0000", :message=\>"Error: Expected one of #, input, filter, output at line 81, column 1 (byte 1426) after "}
{:timestamp=\>"2016-03-06T05:21:37.811000+0000", :message=\>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}
I don't know what this is referring to?
**..at line 81, column 1 (byte 1426) after "}**
logstash.conf does not have 81 lines
[Next page](https://discuss.elastic.co/t/topbeat-and-logstash-error/43213.md?page=2)