Newbie: logstash not talking to elasticsearch

Elastic Stack Logstash
1

I'm been struggling for three days more or less to get pfsense logs into elasticsearch. I'm running debian jessie on a VM. pfsense is running real. I did the easy config in pfsense, setting up IP local IP and port 514. I installed the two debian packages logstash and elasticsurch via dpkg. I have to manually start the services via systemctl but it looks all good.

I made a pfsense.conf file, just eash input output

input {
tcp {
    type => syslog
    port => 514
}
udp {
    type => syslog
    port => 514
}

}

output {

stdout { codec => rubydebug }
elasticsearch {
host => "127.0.0.1"
cluster => "elasticsearch"
}
}

I'm getting output in bash, everything seems to be fine, stuff is happening like this

{
   "message" => "<134>Jul 25 18:02:51 filterlog: 9,16777216,,1000000103,re0,match,block,in,4,0x0,,117,6044,0,none,17,udp,134,188.114.19.225,176.10.137.213,34544,40138,114",
  "@version" => "1",
"@timestamp" => "2015-07-25T16:02:38.535Z",
      "type" => "syslog",
      "host" => "10.10.10.1"
}

What I'm unsure of is if the logs are actually stored in elasticsearch, if I understand correctly logstash only "forwards" what's happening and elasticsearch actually stores it. And this is where I get confused. in the elasticsearch.yml I haven't touched much, only the name of the cluster which is the same as my pfsense.conf file. I've been looking for logs but can't find any. I'm been looking at /var/log/elasticsearch/elasticsearch.log which seems to be overwritten every time I restart ES service.

Please help. I'm not fluent in *nix but I'm learning.

2

What I'm unsure of is if the logs are actually stored in elasticsearch,

Use one of the available REST APIs to check what's going on in ES. The cat APIs should be useful, as should the search APIs. For example, run curl localhost:9200/_cat/indices to list all indexes and how many documents they contain. You should have at least one logstash-YYYY.MM.DD index.

If you're intending on using Kibana later on, install and configure it already now and use it as a debugging tool.

if I understand correctly logstash only "forwards" what's happening and elasticsearch actually stores it.

Correct. Logstash has no built-in storage of logs.

I'm been looking at /var/log/elasticsearch/elasticsearch.log which seems to be overwritten every time I restart ES service.

I don't think it's supposed to do that. Are you sure the original log files just hasn't been rotated (i.e. is saved under another name in the same directory)? Either way, ES mostly logs abnormal situations. If everything is working it'll be mostly quiet.

3

Thank you for you're reply.

/var/log/elasticsearch only contains four files, but from what I see it just appends to the elasticsearch.log, atleast for now. there is a second file that ends with a date from three days ago. I guess it'll stop append at midnight. I've been doing rollback to snapshots during these days so that's might be why there aren't other log files.

I did install kibana now but still haven't read up on how I use it or debug.

Edit: Well it seems data are actually stored, there is at least stuff that Kibana can see.

4

I'm getting the services to start with systemd now. figured out where the conf-files go /etc/logstash/conf.d/
From what I understand it goes through all the files here when the service is initialized and goes through the files alphabetically. No probs there, but checking the logfile in /var/log/logstash/logstash.log tells me there are permissions issues reading the config file.

Doing some troubleshooting and even changed the permissions on the config-file so that anybody can read it but it stills gives the same fail message

{:timestamp=>"2015-07-26T16:28:36.261000+0200", :message=>"The error reported is: \n  Permission                    denied - bind(2)"}

Any ideas on where it's failin?

5

Non-root users can only listen on ports >1024. You could run Logstash as root (not recommended), reconfigure the syslog senders to send to a non-priviliged port, or use any of the workarounds described at http://stackoverflow.com/questions/413807/is-there-a-way-for-non-root-processes-to-bind-to-privileged-ports-1024-on-l.

6

Awesome, I just changed in the .conf file to another port and now everything also starts at boot. Thank you for those last bits of information I needed.