LogStash not sending data to ElasticSearch

Elastic Stack Logstash
1

This is my logstash conf -
input {
file {
path => "C:\LogStash Logs*"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

ES is working fine. What ever files are present under LogStash Logs folder should be seen in Kibana, but its not coming up.

What am I missing here ?

2 
path => "C:\LogStash Logs*"

This should probably be:

path => "C:/LogStash Logs/*"

What ever files are present under LogStash Logs folder should be seen in Kibana,

No, but all new data appended to the chosen files will be read. Make sure you understand the file input's start_position option.

3

Still the same -

input {
file {
path => "C:/LogStash Logs/*"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

What should I try next ?

4

What about the start_position option—doesn't it apply here?

If the input file is older than 24 hours, adjust the file input's ignore_older option. If that doesn't help, enable verbose log output with the --verbose option. The resulting logs should contain clues about which files are being monitored

5

After setting the start_position => "beginning" it worked. I can my file lines are getting read. But there are 2 problems here which I can see.

  1. If my file ends without a ENTER, the last line is NOT READ by logstash. If I hit ENTER after the last line, save it, then the last line is read.
  2. I tried using "beginning" and "end", but my OLD log files are not saved in ES and hence couldnt be searched in Kibana. It is only taking my new file and its changes.
6
  1. Indeed. The file input with the default codec only reads whole lines.
  2. Did you look at the ignore_older option that I pointed you to? If yes, maybe you at some point had those files listed in a filename pattern for a file input without start_position => beginning. Then the file input plugin will have recorded the current position of those files to be at the end of them, so it's still tailing the files. The file input documentation explains how the sincedb mechanism works, and starting Logstash with `--verbose´ will give you some insights into what's going on.
7

This worked fine. Thanks Magnus !!!!

Since, I got the connectivity done from LOg-> ES, now I am trying to connect FB -> LG -> ES. The only change that I need to do is update the FB.YML file with uncommenting the Logstash configuration section. Since, I am not using the localhost, i need to provide my server name. What is the port I need to use inorder to connect to logstash and FB to send data to Lg. 9200/9300 is not working.

8

Pick any unused port. The port in your Logstash configuration must obviously match the one in your Filebeat configuration.

9

How and Where can I configure logstash port ? is this something I need to do under the CONF file ?

What is the expected result if I browse - http://myserverpath:logstashPort/

10

How and Where can I configure logstash port ? is this something I need to do under the CONF file ?

Yes. The beats input.

What is the expected result if I browse - http://myserverpath:logstashPort/

That won't work since the Beats protocol isn't HTTP.

11

Do I need to install the input beats plugin? If yes, I am unable to do it.

https://discuss.elastic.co/t/unable-to-install-logstash-input-plugin-in-windows/44793?source_topic_id=44791

Or I need to configure the FB itself.

12

Do I need to install the input beats plugin?

Yes. It's preinstalled with Logstash 2.0+ I think.

If yes, I am unable to do it.
https://discuss.elastic.co/t/unable-to-install-logstash-input-plugin-in-windows/44793

Weird, I can't access that link.

13

If it is installed already...how can i figure that out ? And how can i find the port number to be used

14

If it is installed already...how can i figure that out ?

Logstash's plugin command can tell you that.

And how can i find the port number to be used

As I said, pick any unused port that you want to use.

15

I dont see any plugin installed with logstash. What can I do now ?

16

How did you reach the conclusion that you don't have any plugins installed? Show us what you're doing, not your interpretation of what's going on. We are wasting time and patience.

Recent versions of Logstash include the beats plugin and if you're not using a recent version you should upgrade.

17

Magnus - I am using logstash 2.2.2 and was referring to the BIN folder where I thght all the plugins will be installed. When I used "plugin list" command i saw that logstash-input-beats plugin is already installed. But still I am unable to locate the directory where this is installed.

I am still unable to connect my FB -> LS. This is something new for me and trying hard to get it connected. I tried using a port in the logstash conf -> input -> beats { port => 4434} and made the same logstash config changes in FB.YML file host ["myserver:4434"]. Is there any other config that I am missing ?

Thanks and I truely appreciate your help here.

18

I am using logstash 2.2.2 and was referring to the BIN folder where I thght all the plugins will be installed.

No, that's not where they're installed.

When I used "plugin list" command i saw that logstash-input-beats plugin is already installed. But still I am unable to locate the directory where this is installed.

I don't see why that even matters. But looking for files with "beats" in their names should be fruitful.

I am still unable to connect my FB -> LS. This is something new for me and trying hard to get it connected. I tried using a port in the logstash conf -> input -> beats { port => 4434} and made the same logstash config changes in FB.YML file host ["myserver:4434"]. Is there any other config that I am missing ?

Sounds good so far. Check your Filebeat logs for clues.

19

It is unable to connect to http://myserver:4434 port.

FB Logs -
tcp [fe80::9003:661e:819e:5ace%Wireless Network Connection]:4434: connectex: No connection could be made because the target machine actively refused it.
2016-03-25T11:53:04-05:00 INFO send fail
2016-03-25T11:53:04-05:00 INFO backoff retry: 1s
2016-03-25T11:53:08-05:00 INFO Connecting error publishing events (retrying): Head http://myserver:4434: dial tcp [fe80::9003:661e:819e:5ace%Wireless Network Connection]:4434: connectex: No connection could be made because the target machine actively refused it.
2016-03-25T11:53:08-05:00 INFO send fail
2016-03-25T11:53:08-05:00 INFO backoff retry: 2s
2016-03-25T11:53:13-05:00 INFO Connecting error publishing events (retrying): Head http://myserver:4434: dial tcp [fe80::9003:661e:819e:5ace%Wireless Network Connection]:4434: connectex: No connection could be made because the target machine actively refused it.
2016-03-25T11:53:13-05:00 INFO send fail
2016-03-25T11:53:13-05:00 INFO backoff retry: 4s
2016-03-25T11:53:20-05:00 INFO Connecting error publishing events (retrying): Head http://myserver:4434: dial tcp [fe80::9003:661e:819e:5ace%Wireless Network Connection]:4434: connectex: No connection could be made because the target machine actively refused it.
2016-03-25T11:53:20-05:00 INFO send fail
2016-03-25T11:53:20-05:00 INFO backoff retry: 8s
2016-03-25T11:53:31-05:00 INFO Connecting error publishing events (retrying): Head http://myserver:4434: dial tcp [fe80::9003:661e:819e:5ace%Wireless Network Connection]:4434: connectex: No connection could be made because the target machine actively refused it.
2016-03-25T11:53:31-05:00 INFO send fail
2016-03-25T11:53:31-05:00 INFO backoff retry: 16s

Logstash Logs -
←[33mBeats Input: Remote connection closed {:peer=>"fe80:0:0:0:9003:661e:819e:5a
ce%16:54078", :exception=>#<Lumberjack::Beats::Connection::ConnectionClosed: Lum
berjack::Beats::Connection::ConnectionClosed wrapping: Lumberjack::Beats::Parser
::UnsupportedProtocol, unsupported protocol 72>, :level=>:warn}←[0m

20

Show your Filebeat configuration file and format it as code with the </> toolbar button.

Might as well show your Logstash configuration too.