Logstash output to external Elastic Search server

Elastic Stack Logstash
1

HI, have been struggling to get the output of one logstash servers output into an elastics search server. When doing a tcpdump on the ES installation I can see the server is connecting, however when looking at the indexes of ES there are none created for the external logstash server. The output of the logstash server loos as follows:
output { elasticsearch { hosts => [ "192.168.163.21:9200" ] sniffing => true flush_size => 20 idle_flush_time => 15 index => "IDNS-" } stdout { codec => rubydebug } file { path => "/var/log/syslog/ES/IBIND-%{+YYYY-MM-dd}.log" codec => "rubydebug" } }

The firewall is open on the port ranges 9200 to 9300 however the information is not reaching the ES installation on 192.168.163.21. Could anyone maybe assist as the logs of ES are not showing anything?

2

Can you telnet 192.168.163.21 9200 from the LS server?
What do the LS logs show, did you try --debug?

3

Yes, I can telnet to the port:
[root@localhost ~]# telnet 192.168.163.21 9200 Trying 192.168.163.21... Connected to 192.168.163.21. Escape character is '^]'.

In the --debug option all starts up however is there something specific I should look for in the output?

4

Here is the debug output section:
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@hosts = ["192.168.163.21:9200"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@sniffing = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@flush_size = 20 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@idle_flush_time = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@index = "IDNS-" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@manage_template = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@template_name = "logstash" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@template_overwrite = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@parent = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@upsert = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@doc_as_upsert = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@max_retries = 3 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@script = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@script_type = "inline" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@script_lang = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@script_var_name = "event" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@scripted_upsert = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@retry_max_interval = 2 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@retry_max_items = 500 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@action = "index" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@path = "/" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@ssl_certificate_verification = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Outputs::ElasticSearch/@sniffing_delay = 5 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} Normalizing http path {:path=>"/", :normalized=>"/", :level=>:debug, :file=>"logstash/outputs/elasticsearch/http_client_builder.rb", :line=>"14", :method=>"build"} Elasticsearch HTTP client options {:hosts=>["http://192.168.163.21:9200/"], :ssl=>nil, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :level=>:debug, :file=>"logstash/outputs/elasticsearch/http_client.rb", :line=>"127", :method=>"build_client"}

5

What inputs do you have?

6

The input configuration is as follows:
input { file { type => "IDNS" path => [ "/var/log/syslog/172.29.58.3-network.log" ] start_position => beginning } file { type => "IDNS" path => [ "/var/log/syslog/172.29.58.131-network.log" ] start_position => beginning } }

The debug information the the input:
config LogStash::Inputs::File/@type = "IDNS" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@path = ["/var/log/syslog/172.29.58.3-network.log"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@stat_interval = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@discover_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@sincedb_write_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@start_position = "end" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@delimiter = "\n" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@ignore_older = 86400 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@close_older = 3600 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@type = "IDNS" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@path = ["/var/log/syslog/172.29.58.131-network.log"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@stat_interval = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@discover_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@sincedb_write_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@start_position = "end" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@delimiter = "\n" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@ignore_older = 86400 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@close_older = 3600 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"mutate", :path=>"logstash/filters/mutate", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}

7

Comment out the elasticsearch output. Are you getting any data to your stdout and file outputs?

8

Yes, also with commenting out the elasticsearch section in the output, I get information written to the output file.

9

The index is still not available:
[root@syslog ~]# curl '192.168.163.21:9200/_cat/indices?v' health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open logstash-2016.02.28 5 1 1581621 0 1.1gb 1.1gb yellow open logstash-2016.02.27 5 1 1823938 0 1.2gb 1.2gb yellow open logstash-2016.02.26 5 1 1856485 0 1.3gb 1.3gb yellow open logstash-2016.02.25 5 1 2068778 0 1.4gb 1.4gb yellow open logstash-2016.02.19 5 1 1406589 0 1gb 1gb yellow open logstash-2016.02.18 5 1 399705 0 300.4mb 300.4mb yellow open logstash-2016.02.29 5 1 1211654 0 927.8mb 927.8mb yellow open .kibana 1 1 11 1 37.9kb 37.9kb yellow open logstash-2016.02.20 5 1 1326447 0 955.9mb 955.9mb yellow open logstash-2016.02.24 5 1 1866926 0 1.2gb 1.2gb yellow open logstash-2016.02.23 5 1 1737767 0 1.2gb 1.2gb yellow open logstash-2016.02.22 5 1 1506259 0 1gb 1gb yellow open logstash-2016.02.21 5 1 1366443 0 933.5mb 933.5mb [root@syslog ~]#

10

Should I try to get assistance on the Elasticsearch section, or does it more point to logstash this challenge?

11

Try changing your "INDEX" name to lowercase. Found LS has and issue with uppercase INDICES.

index => "idns-"
}

logstash.log
"reason"=>"Invalid index name [INDEX-2016.], must be lowercase"