Elasticsearch output cluster_name

Elastic Stack Logstash
1

Before upgrading to version 2.0 I was able to reference my elasticsearch cluster in my logstash conf file with the cluster name option. With the upgrade to ES 2.0 and logstash 2.o that option is no longer available. Now I get

message=>"Failed to flush outgoing items", :outgoing_count=>8, :exception=>org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master]

errors when I start my logstash nodes and eventually they die off. I left all things the same with my logstash conf file with the exception of removing the cluster name section because --configtest stated it could no longer be used.

How do I get my logstash nodes to join my elasticsearch cluster now. I do have the output set to

hosts => ["domain name"]

Thanks

2

Are you 100% sure that error message is from Logstash 2.0? Because it suggests that you're trying to connect using the node or transport protocols which aren't supported in 2.0's elasticsearch output.

3

/opt/logstash/bin/logstash --version
logstash 2.0.0

output config:

output {
if [type] == "BRO" {
if [XXX] == "XXX" {
elasticsearch {
hosts => [ "DOMAIN NAME" ]
index => "cobalt-bro-%{+YYYY.MM.dd}"
}
}
}
}

Also, I am not running elasticsearch on my logstash index nodes. I never have. Logstash has always joined my ES cluster as logstash nodes.

Not sure where I broke it with the 2.0 upgrade

4

Logstash 2.0 does not use the node or transport protocols by default. This is not recommended for Elasticsearch 2.0. If you insist on using the node protocol, you must install the logstash-output-elasticsearch_java plugin, which has those features separated out.

bin/plugin install logstash-output-elasticsearch_java

And then:

elasticsearch_java {
  hosts => [ "DOMAIN NAME" ]
  index => "cobalt-bro-%{+YYYY.MM.dd}"
}
5

What is the preferred way to do this in 2. . I would rather do thing the right way than just use a work around.

Is there a way to get my logstash nodes to join my ES cluster?

6

The preferred way is to have them not join as nodes, but speak to an ES client (any node can work) via the http protocol. This is what the elasticsearch output block does by default now.

As stated in my previous post, if you desire to use node protocol still, you have to install a separate plugin and change the configuration to use the elasticsearch_java output.

7

Thank you. I will continue to troubleshoot the issue. Don't want to use elasticsearch_java if I don't have to.

Thanks for the help

8

I'd like to revisit this for a moment.

This is my current output for my log data from my LS indexers (4 of them) into my 10 node cluster. Each node has 32 GB of ram and plenty of HDD space.

output {
  if [type] == "BRO" {
    if [sensor1] == "sensor" {
      elasticsearch {
        hosts => [ "es-op-01-cl-55" ]
        manage_template => false
        index => "sensor1-bro-%{+YYYY.MM.dd}"
      }
    }
  }
}

Question: the referenced domain name has all 10 ES nodes assigned to it. My thought was that LS would treat them as an array, send data to each as it needed, but I'm wondering if it is working that way. Is it better to list out the IP's of each node rather than have them in a domain?

I ask because it appears that my LS nodes are getting overwhelmed, I get the following error on the indexer nodes quite often and constant:

{:timestamp=>"2015-11-19T06:33:32.651000-0500", :message=>"retrying failed action with response code: 429", :level=>:warn}

My ES nodes get the following error:

[2015-11-19 06:34:02,734][DEBUG][action.bulk              ] [WORKER_NODE_1] [sensor1-bro-2015.11.15][29] failed to execute bulk item (index) index {[sensor1-bro-2015.11.15][BRO][AVEfhVzWtyozxrGteR4H], source[{"ts":"2015-11-15T21:02:02.785421Z","uid":"CPztEE1K5ieUuKMpg","trans_depth":2,"method":"GET","host":"au.v4.download.windowsupdate.com","uri":"/msdownload/update/driver/drvs/2012/12/20288081_f78ee4b887c9c6047312311a6ffcb861dec5154e.cab","user_agent":"Microsoft BITS/7.8","request_body_len":0,"response_body_len":5630,"status_code":206,"status_msg":"Partial Content","tags":[],"resp_fuids":["FvVBtp1BvGH71O9wR4"],"resp_mime_types":["application/vnd.ms-cab-compressed"],"@version":"1","@timestamp":"2015-11-15T21:02:02.785Z","path":"/nsm/bro/logs/current/http_eth2.log","type":"BRO","csp_sensor":"cobalt","log_path":"http_eth2","src_ip":"192.168.10.143","src_port":49508,"resp_ip":"23.62.7.146","resp_port":80,"geoip_resp":{"ip":"23.62.7.146","country_code2":"US","country_code3":"USA","country_name":"United States","continent_code":"NA","region_name":"MA","city_name":"Cambridge","postal_code":"02142","latitude":42.362599999999986,"longitude":-71.0843,"dma_code":506,"area_code":617,"timezone":"America/New_York","real_region_name":"Massachusetts","location":[-71.0843,42.362599999999986]},"resp_Senderbase_lookup":"http://www.senderbase.org/lookup/?search_string=23.62.7.146","resp_CBL_lookup":"http://cbl.abuseat.org/lookup.cgi?ip=23.62.7.146","resp_Spamhaus_lookup":"http://www.spamhaus.org/query/bl?ip=23.62.7.146","resp_DomainTools_lookup":"http://whois.domaintools.com/23.62.7.146","url_full":"au.v4.download.windowsupdate.com/msdownload/update/driver/drvs/2012/12/20288081_f78ee4b887c9c6047312311a6ffcb861dec5154e.cab"}]}

The reason I wanted to use the domain with multiple IP's assigned to it was to make it easier to add nodes for scaling. By using the domain I'd be able to add a node to the domain without having to change any configs on the nodes themselves.

Also, my input looks like this:

input {
  redis {
    host => [ "HOST1" ]
    data_type => "list"
    key => "bro"
  }

repeated 14 times for a 14 node redis cluster. This config is on each of my 4 LS indexers. My thought was that I'd have 4 LS indexers accepting connections from all my redis nodes at all times to take messages as they become available. They would then do their filtering and pass the data to the ES cluster. I am not pushing much data at this time so overwhelming the LS or ES nodes seemed unlikely but I am getting these errors.

Should I:

  1. Add more indexers?
  2. Split my redis inputs among my indexers so each indexer accepts connections from a subset of redis nodes?
  3. list out my hosts in the output section of my elasticsearch output?
  4. all of the above
  5. get a new job where I don't have to deal with this daily?

Thanks

9

Question: the referenced domain name has all 10 ES nodes assigned to it. My thought was that LS would treat them as an array, send data to each as it needed, but I'm wondering if it is working that way. Is it better to list out the IP's of each node rather than have them in a domain?

What does "have them in a domain" mean, exactly? Do you have a DNS CNAME or A record that points to multiple hosts? I'm not sure how Logstash or its HTTP libraries respond to that. It'll work, but will it retry with the next host if the first one is down?

I ask because it appears that my LS nodes are getting overwhelmed

Elasticsearch is overwhelmed, not Logstash.

10

Thank you. Yes I have an A record with all my ES nodes pointed at it. I
thought (and obviously its novice error) that it would be able to send the
data to all hosts in the A records and that it would be easy to add and
subtract hosts if needed.

I am in the process of editing the configs to list the IP's in an array:

hosts => ["host1","host2","host3",.....]

I am interested to see if that helps take the load of the ES cluster. I
wouldn't have thought with what little data I am pushing that I could so
easily overwhelm the cluster but I must have a misconfiguration somewhere.

Thanks