Upgrading to ELK 2 connectivity problems


(Philip Thomson) #1

Ok, I have an issue I suspect in an elasticsearch problem but I could be wrong,

So I have a CENT0S 6 box that has Elasticsearch 2.1.1 installed, this is a fresh install via YUM, and not an upgrade of my existing ELK stack.

I am unable to have either Logstash 2 or another elasticsearch 2 machine connect to this.

I have enabled the bind to the lan IP using network.host, firewalls, and selinux are disabled, and even with additional logging there is nothing in the logs,
I am able to connect to check stats using "/_nodes" from any machine on the network .

In logstash 2 on a 2nd machine the the logs simply fill up with :
> message=>"Flushing buffer at interval", :instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x3c6bf360 @operations_mutex=#<Mutex:0x32ac0a9d>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x36939810>, @submit_proc=#<Proc:0x10021d47@/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.2.0-java/lib/logstash/outputs/elasticsearch/common.rb:55>, @logger=#<Cabin::Channel:0x42281538 @metrics=#<Cabin::Metrics:0x68515675 @metrics_lock=#<Mutex:0x13beffe9>, @metrics={}, @channel=#<Cabin::Channel:0x42281538 ...>>, @subscriber_lock=#<Mutex:0x13d630fc>, @level=:debug, @subscribers={12694=>#<Cabin::Outputs::IO:0x735578ca @io=#<File:/var/log/logstash/logstash.log>, @lock=#<Mutex:0x17f080fe>>}, @data={}>, @last_flush=2015-12-28 11:36:42 -0500, @flush_interval=1, @stopping=#<Concurrent::AtomicBoolean:0x3583132c>, @buffer=[], @flush_thread=#<Thread:0xf5fcdd9 run>>", :interval=>1, :level=>:info, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}

Does anyone have any idea's on where to goto next?


(Philip Thomson) #2

if it helps here is the current status of the ES2 via the API:

{ "cluster_name" : "inboxlog", "nodes" : { "bs3Lwv_9TaeCcvV2o2es9w" : { "name" : "Angela Del Toro", "transport_address" : "192.168.76.250:9300", "host" : "192.168.76.250", "ip" : "192.168.76.250", "version" : "2.1.1", "build" : "40e2c53", "http_address" : "192.168.76.250:9200", "settings" : { "client" : { "type" : "node" }, "http" : { "port" : "9200" }, "name" : "Angela Del Toro", "pidfile" : "/var/run/elasticsearch/elasticsearch.pid", "path" : { "data" : "/logdata/", "home" : "/usr/share/elasticsearch", "conf" : "/etc/elasticsearch", "logs" : "/var/log/elasticsearch" }, "config" : { "ignore_system_properties" : "true" }, "cluster" : { "name" : "inboxlog" }, "network" : { "host" : "192.168.76.250" }, "foreground" : "false" }, "os" : { "refresh_interval_in_millis" : 1000, "name" : "Linux", "arch" : "amd64", "version" : "2.6.32-504.3.3.el6.x86_64", "available_processors" : 8, "allocated_processors" : 8 }, "process" : { "refresh_interval_in_millis" : 1000, "id" : 2199, "mlockall" : false }, "jvm" : { "pid" : 2199, "version" : "1.7.0_71", "vm_name" : "OpenJDK 64-Bit Server VM", "vm_version" : "24.65-b04", "vm_vendor" : "Oracle Corporation", "start_time_in_millis" : 1451322596346, "mem" : { "heap_init_in_bytes" : 16106127360, "heap_max_in_bytes" : 16036397056, "non_heap_init_in_bytes" : 24313856, "non_heap_max_in_bytes" : 224395264, "direct_max_in_bytes" : 16036397056 }, "gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ], "memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ] }, / /I removed stuff here// }, "transport" : { "bound_address" : [ "192.168.76.250:9300" ], "publish_address" : "192.168.76.250:9300", "profiles" : { } }, "http" : { "bound_address" : [ "192.168.76.250:9200" ], "publish_address" : "192.168.76.250:9200", "max_content_length_in_bytes" : 104857600 }, "plugins" : [ { "name" : "discovery-multicast", "version" : "2.1.1", "description" : "The Multicast Discovery plugin allows discovery other nodes using multicast requests", "jvm" : true, "classname" : "org.elasticsearch.plugin.discovery.multicast.MulticastDiscoveryPlugin", "isolated" : true, "site" : false }, { "name" : "kopf", "version" : "2.1.1", "description" : "kopf - simple web administration tool for Elasticsearch", "url" : "/_plugin/kopf/", "jvm" : false, "site" : true } ] } } }


(Nitesh) #3

Hi Philip, How this was resolved?


(system) #4