Communication Problem between logstash and elasticsearch

Hi all,

I have some problems in intergrating logstash with elasticsearch.

My model is below:

(input Message Queue) logstash (output elastic search) -->
elasticsearch + logstash web.

Logstash and elasticsearch are installed on two separate servers.

After configuration, something was wrong. I use tcpdump to catch
packet from logstash server send to elastic server (port 9300). I
relize that have some metadata send to elastic server, but don't have
any log data.

Elasticsearch dump some warning log as:

[2011-10-28 10:44:13,940][WARN ][transport.netty ] [Fantastic
Four] Exception caught on netty layer [[id: 0x76659bde]]
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
592)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.run(NioClientSocketPipelineSink.java:276)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

If I search data from logstash web install on elasticsearch server, i
received message

The query 'text:a' resulted the following error:

org.elasticsearch.action.search.SearchPhaseExecutionException: Failed
to execute phase [initial], No indices / shards to search on,
requested indices are []

This may mean you have no data in ElasticSearch.

I think problem not at logstash server because if I stop logstash
process, and count message queue, I see some element in the queue. If
i start logstash process, none eleement in the queue.

Below is my configurations:

Logstash config:

input {
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "elastic"
}
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "graylog"
}
}

output {
stdout {
debug => true
}

If you can't discover using multicast, set the address explicitly

elasticsearch {
cluster => "elasticsearch"
host => "Ip address of Elastic server"
port => "9300"
index => "logstash-%{+YYYY.MM.dd}"
type => "raw_public_log"
}

}

Elasticsearch config

The cluster name

cluster.name: elasticsearch

#cluster:

name: MyCluster

network:
host: Ip address of Elastic server

Path Settings

#path.conf: /path/to/conf
#path.data: /path/to/data
path.data: /u03/elasticsearch/data/elasticsearch

path.work: /u03/elasticsearch/work/elasticsearch
#path.logs: /path/to/logs
path.logs: /u03/elasticsearch/logs/elasticsearch

Force all memory to be locked, forcing the JVM to never swap

#bootstrap.mlockall: true

node.master: true
node.data: true

Gateway Settings

Controls when to start the initial recovery process when starting

a new cluster

allowing for better reused of existing data during recovery.

#gateway.recover_after_nodes: 1
#gateway.recover_after_time: 5m
#gateway.expected_nodes: 2

Controls the minimum number of master eligible nodes this node

should "see"

in order to operate within the cluster.

Set this to a higher value (2-4) when running more than 2 nodes in

the cluster
#discovery.zen.minimum_master_nodes: 1

The time to wait for ping responses from other nodes when doing node

discovery
#discovery.zen.ping.timeout: 3s

Unicast Discovery (disable multicast)

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts: ["Ip address of Elastic server"]

So I have some questions:

  1. How logstash communicate with elasticsearch?
  2. What is my problem and how to resolve it?
  3. Can you recommend some detail document for elasticsearch (and
    logstash)?

Thanks!

Probably this is more a logstash config related question? Did you ask
on their mailinglist too?

Peter

On 28 Okt., 05:02, medas3c medas3c meda...@gmail.com wrote:

Hi all,

I have some problems in intergrating logstash with elasticsearch.

My model is below:

(input Message Queue) logstash (output Elasticsearch) -->
elasticsearch + logstash web.

Logstash and elasticsearch are installed on two separate servers.

After configuration, something was wrong. I use tcpdump to catch
packet from logstash server send to elastic server (port 9300). I
relize that have some metadata send to elastic server, but don't have
any log data.

Elasticsearch dump some warning log as:

[2011-10-28 10:44:13,940][WARN ][transport.netty ] [Fantastic
Four] Exception caught on netty layer [[id: 0x76659bde]]
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
592)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.run(NioClientSocketPipelineSink.java:276)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

If I search data from logstash web install on elasticsearch server, i
received message

The query 'text:a' resulted the following error:

org.elasticsearch.action.search.SearchPhaseExecutionException: Failed
to execute phase [initial], No indices / shards to search on,
requested indices are

This may mean you have no data in Elasticsearch.

I think problem not at logstash server because if I stop logstash
process, and count message queue, I see some element in the queue. If
i start logstash process, none eleement in the queue.

Below is my configurations:

Logstash config:

input {
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "elastic"
}
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "graylog"
}

}

output {
stdout {
debug => true
}

If you can't discover using multicast, set the address explicitly

elasticsearch {
cluster => "elasticsearch"
host => "Ip address of Elastic server"
port => "9300"
index => "logstash-%{+YYYY.MM.dd}"
type => "raw_public_log"
}

}


Elasticsearch config

The cluster name

cluster.name: elasticsearch

#cluster:

name: MyCluster

network:
host: Ip address of Elastic server

Path Settings

#path.conf: /path/to/conf
#path.data: /path/to/data
path.data: /u03/elasticsearch/data/elasticsearch

path.work: /u03/elasticsearch/work/elasticsearch
#path.logs: /path/to/logs
path.logs: /u03/elasticsearch/logs/elasticsearch

Force all memory to be locked, forcing the JVM to never swap

#bootstrap.mlockall: true

node.master: true
node.data: true

Gateway Settings

Controls when to start the initial recovery process when starting

a new cluster

allowing for better reused of existing data during recovery.

#gateway.recover_after_nodes: 1
#gateway.recover_after_time: 5m
#gateway.expected_nodes: 2

Controls the minimum number of master eligible nodes this node

should "see"

in order to operate within the cluster.

Set this to a higher value (2-4) when running more than 2 nodes in

the cluster
#discovery.zen.minimum_master_nodes: 1

The time to wait for ping responses from other nodes when doing node

discovery
#discovery.zen.ping.timeout: 3s

Unicast Discovery (disable multicast)

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts: ["Ip address of Elastic server"]

So I have some questions:

  1. How logstash communicate with elasticsearch?
  2. What is my problem and how to resolve it?
  3. Can you recommend some detail document for elasticsearch (and
    logstash)?

Thanks!

I have posted it on logstash user groups
https://groups.google.com/group/logstash-users/browse_thread/thread/a78ebf73e26b20b0

and the problem has been resolved.

Thanks you.

On Oct 31, 3:28 pm, Karussell tableyourt...@googlemail.com wrote:

Probably this is more a logstash config related question? Did you ask
on their mailinglist too?

Peter

On 28 Okt., 05:02, medas3c medas3c meda...@gmail.com wrote:

Hi all,

I have some problems in intergrating logstash with elasticsearch.

My model is below:

(input Message Queue) logstash (output Elasticsearch) -->
elasticsearch + logstash web.

Logstash and elasticsearch are installed on two separate servers.

After configuration, something was wrong. I use tcpdump to catch
packet from logstash server send to elastic server (port 9300). I
relize that have some metadata send to elastic server, but don't have
any log data.

Elasticsearch dump some warning log as:

[2011-10-28 10:44:13,940][WARN ][transport.netty ] [Fantastic
Four] Exception caught on netty layer [[id: 0x76659bde]]
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
592)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.run(NioClientSocketPipelineSink.java:276)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

If I search data from logstash web install on elasticsearch server, i
received message

The query 'text:a' resulted the following error:

org.elasticsearch.action.search.SearchPhaseExecutionException: Failed
to execute phase [initial], No indices / shards to search on,
requested indices are

This may mean you have no data in Elasticsearch.

I think problem not at logstash server because if I stop logstash
process, and count message queue, I see some element in the queue. If
i start logstash process, none eleement in the queue.

Below is my configurations:

Logstash config:

input {
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "elastic"
}
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "graylog"
}

}

output {
stdout {
debug => true
}

If you can't discover using multicast, set the address explicitly

elasticsearch {
cluster => "elasticsearch"
host => "Ip address of Elastic server"
port => "9300"
index => "logstash-%{+YYYY.MM.dd}"
type => "raw_public_log"
}

}


Elasticsearch config

The cluster name

cluster.name: elasticsearch

#cluster:

name: MyCluster

network:
host: Ip address of Elastic server

Path Settings

#path.conf: /path/to/conf
#path.data: /path/to/data
path.data: /u03/elasticsearch/data/elasticsearch

path.work: /u03/elasticsearch/work/elasticsearch
#path.logs: /path/to/logs
path.logs: /u03/elasticsearch/logs/elasticsearch

Force all memory to be locked, forcing the JVM to never swap

#bootstrap.mlockall: true

node.master: true
node.data: true

Gateway Settings

Controls when to start the initial recovery process when starting

a new cluster

allowing for better reused of existing data during recovery.

#gateway.recover_after_nodes: 1
#gateway.recover_after_time: 5m
#gateway.expected_nodes: 2

Controls the minimum number of master eligible nodes this node

should "see"

in order to operate within the cluster.

Set this to a higher value (2-4) when running more than 2 nodes in

the cluster
#discovery.zen.minimum_master_nodes: 1

The time to wait for ping responses from other nodes when doing node

discovery
#discovery.zen.ping.timeout: 3s

Unicast Discovery (disable multicast)

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts: ["Ip address of Elastic server"]

So I have some questions:

  1. How logstash communicate with elasticsearch?
  2. What is my problem and how to resolve it?
  3. Can you recommend some detail document for elasticsearch (and
    logstash)?

Thanks!