Communication Problem between logstash and elasticsearch


(medas3c medas3c) #1

Hi all,

I have some problems in intergrating logstash with elasticsearch.

My model is below:

(input Message Queue) logstash (output elastic search) -->
elasticsearch + logstash web.

Logstash and elasticsearch are installed on two separate servers.

After configuration, something was wrong. I use tcpdump to catch
packet from logstash server send to elastic server (port 9300). I
relize that have some metadata send to elastic server, but don't have
any log data.

Elasticsearch dump some warning log as:

[2011-10-28 10:44:13,940][WARN ][transport.netty ] [Fantastic
Four] Exception caught on netty layer [[id: 0x76659bde]]
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
592)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.run(NioClientSocketPipelineSink.java:276)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

If I search data from logstash web install on elasticsearch server, i
received message

The query 'text:a' resulted the following error:

org.elasticsearch.action.search.SearchPhaseExecutionException: Failed
to execute phase [initial], No indices / shards to search on,
requested indices are []

This may mean you have no data in ElasticSearch.

I think problem not at logstash server because if I stop logstash
process, and count message queue, I see some element in the queue. If
i start logstash process, none eleement in the queue.

Below is my configurations:

Logstash config:

input {
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "elastic"
}
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "graylog"
}
}

output {
stdout {
debug => true
}

If you can't discover using multicast, set the address explicitly

elasticsearch {
cluster => "elasticsearch"
host => "Ip address of Elastic server"
port => "9300"
index => "logstash-%{+YYYY.MM.dd}"
type => "raw_public_log"
}

}

Elasticsearch config

The cluster name

cluster.name: elasticsearch

#cluster:

name: MyCluster

network:
host: Ip address of Elastic server

Path Settings

#path.conf: /path/to/conf
#path.data: /path/to/data
path.data: /u03/elasticsearch/data/elasticsearch

path.work: /u03/elasticsearch/work/elasticsearch
#path.logs: /path/to/logs
path.logs: /u03/elasticsearch/logs/elasticsearch

Force all memory to be locked, forcing the JVM to never swap

#bootstrap.mlockall: true

node.master: true
node.data: true

Gateway Settings

Controls when to start the initial recovery process when starting

a new cluster

allowing for better reused of existing data during recovery.

#gateway.recover_after_nodes: 1
#gateway.recover_after_time: 5m
#gateway.expected_nodes: 2

Controls the minimum number of master eligible nodes this node

should "see"

in order to operate within the cluster.

Set this to a higher value (2-4) when running more than 2 nodes in

the cluster
#discovery.zen.minimum_master_nodes: 1

The time to wait for ping responses from other nodes when doing node

discovery
#discovery.zen.ping.timeout: 3s

Unicast Discovery (disable multicast)

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts: ["Ip address of Elastic server"]

So I have some questions:

  1. How logstash communicate with elasticsearch?
  2. What is my problem and how to resolve it?
  3. Can you recommend some detail document for elasticsearch (and
    logstash)?

Thanks!


(Karussell) #2

Probably this is more a logstash config related question? Did you ask
on their mailinglist too?

Peter

On 28 Okt., 05:02, medas3c medas3c meda...@gmail.com wrote:

Hi all,

I have some problems in intergrating logstash with elasticsearch.

My model is below:

(input Message Queue) logstash (output elastic search) -->
elasticsearch + logstash web.

Logstash and elasticsearch are installed on two separate servers.

After configuration, something was wrong. I use tcpdump to catch
packet from logstash server send to elastic server (port 9300). I
relize that have some metadata send to elastic server, but don't have
any log data.

Elasticsearch dump some warning log as:

[2011-10-28 10:44:13,940][WARN ][transport.netty ] [Fantastic
Four] Exception caught on netty layer [[id: 0x76659bde]]
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
592)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.run(NioClientSocketPipelineSink.java:276)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

If I search data from logstash web install on elasticsearch server, i
received message

The query 'text:a' resulted the following error:

org.elasticsearch.action.search.SearchPhaseExecutionException: Failed
to execute phase [initial], No indices / shards to search on,
requested indices are []

This may mean you have no data in ElasticSearch.

I think problem not at logstash server because if I stop logstash
process, and count message queue, I see some element in the queue. If
i start logstash process, none eleement in the queue.

Below is my configurations:

Logstash config:

input {
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "elastic"
}
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "graylog"
}

}

output {
stdout {
debug => true
}

If you can't discover using multicast, set the address explicitly

elasticsearch {
cluster => "elasticsearch"
host => "Ip address of Elastic server"
port => "9300"
index => "logstash-%{+YYYY.MM.dd}"
type => "raw_public_log"
}

}


Elasticsearch config

The cluster name

cluster.name: elasticsearch

#cluster:

name: MyCluster

network:
host: Ip address of Elastic server

Path Settings

#path.conf: /path/to/conf
#path.data: /path/to/data
path.data: /u03/elasticsearch/data/elasticsearch

path.work: /u03/elasticsearch/work/elasticsearch
#path.logs: /path/to/logs
path.logs: /u03/elasticsearch/logs/elasticsearch

Force all memory to be locked, forcing the JVM to never swap

#bootstrap.mlockall: true

node.master: true
node.data: true

Gateway Settings

Controls when to start the initial recovery process when starting

a new cluster

allowing for better reused of existing data during recovery.

#gateway.recover_after_nodes: 1
#gateway.recover_after_time: 5m
#gateway.expected_nodes: 2

Controls the minimum number of master eligible nodes this node

should "see"

in order to operate within the cluster.

Set this to a higher value (2-4) when running more than 2 nodes in

the cluster
#discovery.zen.minimum_master_nodes: 1

The time to wait for ping responses from other nodes when doing node

discovery
#discovery.zen.ping.timeout: 3s

Unicast Discovery (disable multicast)

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts: ["Ip address of Elastic server"]

So I have some questions:

  1. How logstash communicate with elasticsearch?
  2. What is my problem and how to resolve it?
  3. Can you recommend some detail document for elasticsearch (and
    logstash)?

Thanks!


(medas3c medas3c) #3

I have posted it on logstash user groups
https://groups.google.com/group/logstash-users/browse_thread/thread/a78ebf73e26b20b0

and the problem has been resolved.

Thanks you.

On Oct 31, 3:28 pm, Karussell tableyourt...@googlemail.com wrote:

Probably this is more a logstash config related question? Did you ask
on their mailinglist too?

Peter

On 28 Okt., 05:02, medas3c medas3c meda...@gmail.com wrote:

Hi all,

I have some problems in intergrating logstash with elasticsearch.

My model is below:

(input Message Queue) logstash (output elastic search) -->
elasticsearch + logstash web.

Logstash and elasticsearch are installed on two separate servers.

After configuration, something was wrong. I use tcpdump to catch
packet from logstash server send to elastic server (port 9300). I
relize that have some metadata send to elastic server, but don't have
any log data.

Elasticsearch dump some warning log as:

[2011-10-28 10:44:13,940][WARN ][transport.netty ] [Fantastic
Four] Exception caught on netty layer [[id: 0x76659bde]]
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
592)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.run(NioClientSocketPipelineSink.java:276)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

If I search data from logstash web install on elasticsearch server, i
received message

The query 'text:a' resulted the following error:

org.elasticsearch.action.search.SearchPhaseExecutionException: Failed
to execute phase [initial], No indices / shards to search on,
requested indices are []

This may mean you have no data in ElasticSearch.

I think problem not at logstash server because if I stop logstash
process, and count message queue, I see some element in the queue. If
i start logstash process, none eleement in the queue.

Below is my configurations:

Logstash config:

input {
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "elastic"
}
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "graylog"
}

}

output {
stdout {
debug => true
}

If you can't discover using multicast, set the address explicitly

elasticsearch {
cluster => "elasticsearch"
host => "Ip address of Elastic server"
port => "9300"
index => "logstash-%{+YYYY.MM.dd}"
type => "raw_public_log"
}

}


Elasticsearch config

The cluster name

cluster.name: elasticsearch

#cluster:

name: MyCluster

network:
host: Ip address of Elastic server

Path Settings

#path.conf: /path/to/conf
#path.data: /path/to/data
path.data: /u03/elasticsearch/data/elasticsearch

path.work: /u03/elasticsearch/work/elasticsearch
#path.logs: /path/to/logs
path.logs: /u03/elasticsearch/logs/elasticsearch

Force all memory to be locked, forcing the JVM to never swap

#bootstrap.mlockall: true

node.master: true
node.data: true

Gateway Settings

Controls when to start the initial recovery process when starting

a new cluster

allowing for better reused of existing data during recovery.

#gateway.recover_after_nodes: 1
#gateway.recover_after_time: 5m
#gateway.expected_nodes: 2

Controls the minimum number of master eligible nodes this node

should "see"

in order to operate within the cluster.

Set this to a higher value (2-4) when running more than 2 nodes in

the cluster
#discovery.zen.minimum_master_nodes: 1

The time to wait for ping responses from other nodes when doing node

discovery
#discovery.zen.ping.timeout: 3s

Unicast Discovery (disable multicast)

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts: ["Ip address of Elastic server"]

So I have some questions:

  1. How logstash communicate with elasticsearch?
  2. What is my problem and how to resolve it?
  3. Can you recommend some detail document for elasticsearch (and
    logstash)?

Thanks!


(system) #4