Hi all,
I have some problems in intergrating logstash with elasticsearch.
My model is below:
(input Message Queue) logstash (output elastic search) -->
elasticsearch + logstash web.
Logstash and elasticsearch are installed on two separate servers.
After configuration, something was wrong. I use tcpdump to catch
packet from logstash server send to elastic server (port 9300). I
relize that have some metadata send to elastic server, but don't have
any log data.
Elasticsearch dump some warning log as:
[2011-10-28 10:44:13,940][WARN ][transport.netty ] [Fantastic
Four] Exception caught on netty layer [[id: 0x76659bde]]
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
592)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.run(NioClientSocketPipelineSink.java:276)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
If I search data from logstash web install on elasticsearch server, i
received message
The query 'text:a' resulted the following error:
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed
to execute phase [initial], No indices / shards to search on,
requested indices are []
This may mean you have no data in ElasticSearch.
I think problem not at logstash server because if I stop logstash
process, and count message queue, I see some element in the queue. If
i start logstash process, none eleement in the queue.
Below is my configurations:
Logstash config:
input {
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "elastic"
}
amqp {
# ship logs to the 'rawlogs' fanout queue.
debug => true
type => "all"
host => "127.0.0.1"
exchange_type => "fanout"
name => "graylog"
}
}
output {
stdout {
debug => true
}
If you can't discover using multicast, set the address explicitly
elasticsearch {
cluster => "elasticsearch"
host => "Ip address of Elastic server"
port => "9300"
index => "logstash-%{+YYYY.MM.dd}"
type => "raw_public_log"
}
}
Elasticsearch config
The cluster name
cluster.name: elasticsearch
#cluster:
name: MyCluster
network:
host: Ip address of Elastic server
Path Settings
#path.conf: /path/to/conf
#path.data: /path/to/data
path.data: /u03/elasticsearch/data/elasticsearch
path.work: /u03/elasticsearch/work/elasticsearch
#path.logs: /path/to/logs
path.logs: /u03/elasticsearch/logs/elasticsearch
Force all memory to be locked, forcing the JVM to never swap
#bootstrap.mlockall: true
node.master: true
node.data: true
Gateway Settings
Controls when to start the initial recovery process when starting
a new cluster
allowing for better reused of existing data during recovery.
#gateway.recover_after_nodes: 1
#gateway.recover_after_time: 5m
#gateway.expected_nodes: 2
Controls the minimum number of master eligible nodes this node
should "see"
in order to operate within the cluster.
Set this to a higher value (2-4) when running more than 2 nodes in
the cluster
#discovery.zen.minimum_master_nodes: 1
The time to wait for ping responses from other nodes when doing node
discovery
#discovery.zen.ping.timeout: 3s
Unicast Discovery (disable multicast)
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts: ["Ip address of Elastic server"]
So I have some questions:
- How logstash communicate with elasticsearch?
- What is my problem and how to resolve it?
- Can you recommend some detail document for elasticsearch (and
logstash)?
Thanks!