I am using ELK stack for log processing. Logstash 1.4.2 (Single Instance)
and Elasticsearch 1.2.2(Cluster of 2 nodes), and redis as broker between
logstash and elasticsearch.
I am able to parse my logs using Logstash, get the parsed document in
Redis. But not able to get the documents inside Elasticsearch.
The following is my Central logstash server configuration file.
input { redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5} redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5}} filter { date { match =>
["timestamp_nsstats",%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[,]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?"]
} } output { stdout { } elasticsearch
{ cluster => "logstash" host =>
"xx.xx.xx.xx" protocol => "http" node_name =>
"Node1" index => "logstash-%{+YYYY.MM.dd}"} }
The following is the health of Elasticsearch Instances.
{
"cluster_name": "logstash",
"status": "green",
"timed_out": false,
"number_of_nodes": 2,
"number_of_data_nodes": 2,
"active_primary_shards": 3,
"active_shards": 6,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}
I am not able to find out the reason, that is causing logstash being
not able to reach elasticsearch. Do I need to make any changes in
configuration file, which might make logstash aware of the
elasticsearch server.
Please let me know what you think, that might be causing the issue. If
you need any other information please let me know.
You should probably start this conversation on the logstash list.
However if you are using the http output then you don't need cluster or
node_name. It'd be worth testing that you can telnet from your LS host to
your ES host on port 9200, and also try enabling verbose logging in LS with
-v.
I am using ELK stack for log processing. Logstash 1.4.2 (Single Instance)
and Elasticsearch 1.2.2(Cluster of 2 nodes), and redis as broker between
logstash and elasticsearch.
I am able to parse my logs using Logstash, get the parsed document in
Redis. But not able to get the documents inside Elasticsearch.
The following is my Central logstash server configuration file.
input { redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5 } redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5 }} filter { date { match =>
["timestamp_nsstats",%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[,]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?"]
} } output { stdout { } elasticsearch
{ cluster => "logstash" host =>
"xx.xx.xx.xx" protocol => "http" node_name =>
"Node1" index => "logstash-%{+YYYY.MM.dd}"} }
The following is the health of Elasticsearch Instances.
{
"cluster_name": "logstash",
"status": "green",
"timed_out": false,
"number_of_nodes": 2,
"number_of_data_nodes": 2,
"active_primary_shards": 3,
"active_shards": 6,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}
I am not able to find out the reason, that is causing logstash being not able to reach elasticsearch. Do I need to make any changes in configuration file, which might make logstash aware of the elasticsearch server.
Please let me know what you think, that might be causing the issue. If you need any other information please let me know.
Hey I started the conversation on logstash list as suggested by you. I
tried testing to telnet from the Logstash central server to ES server.
Following is the output I am getting.
$ telnet 10.12.7.2 9200
Trying 10.12.7.2...
Connected to xxxxxxxxxxxx (10.12.7.2).
Escape character is '^]'.
On Tuesday, July 15, 2014 3:22:43 PM UTC-7, Mark Walkom wrote:
You should probably start this conversation on the logstash list.
However if you are using the http output then you don't need cluster or
node_name. It'd be worth testing that you can telnet from your LS host to
your ES host on port 9200, and also try enabling verbose logging in LS with
-v.
On 16 July 2014 07:15, Voc Austin <voc...@gmail.com <javascript:>> wrote:
Hey,
I am using ELK stack for log processing. Logstash 1.4.2 (Single Instance)
and Elasticsearch 1.2.2(Cluster of 2 nodes), and redis as broker between
logstash and elasticsearch.
I am able to parse my logs using Logstash, get the parsed document in
Redis. But not able to get the documents inside Elasticsearch.
The following is my Central logstash server configuration file.
input { redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5 } redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5 }} filter { date { match =>
["timestamp_nsstats",%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[,]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?"]
} } output { stdout { } elasticsearch
{ cluster => "logstash" host =>
"xx.xx.xx.xx" protocol => "http" node_name =>
"Node1" index => "logstash-%{+YYYY.MM.dd}"} }
The following is the health of Elasticsearch Instances.
{
"cluster_name": "logstash",
"status": "green",
"timed_out": false,
"number_of_nodes": 2,
"number_of_data_nodes": 2,
"active_primary_shards": 3,
"active_shards": 6,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}
I am not able to find out the reason, that is causing logstash being not able to reach elasticsearch. Do I need to make any changes in configuration file, which might make logstash aware of the elasticsearch server.
Please let me know what you think, that might be causing the issue. If you need any other information please let me know.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.