Issue Mapping Elasticsearch


(asal masuk) #1

Hey,

I am using ELK stack for log processing. Logstash 1.4.2 (Single Instance)
and Elasticsearch 1.2.2(Cluster of 2 nodes), and redis as broker between
logstash and elasticsearch.

I am able to parse my logs using Logstash, get the parsed document in
Redis. But not able to get the documents inside Elasticsearch.

The following is my Central logstash server configuration file.

input { redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5} redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5}} filter { date { match =>
["timestamp_nsstats",%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[,]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?"]
} } output { stdout { } elasticsearch
{ cluster => "logstash" host =>
"xx.xx.xx.xx" protocol => "http" node_name =>
"Node1" index => "logstash-%{+YYYY.MM.dd}"} }

The following is the health of Elasticsearch Instances.

  1. {
  2. "cluster_name": "logstash",
  3. "status": "green",
  4. "timed_out": false,
  5. "number_of_nodes": 2,
  6. "number_of_data_nodes": 2,
  7. "active_primary_shards": 3,
  8. "active_shards": 6,
  9. "relocating_shards": 0,
  10. "initializing_shards": 0,
  11. "unassigned_shards": 0
  12. }

I am not able to find out the reason, that is causing logstash being
not able to reach elasticsearch. Do I need to make any changes in
configuration file, which might make logstash aware of the
elasticsearch server.

Please let me know what you think, that might be causing the issue. If
you need any other information please let me know.

Thank you

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKKFHoG3_rY%3Dtx6x7ibmXVRk7btXw_Xaxv_S-FEMHX0b_g8-oA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

You should probably start this conversation on the logstash list.
However if you are using the http output then you don't need cluster or
node_name. It'd be worth testing that you can telnet from your LS host to
your ES host on port 9200, and also try enabling verbose logging in LS with
-v.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 16 July 2014 07:15, Voc Austin vocasal@gmail.com wrote:

Hey,

I am using ELK stack for log processing. Logstash 1.4.2 (Single Instance)
and Elasticsearch 1.2.2(Cluster of 2 nodes), and redis as broker between
logstash and elasticsearch.

I am able to parse my logs using Logstash, get the parsed document in
Redis. But not able to get the documents inside Elasticsearch.

The following is my Central logstash server configuration file.

input { redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5 } redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5 }} filter { date { match =>
["timestamp_nsstats",%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[,]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?"]
} } output { stdout { } elasticsearch
{ cluster => "logstash" host =>
"xx.xx.xx.xx" protocol => "http" node_name =>
"Node1" index => "logstash-%{+YYYY.MM.dd}"} }

The following is the health of Elasticsearch Instances.

  1. {

  2. "cluster_name": "logstash",

  3. "status": "green",

  4. "timed_out": false,

  5. "number_of_nodes": 2,

  6. "number_of_data_nodes": 2,

  7. "active_primary_shards": 3,

  8. "active_shards": 6,

  9. "relocating_shards": 0,

  10. "initializing_shards": 0,

  11. "unassigned_shards": 0

  12. }

I am not able to find out the reason, that is causing logstash being not able to reach elasticsearch. Do I need to make any changes in configuration file, which might make logstash aware of the elasticsearch server.

Please let me know what you think, that might be causing the issue. If you need any other information please let me know.

Thank you

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKKFHoG3_rY%3Dtx6x7ibmXVRk7btXw_Xaxv_S-FEMHX0b_g8-oA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKKFHoG3_rY%3Dtx6x7ibmXVRk7btXw_Xaxv_S-FEMHX0b_g8-oA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624befEXSocmRZo-rniVkoeP%3DpgO0YLwHS-m%3Dmb1EAeZNow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(shriyansh jain) #3

Hey I started the conversation on logstash list as suggested by you. I
tried testing to telnet from the Logstash central server to ES server.
Following is the output I am getting.

$ telnet 10.12.7.2 9200
Trying 10.12.7.2...
Connected to xxxxxxxxxxxx (10.12.7.2).
Escape character is '^]'.

Here is the link to the discussion,
https://groups.google.com/forum/#!topic/logstash-users/9IPSEtCwrNc
Any input will be greatly appreciated.

-Shriyansh

On Tuesday, July 15, 2014 3:22:43 PM UTC-7, Mark Walkom wrote:

You should probably start this conversation on the logstash list.
However if you are using the http output then you don't need cluster or
node_name. It'd be worth testing that you can telnet from your LS host to
your ES host on port 9200, and also try enabling verbose logging in LS with
-v.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 16 July 2014 07:15, Voc Austin <voc...@gmail.com <javascript:>> wrote:

Hey,

I am using ELK stack for log processing. Logstash 1.4.2 (Single Instance)
and Elasticsearch 1.2.2(Cluster of 2 nodes), and redis as broker between
logstash and elasticsearch.

I am able to parse my logs using Logstash, get the parsed document in
Redis. But not able to get the documents inside Elasticsearch.

The following is my Central logstash server configuration file.

input { redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5 } redis { host => "xx.xx.xx.xx" type =>
"redis-input" data_type => "list" key => "logstash"
threads => 5 }} filter { date { match =>
["timestamp_nsstats",%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[,]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?"]
} } output { stdout { } elasticsearch
{ cluster => "logstash" host =>
"xx.xx.xx.xx" protocol => "http" node_name =>
"Node1" index => "logstash-%{+YYYY.MM.dd}"} }

The following is the health of Elasticsearch Instances.

  1. {

  2. "cluster_name": "logstash",

  3. "status": "green",

  4. "timed_out": false,

  5. "number_of_nodes": 2,

  6. "number_of_data_nodes": 2,

  7. "active_primary_shards": 3,

  8. "active_shards": 6,

  9. "relocating_shards": 0,

  10. "initializing_shards": 0,

  11. "unassigned_shards": 0

  12. }

I am not able to find out the reason, that is causing logstash being not able to reach elasticsearch. Do I need to make any changes in configuration file, which might make logstash aware of the elasticsearch server.

Please let me know what you think, that might be causing the issue. If you need any other information please let me know.

Thank you

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKKFHoG3_rY%3Dtx6x7ibmXVRk7btXw_Xaxv_S-FEMHX0b_g8-oA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKKFHoG3_rY%3Dtx6x7ibmXVRk7btXw_Xaxv_S-FEMHX0b_g8-oA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e6605cd3-8571-47ae-b4fc-f3bce5af7fa9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4