Help with Cluster

Eric_Luellen · December 17, 2013, 3:32pm

I am working on building out a small POC for Logstash and Elasticsearch. To
start, I have a 2 server setup.

Server 1 - logstash1 - running "java -jar logstash-1.2.2-flatjar.jar
agent -f indexer.conf"
This server is tailing logs from a syslog config file and then sending
them to an Elasticsearch server.
Server 2 - elasticsearch1 - running elasticsearch as a daemon (CentOS
box that i used an rpm instal - version - 0.90.3.)
- This server is also running Kibana."java -jar
  /etc/logstash/logstash-1.2.2-flatjar.jar web"

Overall things seem to be working pretty well. I started to do some general
diagnostics on the elasticsearch server to see how the cluster was doing,
and I saw that it was red.

[root@elasticsearch1 elasticsearch]# curl -XGET

'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "my-cluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 35,
"active_shards" : 35,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 55

When I saw that it was red and that there were 2 nodes, I was confused as
there should only be 1 elasticsearch node. Upon digging further, I see this:

[root@elasticsearch1 elasticsearch]# curl

localhost:9200/_nodes/process?pretty
{
"ok" : true,
"cluster_name" : "my-cluster",
"nodes" : {
"ab8COl6pTj-kJSzrXZTE2w" : {
"name" : "Lupo",
"transport_address" : "inet[/192.168.0.10:9300]",
"hostname" : "logstash1",
"version" : "0.90.3",
"attributes" : {
"client" : "true",
"data" : "false"
},
"process" : {
"refresh_interval" : 1000,
"id" : 4380,
"max_file_descriptors" : 3200
}
},
"FMgeliZPRdQZwy-IZ9MUIp" : {
"name" : "Elasticsearch Server1",
"transport_address" : "inet[/192.168.0.20:9300]",
"hostname" : "elasticsearch1",
"version" : "0.90.3",
"http_address" : "inet[/192.168.0.20:9200]",
"attributes" : {
"master" : "true"
},
"process" : {
"refresh_interval" : 1000,
"id" : 15653,
"max_file_descriptors" : 65535
}
}
}

I am confused why server1, logstash1, is showing up in the elasticsearch
cluster. I'm only running logstash as an indexer and not the built in
elasticsearch feature. How do I get this server to stop showing up in my
cluster? When I look on the logstash1 server, I don't see any elasticsearch
data or indexes like I do on my elasticsearch1 servers. So I don't think
data is truly going to it, but I don't want it to show up.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79821bd7-3679-4fb9-b78f-8c4b292357c7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · December 17, 2013, 3:46pm

I'd not worry of the non data node.
It's only a node which connect to the cluster to give a client to logstash.

If you really don't want it, then you can use http://logstash.net/docs/1.3.1/outputs/elasticsearch_http

HTH

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 décembre 2013 at 16:32:33, Eric Luellen (eric.luellen@gmail.com) a écrit:

I am working on building out a small POC for Logstash and Elasticsearch. To start, I have a 2 server setup.

Server 1 - logstash1 - running "java -jar logstash-1.2.2-flatjar.jar agent -f indexer.conf"
This server is tailing logs from a syslog config file and then sending them to an ElasticSearch server.
Server 2 - elasticsearch1 - running elasticsearch as a daemon (CentOS box that i used an rpm instal - version - 0.90.3.)
This server is also running Kibana."java -jar /etc/logstash/logstash-1.2.2-flatjar.jar web"
Overall things seem to be working pretty well. I started to do some general diagnostics on the elasticsearch server to see how the cluster was doing, and I saw that it was red.

[root@elasticsearch1 elasticsearch]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "my-cluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 35,
"active_shards" : 35,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 55

When I saw that it was red and that there were 2 nodes, I was confused as there should only be 1 elasticsearch node. Upon digging further, I see this:

[root@elasticsearch1 elasticsearch]# curl localhost:9200/_nodes/process?pretty
{
"ok" : true,
"cluster_name" : "my-cluster",
"nodes" : {
"ab8COl6pTj-kJSzrXZTE2w" : {
"name" : "Lupo",
"transport_address" : "inet[/192.168.0.10:9300]",
"hostname" : "logstash1",
"version" : "0.90.3",
"attributes" : {
"client" : "true",
"data" : "false"
},
"process" : {
"refresh_interval" : 1000,
"id" : 4380,
"max_file_descriptors" : 3200
}
},
"FMgeliZPRdQZwy-IZ9MUIp" : {
"name" : "ElasticSearch Server1",
"transport_address" : "inet[/192.168.0.20:9300]",
"hostname" : "elasticsearch1",
"version" : "0.90.3",
"http_address" : "inet[/192.168.0.20:9200]",
"attributes" : {
"master" : "true"
},
"process" : {
"refresh_interval" : 1000,
"id" : 15653,
"max_file_descriptors" : 65535
}
}
}

I am confused why server1, logstash1, is showing up in the elasticsearch cluster. I'm only running logstash as an indexer and not the built in elasticsearch feature. How do I get this server to stop showing up in my cluster? When I look on the logstash1 server, I don't see any elasticsearch data or indexes like I do on my elasticsearch1 servers. So I don't think data is truly going to it, but I don't want it to show up.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79821bd7-3679-4fb9-b78f-8c4b292357c7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52b071f1.d34b6a8.6956%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.

Eric_Luellen · December 17, 2013, 6:45pm

Thanks for the information. I don't mind it being there, I would just
confused of why it was there. If it stays there, will my cluster status
continue to show red on the health? That was my main concern. Also, if it
stays there, I wish I could rename it from the default Lupo it is to the
name of the server so I can distinguish it better.

On Tuesday, December 17, 2013 10:46:56 AM UTC-5, David Pilato wrote:

I'd not worry of the non data node.
It's only a node which connect to the cluster to give a client to logstash.

If you really don't want it, then you can use
Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

HTH

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 17 décembre 2013 at 16:32:33, Eric Luellen (eric.l...@gmail.com<javascript:>)
a écrit:

I am working on building out a small POC for Logstash and Elasticsearch.
To start, I have a 2 server setup.

Server 1 - logstash1 - running "java -jar logstash-1.2.2-flatjar.jar
agent -f indexer.conf"

This server is tailing logs from a syslog config file and then
sending them to an Elasticsearch server.
- Server 2 - elasticsearch1 - running elasticsearch as a daemon 
(CentOS box that i used an rpm instal - version - 0.90.3.)
  - This server is also running Kibana."java -jar 
  /etc/logstash/logstash-1.2.2-flatjar.jar web" 
Overall things seem to be working pretty well. I started to do some
general diagnostics on the elasticsearch server to see how the cluster was
doing, and I saw that it was red.

[root@elasticsearch1 elasticsearch]# curl -XGET '

http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "my-cluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 35,
"active_shards" : 35,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 55

When I saw that it was red and that there were 2 nodes, I was confused as
there should only be 1 elasticsearch node. Upon digging further, I see this:

[root@elasticsearch1 elasticsearch]# curl

localhost:9200/_nodes/process?pretty
{
"ok" : true,
"cluster_name" : "my-cluster",
"nodes" : {
"ab8COl6pTj-kJSzrXZTE2w" : {
"name" : "Lupo",
"transport_address" : "inet[/192.168.0.10:9300]",
"hostname" : "logstash1",
"version" : "0.90.3",
"attributes" : {
"client" : "true",
"data" : "false"
},
"process" : {
"refresh_interval" : 1000,
"id" : 4380,
"max_file_descriptors" : 3200
}
},
"FMgeliZPRdQZwy-IZ9MUIp" : {
"name" : "Elasticsearch Server1",
"transport_address" : "inet[/192.168.0.20:9300]",
"hostname" : "elasticsearch1",
"version" : "0.90.3",
"http_address" : "inet[/192.168.0.20:9200]",
"attributes" : {
"master" : "true"
},
"process" : {
"refresh_interval" : 1000,
"id" : 15653,
"max_file_descriptors" : 65535
}
}
}

I am confused why server1, logstash1, is showing up in the elasticsearch
cluster. I'm only running logstash as an indexer and not the built in
elasticsearch feature. How do I get this server to stop showing up in my
cluster? When I look on the logstash1 server, I don't see any elasticsearch
data or indexes like I do on my elasticsearch1 servers. So I don't think
data is truly going to it, but I don't want it to show up.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/79821bd7-3679-4fb9-b78f-8c4b292357c7%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0b9275fb-8f59-4b59-b532-a153167e8ed1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · December 17, 2013, 7:07pm

Yes you can rename it using http://logstash.net/docs/1.3.1/outputs/elasticsearch#node_name

You have a real problem here as your cluster should not be red.
But it should not be caused by the logstash node.

Did you set embedded to false (it's default on 1.3.1 but not sure about previous version)?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 décembre 2013 at 19:45:18, Eric Luellen (eric.luellen@gmail.com) a écrit:

Thanks for the information. I don't mind it being there, I would just confused of why it was there. If it stays there, will my cluster status continue to show red on the health? That was my main concern. Also, if it stays there, I wish I could rename it from the default Lupo it is to the name of the server so I can distinguish it better.

On Tuesday, December 17, 2013 10:46:56 AM UTC-5, David Pilato wrote:
I'd not worry of the non data node.
It's only a node which connect to the cluster to give a client to logstash.

If you really don't want it, then you can use http://logstash.net/docs/1.3.1/outputs/elasticsearch_http

HTH

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 décembre 2013 at 16:32:33, Eric Luellen (eric.l...@gmail.com) a écrit:

I am working on building out a small POC for Logstash and Elasticsearch. To start, I have a 2 server setup.

Server 1 - logstash1 - running "java -jar logstash-1.2.2-flatjar.jar agent -f indexer.conf"
This server is tailing logs from a syslog config file and then sending them to an ElasticSearch server.
Server 2 - elasticsearch1 - running elasticsearch as a daemon (CentOS box that i used an rpm instal - version - 0.90.3.)
This server is also running Kibana."java -jar /etc/logstash/logstash-1.2.2-flatjar.jar web"
Overall things seem to be working pretty well. I started to do some general diagnostics on the elasticsearch server to see how the cluster was doing, and I saw that it was red.

[root@elasticsearch1 elasticsearch]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "my-cluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 35,
"active_shards" : 35,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 55

When I saw that it was red and that there were 2 nodes, I was confused as there should only be 1 elasticsearch node. Upon digging further, I see this:

[root@elasticsearch1 elasticsearch]# curl localhost:9200/_nodes/process?pretty
{
"ok" : true,
"cluster_name" : "my-cluster",
"nodes" : {
"ab8COl6pTj-kJSzrXZTE2w" : {
"name" : "Lupo",
"transport_address" : "inet[/192.168.0.10:9300]",
"hostname" : "logstash1",
"version" : "0.90.3",
"attributes" : {
"client" : "true",
"data" : "false"
},
"process" : {
"refresh_interval" : 1000,
"id" : 4380,
"max_file_descriptors" : 3200
}
},
"FMgeliZPRdQZwy-IZ9MUIp" : {
"name" : "ElasticSearch Server1",
"transport_address" : "inet[/192.168.0.20:9300]",
"hostname" : "elasticsearch1",
"version" : "0.90.3",
"http_address" : "inet[/192.168.0.20:9200]",
"attributes" : {
"master" : "true"
},
"process" : {
"refresh_interval" : 1000,
"id" : 15653,
"max_file_descriptors" : 65535
}
}
}

I am confused why server1, logstash1, is showing up in the elasticsearch cluster. I'm only running logstash as an indexer and not the built in elasticsearch feature. How do I get this server to stop showing up in my cluster? When I look on the logstash1 server, I don't see any elasticsearch data or indexes like I do on my elasticsearch1 servers. So I don't think data is truly going to it, but I don't want it to show up.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79821bd7-3679-4fb9-b78f-8c4b292357c7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0b9275fb-8f59-4b59-b532-a153167e8ed1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52b0a0f1.42963e5a.6956%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.

Eric_Luellen · December 17, 2013, 7:34pm

Hmmm. I'm not sure why my status is red then. The only thing I can see from
the cluster-health documentation page is that a specific shard is not
allocated in the cluster. When I look at my cluster health, I do see this:

"unassigned_shards" : 60

Guess I need to figure out why I have so many unassigned shards. I think I
am feeding too much data in elasticsearch at the moment. I've turned on the
logstash server shipping to elasticsearch and I'm still getting logs coming
in and it's been about 10 minutes.

As far as the logstash node goes, I have this config on the elasticsearch
portion.

output {
elasticsearch {
embedded => "false"
host => "192.168.0.20" cluster => "my-cluster"
}
}

So there is no reason it should be there. However, as you said, I'm not
terribly worried about that now, but I am concerned about my red status.

On Tuesday, December 17, 2013 2:07:29 PM UTC-5, David Pilato wrote:

Yes you can rename it using
Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

You have a real problem here as your cluster should not be red.
But it should not be caused by the logstash node.

Did you set embedded to false (it's default on 1.3.1 but not sure about
previous version)?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 17 décembre 2013 at 19:45:18, Eric Luellen (eric.l...@gmail.com<javascript:>)
a écrit:

Thanks for the information. I don't mind it being there, I would just
confused of why it was there. If it stays there, will my cluster status
continue to show red on the health? That was my main concern. Also, if it
stays there, I wish I could rename it from the default Lupo it is to the
name of the server so I can distinguish it better.

On Tuesday, December 17, 2013 10:46:56 AM UTC-5, David Pilato wrote:
I'd not worry of the non data node.
It's only a node which connect to the cluster to give a client to
logstash.

If you really don't want it, then you can use
Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

HTH
 -- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 17 décembre 2013 at 16:32:33, Eric Luellen (eric.l...@gmail.com) a
écrit:

I am working on building out a small POC for Logstash and
Elasticsearch. To start, I have a 2 server setup.

Server 1 - logstash1 - running "java -jar
logstash-1.2.2-flatjar.jar agent -f indexer.conf"

This server is tailing logs from a syslog config file and then
sending them to an Elasticsearch server.
- Server 2 - elasticsearch1 - running elasticsearch as a daemon 
(CentOS box that i used an rpm instal - version - 0.90.3.)
  - This server is also running Kibana."java -jar 
  /etc/logstash/logstash-1.2.2-flatjar.jar web" 
Overall things seem to be working pretty well. I started to do some
general diagnostics on the elasticsearch server to see how the cluster was
doing, and I saw that it was red.

[root@elasticsearch1 elasticsearch]# curl -XGET '

http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "my-cluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 35,
"active_shards" : 35,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 55

When I saw that it was red and that there were 2 nodes, I was confused as
there should only be 1 elasticsearch node. Upon digging further, I see this:

[root@elasticsearch1 elasticsearch]# curl

localhost:9200/_nodes/process?pretty
{
"ok" : true,
"cluster_name" : "my-cluster",
"nodes" : {
"ab8COl6pTj-kJSzrXZTE2w" : {
"name" : "Lupo",
"transport_address" : "inet[/192.168.0.10:9300]",
"hostname" : "logstash1",
"version" : "0.90.3",
"attributes" : {
"client" : "true",
"data" : "false"
},
"process" : {
"refresh_interval" : 1000,
"id" : 4380,
"max_file_descriptors" : 3200
}
},
"FMgeliZPRdQZwy-IZ9MUIp" : {
"name" : "Elasticsearch Server1",
"transport_address" : "inet[/192.168.0.20:9300]",
"hostname" : "elasticsearch1",
"version" : "0.90.3",
"http_address" : "inet[/192.168.0.20:9200]",
"attributes" : {
"master" : "true"
},
"process" : {
"refresh_interval" : 1000,
"id" : 15653,
"max_file_descriptors" : 65535
}
}
}

I am confused why server1, logstash1, is showing up in the elasticsearch
cluster. I'm only running logstash as an indexer and not the built in
elasticsearch feature. How do I get this server to stop showing up in my
cluster? When I look on the logstash1 server, I don't see any elasticsearch
data or indexes like I do on my elasticsearch1 servers. So I don't think
data is truly going to it, but I don't want it to show up.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/79821bd7-3679-4fb9-b78f-8c4b292357c7%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0b9275fb-8f59-4b59-b532-a153167e8ed1%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a59a850f-43fd-4d0d-97f0-5598b94a5f16%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · December 17, 2013, 8:38pm

What gives the following?

curl -XGET 'http://localhost:9200/_cluster/state?pretty'

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 décembre 2013 at 20:34:43, Eric Luellen (eric.luellen@gmail.com) a écrit:

Hmmm. I'm not sure why my status is red then. The only thing I can see from the cluster-health documentation page is that a specific shard is not allocated in the cluster. When I look at my cluster health, I do see this:

"unassigned_shards" : 60

Guess I need to figure out why I have so many unassigned shards. I think I am feeding too much data in elasticsearch at the moment. I've turned on the logstash server shipping to elasticsearch and I'm still getting logs coming in and it's been about 10 minutes.

As far as the logstash node goes, I have this config on the elasticsearch portion.

output {
elasticsearch {
embedded => "false"
host => "192.168.0.20" cluster => "my-cluster"
}
}

So there is no reason it should be there. However, as you said, I'm not terribly worried about that now, but I am concerned about my red status.

On Tuesday, December 17, 2013 2:07:29 PM UTC-5, David Pilato wrote:
Yes you can rename it using http://logstash.net/docs/1.3.1/outputs/elasticsearch#node_name

You have a real problem here as your cluster should not be red.
But it should not be caused by the logstash node.

Did you set embedded to false (it's default on 1.3.1 but not sure about previous version)?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 décembre 2013 at 19:45:18, Eric Luellen (eric.l...@gmail.com) a écrit:

Thanks for the information. I don't mind it being there, I would just confused of why it was there. If it stays there, will my cluster status continue to show red on the health? That was my main concern. Also, if it stays there, I wish I could rename it from the default Lupo it is to the name of the server so I can distinguish it better.

On Tuesday, December 17, 2013 10:46:56 AM UTC-5, David Pilato wrote:
I'd not worry of the non data node.
It's only a node which connect to the cluster to give a client to logstash.

If you really don't want it, then you can use http://logstash.net/docs/1.3.1/outputs/elasticsearch_http

HTH

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 décembre 2013 at 16:32:33, Eric Luellen (eric.l...@gmail.com) a écrit:

I am working on building out a small POC for Logstash and Elasticsearch. To start, I have a 2 server setup.

Server 1 - logstash1 - running "java -jar logstash-1.2.2-flatjar.jar agent -f indexer.conf"
This server is tailing logs from a syslog config file and then sending them to an ElasticSearch server.
Server 2 - elasticsearch1 - running elasticsearch as a daemon (CentOS box that i used an rpm instal - version - 0.90.3.)
This server is also running Kibana."java -jar /etc/logstash/logstash-1.2.2-flatjar.jar web"
Overall things seem to be working pretty well. I started to do some general diagnostics on the elasticsearch server to see how the cluster was doing, and I saw that it was red.

[root@elasticsearch1 elasticsearch]# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "my-cluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 35,
"active_shards" : 35,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 55

When I saw that it was red and that there were 2 nodes, I was confused as there should only be 1 elasticsearch node. Upon digging further, I see this:

[root@elasticsearch1 elasticsearch]# curl localhost:9200/_nodes/process?pretty
{
"ok" : true,
"cluster_name" : "my-cluster",
"nodes" : {
"ab8COl6pTj-kJSzrXZTE2w" : {
"name" : "Lupo",
"transport_address" : "inet[/192.168.0.10:9300]",
"hostname" : "logstash1",
"version" : "0.90.3",
"attributes" : {
"client" : "true",
"data" : "false"
},
"process" : {
"refresh_interval" : 1000,
"id" : 4380,
"max_file_descriptors" : 3200
}
},
"FMgeliZPRdQZwy-IZ9MUIp" : {
"name" : "ElasticSearch Server1",
"transport_address" : "inet[/192.168.0.20:9300]",
"hostname" : "elasticsearch1",
"version" : "0.90.3",
"http_address" : "inet[/192.168.0.20:9200]",
"attributes" : {
"master" : "true"
},
"process" : {
"refresh_interval" : 1000,
"id" : 15653,
"max_file_descriptors" : 65535
}
}
}

I am confused why server1, logstash1, is showing up in the elasticsearch cluster. I'm only running logstash as an indexer and not the built in elasticsearch feature. How do I get this server to stop showing up in my cluster? When I look on the logstash1 server, I don't see any elasticsearch data or indexes like I do on my elasticsearch1 servers. So I don't think data is truly going to it, but I don't want it to show up.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79821bd7-3679-4fb9-b78f-8c4b292357c7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0b9275fb-8f59-4b59-b532-a153167e8ed1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a59a850f-43fd-4d0d-97f0-5598b94a5f16%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52b0b659.74b0dc51.111%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.

Eric_Luellen · December 17, 2013, 9:11pm

I ran that command and saw some fairly old files that were no longer there
that it was trying to read. I believe Elasticsearch got behind on indexing
the files and they were removed before it could finish. I'm not sure but
that's just a guess. I have removed all of the files and started fresh.
Currently everything is green across the board. I guess my issue now is how
to ensure that doesn't happen again and how to make sure syslog-ng -->
logstash --> elasticsearch doesn't drop any packets or get backed up.
Thanks.

On Tuesday, December 17, 2013 3:38:49 PM UTC-5, David Pilato wrote:

What gives the following?

curl -XGET 'http://localhost:9200/_cluster/state?pretty'

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 17 décembre 2013 at 20:34:43, Eric Luellen (eric.l...@gmail.com<javascript:>)
a écrit:

Hmmm. I'm not sure why my status is red then. The only thing I can see
from the cluster-health documentation page is that a specific shard is not
allocated in the cluster. When I look at my cluster health, I do see this:

"unassigned_shards" : 60

Guess I need to figure out why I have so many unassigned shards. I think I
am feeding too much data in elasticsearch at the moment. I've turned on the
logstash server shipping to elasticsearch and I'm still getting logs coming
in and it's been about 10 minutes.

As far as the logstash node goes, I have this config on the elasticsearch
portion.

output {
elasticsearch {
embedded => "false"
host => "192.168.0.20" cluster => "my-cluster"
}
}

So there is no reason it should be there. However, as you said, I'm not
terribly worried about that now, but I am concerned about my red status.

On Tuesday, December 17, 2013 2:07:29 PM UTC-5, David Pilato wrote:
Yes you can rename it using
Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

You have a real problem here as your cluster should not be red.
But it should not be caused by the logstash node.

Did you set embedded to false (it's default on 1.3.1 but not sure about
previous version)?
 -- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 17 décembre 2013 at 19:45:18, Eric Luellen (eric.l...@gmail.com) a
écrit:

Thanks for the information. I don't mind it being there, I would just
confused of why it was there. If it stays there, will my cluster status
continue to show red on the health? That was my main concern. Also, if it
stays there, I wish I could rename it from the default Lupo it is to the
name of the server so I can distinguish it better.

On Tuesday, December 17, 2013 10:46:56 AM UTC-5, David Pilato wrote:
I'd not worry of the non data node.
It's only a node which connect to the cluster to give a client to
logstash.

If you really don't want it, then you can use
Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

HTH
 -- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 17 décembre 2013 at 16:32:33, Eric Luellen (eric.l...@gmail.com) a
écrit:

I am working on building out a small POC for Logstash and
Elasticsearch. To start, I have a 2 server setup.

Server 1 - logstash1 - running "java -jar
logstash-1.2.2-flatjar.jar agent -f indexer.conf"

This server is tailing logs from a syslog config file and then
sending them to an Elasticsearch server.
- Server 2 - elasticsearch1 - running elasticsearch as a daemon 
(CentOS box that i used an rpm instal - version - 0.90.3.)
  - This server is also running Kibana."java -jar 
  /etc/logstash/logstash-1.2.2-flatjar.jar web" 
Overall things seem to be working pretty well. I started to do some
general diagnostics on the elasticsearch server to see how the cluster was
doing, and I saw that it was red.

[root@elasticsearch1 elasticsearch]# curl -XGET '

http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "my-cluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 35,
"active_shards" : 35,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 55

When I saw that it was red and that there were 2 nodes, I was confused
as there should only be 1 elasticsearch node. Upon digging further, I see
this:

[root@elasticsearch1 elasticsearch]# curl

localhost:9200/_nodes/process?pretty
{
"ok" : true,
"cluster_name" : "my-cluster",
"nodes" : {
"ab8COl6pTj-kJSzrXZTE2w" : {
"name" : "Lupo",
"transport_address" : "inet[/192.168.0.10:9300]",
"hostname" : "logstash1",
"version" : "0.90.3",
"attributes" : {
"client" : "true",
"data" : "false"
},
"process" : {
"refresh_interval" : 1000,
"id" : 4380,
"max_file_descriptors" : 3200
}
},
"FMgeliZPRdQZwy-IZ9MUIp" : {
"name" : "Elasticsearch Server1",
"transport_address" : "inet[/192.168.0.20:9300]",
"hostname" : "elasticsearch1",
"version" : "0.90.3",
"http_address" : "inet[/192.168.0.20:9200]",
"attributes" : {
"master" : "true"
},
"process" : {
"refresh_interval" : 1000,
"id" : 15653,
"max_file_descriptors" : 65535
}
}
}

I am confused why server1, logstash1, is showing up in the elasticsearch
cluster. I'm only running logstash as an indexer and not the built in
elasticsearch feature. How do I get this server to stop showing up in my
cluster? When I look on the logstash1 server, I don't see any elasticsearch
data or indexes like I do on my elasticsearch1 servers. So I don't think
data is truly going to it, but I don't want it to show up.

Thanks,
Eric

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/79821bd7-3679-4fb9-b78f-8c4b292357c7%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0b9275fb-8f59-4b59-b532-a153167e8ed1%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a59a850f-43fd-4d0d-97f0-5598b94a5f16%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04aae983-820a-4fff-b006-ebedf602ffad%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.