ES logs and cluster health

Elastic Stack Elasticsearch
1

We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu
and have it set up as a service. We have defined our location for logs as
a specific directory other than /var/log/elasticsearch which is the
default. When this starts, the process shows the value we set for the log
area but no logs are created. It is not due to disk being full.
-Des.default.path.logs=/data/moloch/logs - shows in the process list

Also have found that the /_plugin/head and the _cluster/health do not
always match up. Have see where the plugin shows green while the cluster
health shows yellow. Is this a bug or by design?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/21337ce8-f922-42ce-b4d6-f09fe0c4266b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

You'd have to ask the author of head why it doesn't align.

Does ES have permissions to write to the new log directory?

On 13 February 2015 at 05:47, caspertz christine.tumbusch@gmail.com wrote:

We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu
and have it set up as a service. We have defined our location for logs as
a specific directory other than /var/log/elasticsearch which is the
default. When this starts, the process shows the value we set for the log
area but no logs are created. It is not due to disk being full.
-Des.default.path.logs=/data/moloch/logs - shows in the process list

Also have found that the /_plugin/head and the _cluster/health do not
always match up. Have see where the plugin shows green while the cluster
health shows yellow. Is this a bug or by design?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/21337ce8-f922-42ce-b4d6-f09fe0c4266b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/21337ce8-f922-42ce-b4d6-f09fe0c4266b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Kpt1ceCrazih%2B7czxTmvEztf7tw0OGZ2D87R7xQR3WQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

the owner of ES is root and yes 755 permissions and owned by root. it isnt
going to some other place - there isnt any files/folders called
elasticsearch but the ones in /etc/default and /etc/init.d.
/data/moloch/logs is set for the LOG_DIR

As for the plugin - I put a post on the ES site under github and was told
to ask my questions here. Is there someplace else I should post the
question about the plugin?
On Thursday, February 12, 2015 at 4:36:11 PM UTC-5, Mark Walkom wrote:

You'd have to ask the author of head why it doesn't align.

Does ES have permissions to write to the new log directory?

On 13 February 2015 at 05:47, caspertz <christine...@gmail.com
<javascript:>> wrote:

We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu
and have it set up as a service. We have defined our location for logs as
a specific directory other than /var/log/elasticsearch which is the
default. When this starts, the process shows the value we set for the log
area but no logs are created. It is not due to disk being full.
-Des.default.path.logs=/data/moloch/logs - shows in the process list

Also have found that the /_plugin/head and the _cluster/health do not
always match up. Have see where the plugin shows green while the cluster
health shows yellow. Is this a bug or by design?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/21337ce8-f922-42ce-b4d6-f09fe0c4266b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/21337ce8-f922-42ce-b4d6-f09fe0c4266b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b50a2b7e-c177-44d1-9992-4383c7f10fdb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

The head plugin is not an official plugin, so we (as an Elasticsearch team)
have no control over how it works. You need to ask the author -

Unless you have changed the defaults file, then ES will run as
elasticsearch, not as root, so can you confirm that?

On 13 February 2015 at 23:14, caspertz christine.tumbusch@gmail.com wrote:

the owner of ES is root and yes 755 permissions and owned by root. it
isnt going to some other place - there isnt any files/folders called
elasticsearch but the ones in /etc/default and /etc/init.d.
/data/moloch/logs is set for the LOG_DIR

As for the plugin - I put a post on the ES site under github and was told
to ask my questions here. Is there someplace else I should post the
question about the plugin?
On Thursday, February 12, 2015 at 4:36:11 PM UTC-5, Mark Walkom wrote:

You'd have to ask the author of head why it doesn't align.

Does ES have permissions to write to the new log directory?

On 13 February 2015 at 05:47, caspertz christine...@gmail.com wrote:

We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu
and have it set up as a service. We have defined our location for logs as
a specific directory other than /var/log/elasticsearch which is the
default. When this starts, the process shows the value we set for the log
area but no logs are created. It is not due to disk being full.
-Des.default.path.logs=/data/moloch/logs - shows in the process list

Also have found that the /_plugin/head and the _cluster/health do not
always match up. Have see where the plugin shows green while the cluster
health shows yellow. Is this a bug or by design?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/21337ce8-f922-42ce-b4d6-f09fe0c4266b%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/21337ce8-f922-42ce-b4d6-f09fe0c4266b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b50a2b7e-c177-44d1-9992-4383c7f10fdb%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b50a2b7e-c177-44d1-9992-4383c7f10fdb%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Ox8_8y4MPr5cwav%2BQr40jwDHTW6MDcEafyBWz_K%2B7cg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

I have changed the defaults file so it is running as root.
ES_USER=root
ES_GROUP=root
and below is the process - so you can see it is running as root and has the
es.default.path.logs set for /data/moloch/logs.

root 25847 1 99 Feb11 ? 9-05:47:38
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms24576M -Xmx24576M -Xss256k
-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Delasticsearch
-Des.pidfile=/var/run/elasticsearch.pid
-Des.path.home=/data/moloch/elasticsearch-1.4.1 -cp
:/data/moloch/elasticsearch-1.4.1/lib/elasticsearch-1.4.1.jar:/data/moloch/elasticsearch-1.4.1/lib/:/data/moloch/elasticsearch-1.4.1/lib/sigar/
-Des.default.config=/data/moloch/etc/elasticsearch.yml
-Des.default.path.home=/data/moloch/elasticsearch-1.4.1
-Des.default.path.logs=/data/moloch/logs
-Des.default.path.data=/data/moloch/data
-Des.default.path.work=/tmp/elasticsearch
-Des.default.path.conf=/data/moloch/etc
org.elasticsearch.bootstrap.Elasticsearch

On Thursday, February 12, 2015 at 1:47:38 PM UTC-5, caspertz wrote:

We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu
and have it set up as a service. We have defined our location for logs as
a specific directory other than /var/log/elasticsearch which is the
default. When this starts, the process shows the value we set for the log
area but no logs are created. It is not due to disk being full.
-Des.default.path.logs=/data/moloch/logs - shows in the process list

Also have found that the /_plugin/head and the _cluster/health do not
always match up. Have see where the plugin shows green while the cluster
health shows yellow. Is this a bug or by design?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e791f24a-8cbc-4a38-bc4b-1a744b48417c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Why? This is a security problem!

Can you provide the output from du -sh and mount?

On 16 February 2015 at 22:50, caspertz christine.tumbusch@gmail.com wrote:

I have changed the defaults file so it is running as root.
ES_USER=root
ES_GROUP=root
and below is the process - so you can see it is running as root and has
the es.default.path.logs set for /data/moloch/logs.

root 25847 1 99 Feb11 ? 9-05:47:38
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms24576M -Xmx24576M -Xss256k
-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Delasticsearch
-Des.pidfile=/var/run/elasticsearch.pid
-Des.path.home=/data/moloch/elasticsearch-1.4.1 -cp
:/data/moloch/elasticsearch-1.4.1/lib/elasticsearch-1.4.1.jar:/data/moloch/elasticsearch-1.4.1/lib/:/data/moloch/elasticsearch-1.4.1/lib/sigar/
-Des.default.config=/data/moloch/etc/elasticsearch.yml
-Des.default.path.home=/data/moloch/elasticsearch-1.4.1
-Des.default.path.logs=/data/moloch/logs
-Des.default.path.data=/data/moloch/data
-Des.default.path.work=/tmp/elasticsearch
-Des.default.path.conf=/data/moloch/etc
org.elasticsearch.bootstrap.Elasticsearch

On Thursday, February 12, 2015 at 1:47:38 PM UTC-5, caspertz wrote:

We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu
and have it set up as a service. We have defined our location for logs as
a specific directory other than /var/log/elasticsearch which is the
default. When this starts, the process shows the value we set for the log
area but no logs are created. It is not due to disk being full.
-Des.default.path.logs=/data/moloch/logs - shows in the process list

Also have found that the /_plugin/head and the _cluster/health do not
always match up. Have see where the plugin shows green while the cluster
health shows yellow. Is this a bug or by design?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e791f24a-8cbc-4a38-bc4b-1a744b48417c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e791f24a-8cbc-4a38-bc4b-1a744b48417c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w%3D-S0_PWTaZ-ufpe0HwJg20Ts00q%3DBmv7m2SiA1nxhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

/dev/sdb1 on /data type ext4 (rw,errors=remount-ro)
2.2T /data
4.0K /data/moloch/logs

/dev/sdb1 2882585900 2322227348 413908584 85% /data

/data is not full

As for why we have been running as root - just how my group set it up years
ago. The ES cluster is doing nothing but ES.

On Tuesday, February 17, 2015 at 1:54:14 AM UTC-5, Mark Walkom wrote:

Why? This is a security problem!

Can you provide the output from du -sh and mount?

On 16 February 2015 at 22:50, caspertz <christine...@gmail.com
<javascript:>> wrote:

I have changed the defaults file so it is running as root.
ES_USER=root
ES_GROUP=root
and below is the process - so you can see it is running as root and has
the es.default.path.logs set for /data/moloch/logs.

root 25847 1 99 Feb11 ? 9-05:47:38
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms24576M -Xmx24576M -Xss256k
-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Delasticsearch
-Des.pidfile=/var/run/elasticsearch.pid
-Des.path.home=/data/moloch/elasticsearch-1.4.1 -cp
:/data/moloch/elasticsearch-1.4.1/lib/elasticsearch-1.4.1.jar:/data/moloch/elasticsearch-1.4.1/lib/:/data/moloch/elasticsearch-1.4.1/lib/sigar/
-Des.default.config=/data/moloch/etc/elasticsearch.yml
-Des.default.path.home=/data/moloch/elasticsearch-1.4.1
-Des.default.path.logs=/data/moloch/logs
-Des.default.path.data=/data/moloch/data
-Des.default.path.work=/tmp/elasticsearch
-Des.default.path.conf=/data/moloch/etc
org.elasticsearch.bootstrap.Elasticsearch

On Thursday, February 12, 2015 at 1:47:38 PM UTC-5, caspertz wrote:

We have a 4 node ES cluster running version 1.4.1. We are using Ubuntu
and have it set up as a service. We have defined our location for logs as
a specific directory other than /var/log/elasticsearch which is the
default. When this starts, the process shows the value we set for the log
area but no logs are created. It is not due to disk being full.
-Des.default.path.logs=/data/moloch/logs - shows in the process list

Also have found that the /_plugin/head and the _cluster/health do not
always match up. Have see where the plugin shows green while the cluster
health shows yellow. Is this a bug or by design?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e791f24a-8cbc-4a38-bc4b-1a744b48417c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e791f24a-8cbc-4a38-bc4b-1a744b48417c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e2fd0579-4d45-4dcc-b150-d40dc18d7a6e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.