Issues Deploying to Beanstalk

Elastic Stack Elasticsearch
1

I have a web application with an embedded Elasticsearch (0.15.0) server. I
create a war file and successfully deploy the application to both Tomcat
6.0.29 and Jetty 7 locally. I can also deploy the same web application to
Amazon EC2 using AMI's from Scalr.net and Tomcat.

However, when I deploy the same WAR file to Amazon's Beanstalk (using
Tomcat), the app freezes as ES is starting up.

Here is a gist https://gist.github.com/864207 showing a thread dump which
is stalled.

And here is the server information which might be pertinent:

uname -a && cat /etc/*release
Linux ip-10-124-245-247 2.6.34.7-56.40.amzn1.x86_64 #1 SMP Fri Oct 22
18:48:49 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
Amazon Linux AMI release 2010.11.1 (beta)

cat /proc/sys/fs/file-max
764412

Any thoughts?
-- jim

2

Its stuck in the code segment in Node startup where it tries to obtain a native file system lock for its own data directory location (you can start more than one node on a machine, so they need their own data workspace).

Have you configured the data location for this instance? Maybe it tries to write it to the default one, which if embedded in tomcat will probably be within tomcat somewhere, and maybe its not allowed? Same applies to work dir by the day.
On Thursday, March 10, 2011 at 6:44 PM, James Cook wrote:

I have a web application with an embedded Elasticsearch (0.15.0) server. I create a war file and successfully deploy the application to both Tomcat 6.0.29 and Jetty 7 locally. I can also deploy the same web application to Amazon EC2 using AMI's from Scalr.net and Tomcat.

However, when I deploy the same WAR file to Amazon's Beanstalk (using Tomcat), the app freezes as ES is starting up.

Here is a gist showing a thread dump which is stalled.

And here is the server information which might be pertinent:

uname -a && cat /etc/*release
Linux ip-10-124-245-247 2.6.34.7-56.40.amzn1.x86_64 #1 SMP Fri Oct 22 18:48:49 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
Amazon Linux AMI release 2010.11.1 (beta)

cat /proc/sys/fs/file-max
764412

Any thoughts?
-- jim

3

Thanks Shay. That got me a bit farther.

I have also discovered that the Amazon AMI for beanstalk uses the following
JDK:

java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.1)
(amazon-44.1.9.1.18.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b06, mixed mode)

Are there any known issues with OpenJDK?

-- jim

On Fri, Mar 11, 2011 at 8:07 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Its stuck in the code segment in Node startup where it tries to obtain a
native file system lock for its own data directory location (you can start
more than one node on a machine, so they need their own data workspace).

Have you configured the data location for this instance? Maybe it tries to
write it to the default one, which if embedded in tomcat will probably be
within tomcat somewhere, and maybe its not allowed? Same applies to work
dir by the day.

On Thursday, March 10, 2011 at 6:44 PM, James Cook wrote:

I have a web application with an embedded Elasticsearch (0.15.0) server. I
create a war file and successfully deploy the application to both Tomcat
6.0.29 and Jetty 7 locally. I can also deploy the same web application to
Amazon EC2 using AMI's from Scalr.net and Tomcat.

However, when I deploy the same WAR file to Amazon's Beanstalk (using
Tomcat), the app freezes as ES is starting up.

Here is a gist https://gist.github.com/864207 showing a thread dump
which is stalled.

And here is the server information which might be pertinent:

uname -a && cat /etc/*release
Linux ip-10-124-245-247 2.6.34.7-56.40.amzn1.x86_64 #1 SMP Fri Oct 22
18:48:49 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
Amazon Linux AMI release 2010.11.1 (beta)

cat /proc/sys/fs/file-max
764412

Any thoughts?
-- jim

4

Not that I am aware of. Where is it stuck now?
On Friday, March 11, 2011 at 11:16 PM, James Cook wrote:

Thanks Shay. That got me a bit farther.

I have also discovered that the Amazon AMI for beanstalk uses the following JDK:

java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.1) (amazon-44.1.9.1.18.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b06, mixed mode)

Are there any known issues with OpenJDK?

-- jim
On Fri, Mar 11, 2011 at 8:07 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Its stuck in the code segment in Node startup where it tries to obtain a native file system lock for its own data directory location (you can start more than one node on a machine, so they need their own data workspace).

Have you configured the data location for this instance? Maybe it tries to write it to the default one, which if embedded in tomcat will probably be within tomcat somewhere, and maybe its not allowed? Same applies to work dir by the day.
On Thursday, March 10, 2011 at 6:44 PM, James Cook wrote:

I have a web application with an embedded Elasticsearch (0.15.0) server. I create a war file and successfully deploy the application to both Tomcat 6.0.29 and Jetty 7 locally. I can also deploy the same web application to Amazon EC2 using AMI's from Scalr.net and Tomcat.

However, when I deploy the same WAR file to Amazon's Beanstalk (using Tomcat), the app freezes as ES is starting up.

Here is a gist showing a thread dump which is stalled.

And here is the server information which might be pertinent:

uname -a && cat /etc/*release
Linux ip-10-124-245-247 2.6.34.7-56.40.amzn1.x86_64 #1 SMP Fri Oct 22 18:48:49 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
Amazon Linux AMI release 2010.11.1 (beta)

cat /proc/sys/fs/file-max
764412

Any thoughts?
-- jim

5

Sorry for the late reply. We have it deployed successfully now, but we
haven't tried to use the EC2 discovery mechanism yet. Just need a single
server at the moment.

-- jim

On Sat, Mar 12, 2011 at 2:06 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Not that I am aware of. Where is it stuck now?

On Friday, March 11, 2011 at 11:16 PM, James Cook wrote:

Thanks Shay. That got me a bit farther.

I have also discovered that the Amazon AMI for beanstalk uses the following
JDK:

java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.1)
(amazon-44.1.9.1.18.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b06, mixed mode)

Are there any known issues with OpenJDK?

-- jim

On Fri, Mar 11, 2011 at 8:07 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Its stuck in the code segment in Node startup where it tries to obtain a
native file system lock for its own data directory location (you can start
more than one node on a machine, so they need their own data workspace).

Have you configured the data location for this instance? Maybe it tries to
write it to the default one, which if embedded in tomcat will probably be
within tomcat somewhere, and maybe its not allowed? Same applies to work
dir by the day.

On Thursday, March 10, 2011 at 6:44 PM, James Cook wrote:

I have a web application with an embedded Elasticsearch (0.15.0) server. I
create a war file and successfully deploy the application to both Tomcat
6.0.29 and Jetty 7 locally. I can also deploy the same web application to
Amazon EC2 using AMI's from Scalr.net and Tomcat.

However, when I deploy the same WAR file to Amazon's Beanstalk (using
Tomcat), the app freezes as ES is starting up.

Here is a gist https://gist.github.com/864207 showing a thread dump
which is stalled.

And here is the server information which might be pertinent:

uname -a && cat /etc/*release
Linux ip-10-124-245-247 2.6.34.7-56.40.amzn1.x86_64 #1 SMP Fri Oct 22
18:48:49 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
Amazon Linux AMI release 2010.11.1 (beta)

cat /proc/sys/fs/file-max
764412

Any thoughts?
-- jim