-Des.path.conf issue?


(roger-2) #1

Hi together

I'm working on an elasticsearch server stub for python using
a startup via python subprocess.

Everything is working fine except the es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES download/unzip and pointing
to an external config folder which contains some mappings.
The default config folder inside the donwload/unzip is
still there.

I was reading in some mailinglists that there is/was an issue
with append/merge config from different locations, but what
happens to mapping definiteions locate in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(Shay Banon) #2

The -D parameter does not work on windows, only on *nix systems.
On Wednesday, March 23, 2011 at 4:30 AM, Roger wrote:

Hi together

I'm working on an elasticsearch server stub for python using
a startup via python subprocess.

Everything is working fine except the es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES download/unzip and pointing
to an external config folder which contains some mappings.
The default config folder inside the donwload/unzip is
still there.

I was reading in some mailinglists that there is/was an issue
with append/merge config from different locations, but what
happens to mapping definiteions locate in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(Shay Banon) #3

But, you can pass those in ES_JAVA_OPTS env var.
On Wednesday, March 23, 2011 at 11:59 AM, Shay Banon wrote:

The -D parameter does not work on windows, only on *nix systems.
On Wednesday, March 23, 2011 at 4:30 AM, Roger wrote:

Hi together

I'm working on an elasticsearch server stub for python using
a startup via python subprocess.

Everything is working fine except the es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES download/unzip and pointing
to an external config folder which contains some mappings.
The default config folder inside the donwload/unzip is
still there.

I was reading in some mailinglists that there is/was an issue
with append/merge config from different locations, but what
happens to mapping definiteions locate in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(roger-2) #4

Hi Shay

Betreff: Re: -Des.path.conf issue?

The -D parameter does not work on windows, only on *nix systems.

Thanks for the quick response!
But now I'm confused. The bin/elasticsearch.bat script
defines the following start parameter:


set
ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSION@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch -Des-foreground=yes -Des.path.home="%ES_HOME%"

"%JAVA_HOME%\bin\java" %JAVA_OPTS% %ES_JAVA_OPTS% %ES_PARAMS% -cp
"%ES_CLASSPATH%" "org.elasticsearch.bootstrap.ElasticSearch"

This means a -D parameter is used e.g. -Des.path.home="%ES_HOME%"
Is this working or not? Is only the -Des.path.conf not working?

Regards
Roger Ineichen

On Wednesday, March 23, 2011 at 4:30 AM, Roger wrote:

Hi together

I'm working on an elasticsearch server stub for python using
a startup via python subprocess.

Everything is working fine except the es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES download/unzip and pointing
to an external config folder which contains some mappings.
The default config folder inside the donwload/unzip is
still there.

I was reading in some mailinglists that there is/was an issue
with append/merge config from different locations, but what
happens to mapping definiteions locate in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(Shay Banon) #5

Using: elasticsearch.bat -Des.... will not work, since it only get parsed in the elasticsearch.sh file.

For windows, I see that I did not add the option to add ES_JAVA_OPTS. But if you look at the file, you can see that JAVA_OPTS sets some default values, you can add to that some more -D parameters.

I need to improve that batch shell...
On Wednesday, March 23, 2011 at 1:32 PM, Roger wrote:

Hi Shay

Betreff: Re: -Des.path.conf issue?

The -D parameter does not work on windows, only on *nix systems.

Thanks for the quick response!
But now I'm confused. The bin/elasticsearch.bat script
defines the following start parameter:


set
ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSION@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch -Des-foreground=yes -Des.path.home="%ES_HOME%"

"%JAVA_HOME%\bin\java" %JAVA_OPTS% %ES_JAVA_OPTS% %ES_PARAMS% -cp
"%ES_CLASSPATH%" "org.elasticsearch.bootstrap.ElasticSearch"

This means a -D parameter is used e.g. -Des.path.home="%ES_HOME%"
Is this working or not? Is only the -Des.path.conf not working?

Regards
Roger Ineichen

On Wednesday, March 23, 2011 at 4:30 AM, Roger wrote:

Hi together

I'm working on an elasticsearch server stub for python using
a startup via python subprocess.

Everything is working fine except the es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES download/unzip and pointing
to an external config folder which contains some mappings.
The default config folder inside the donwload/unzip is
still there.

I was reading in some mailinglists that there is/was an issue
with append/merge config from different locations, but what
happens to mapping definiteions locate in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(Shay Banon) #6

Sorry, ES_JAVA_OPTS is there in teh script
On Wednesday, March 23, 2011 at 2:41 PM, Shay Banon wrote:

Using: elasticsearch.bat -Des.... will not work, since it only get parsed in the elasticsearch.sh file.

For windows, I see that I did not add the option to add ES_JAVA_OPTS. But if you look at the file, you can see that JAVA_OPTS sets some default values, you can add to that some more -D parameters.

I need to improve that batch shell...
On Wednesday, March 23, 2011 at 1:32 PM, Roger wrote:

Hi Shay

Betreff: Re: -Des.path.conf issue?

The -D parameter does not work on windows, only on *nix systems.

Thanks for the quick response!
But now I'm confused. The bin/elasticsearch.bat script
defines the following start parameter:


set
ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSION@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch -Des-foreground=yes -Des.path.home="%ES_HOME%"

"%JAVA_HOME%\bin\java" %JAVA_OPTS% %ES_JAVA_OPTS% %ES_PARAMS% -cp
"%ES_CLASSPATH%" "org.elasticsearch.bootstrap.ElasticSearch"

This means a -D parameter is used e.g. -Des.path.home="%ES_HOME%"
Is this working or not? Is only the -Des.path.conf not working?

Regards
Roger Ineichen

On Wednesday, March 23, 2011 at 4:30 AM, Roger wrote:

Hi together

I'm working on an elasticsearch server stub for python using
a startup via python subprocess.

Everything is working fine except the es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES download/unzip and pointing
to an external config folder which contains some mappings.
The default config folder inside the donwload/unzip is
still there.

I was reading in some mailinglists that there is/was an issue
with append/merge config from different locations, but what
happens to mapping definiteions locate in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(Administrator-2) #7

I'm confused in the same way, I've been using this batch script to run on
Windows, and it works just fine when I use the -Des.* parameters on the
command line.

I've also been able to get it to work on the command-line in Azure, when
dynamically mounting drives for data and logs.

This most definitely does work on Windows machines.

  • Nick

-----Original Message-----
From: Roger [mailto:dev@projekt01.ch]
Sent: Wednesday, March 23, 2011 7:32 AM
To: users@elasticsearch.com
Subject: AW: -Des.path.conf issue?

Hi Shay

Betreff: Re: -Des.path.conf issue?

The -D parameter does not work on windows, only on *nix systems.

Thanks for the quick response!
But now I'm confused. The bin/elasticsearch.bat script
defines the following start parameter:


set
ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSION@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch -Des-foreground=yes -Des.path.home="%ES_HOME%"

"%JAVA_HOME%\bin\java" %JAVA_OPTS% %ES_JAVA_OPTS% %ES_PARAMS% -cp
"%ES_CLASSPATH%" "org.elasticsearch.bootstrap.ElasticSearch"

This means a -D parameter is used e.g. -Des.path.home="%ES_HOME%"
Is this working or not? Is only the -Des.path.conf not working?

Regards
Roger Ineichen

On Wednesday, March 23, 2011 at 4:30 AM, Roger wrote:

Hi together

I'm working on an elasticsearch server stub for python using
a startup via python subprocess.

Everything is working fine except the es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES download/unzip and pointing
to an external config folder which contains some mappings.
The default config folder inside the donwload/unzip is
still there.

I was reading in some mailinglists that there is/was an issue
with append/merge config from different locations, but what
happens to mapping definiteions locate in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(Shay Banon) #8

I am talking about executing the shell in this manner:

elasticsearch.bat -Des.path.conf=path/to/conf

This does not work.

You can set the -D parameters in the ES_JAVA_OPS env var.
On Wednesday, March 23, 2011 at 2:57 PM, Administrator wrote:

I'm confused in the same way, I've been using this batch script to run on
Windows, and it works just fine when I use the -Des.* parameters on the
command line.

I've also been able to get it to work on the command-line in Azure, when
dynamically mounting drives for data and logs.

This most definitely does work on Windows machines.

  • Nick

-----Original Message-----
From: Roger [mailto:dev@projekt01.ch]
Sent: Wednesday, March 23, 2011 7:32 AM
To: users@elasticsearch.com
Subject: AW: -Des.path.conf issue?

Hi Shay

Betreff: Re: -Des.path.conf issue?

The -D parameter does not work on windows, only on *nix systems.

Thanks for the quick response!
But now I'm confused. The bin/elasticsearch.bat script
defines the following start parameter:


set
ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSION@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch -Des-foreground=yes -Des.path.home="%ES_HOME%"

"%JAVA_HOME%\bin\java" %JAVA_OPTS% %ES_JAVA_OPTS% %ES_PARAMS% -cp
"%ES_CLASSPATH%" "org.elasticsearch.bootstrap.ElasticSearch"

This means a -D parameter is used e.g. -Des.path.home="%ES_HOME%"
Is this working or not? Is only the -Des.path.conf not working?

Regards
Roger Ineichen

On Wednesday, March 23, 2011 at 4:30 AM, Roger wrote:

Hi together

I'm working on an elasticsearch server stub for python using
a startup via python subprocess.

Everything is working fine except the es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES download/unzip and pointing
to an external config folder which contains some mappings.
The default config folder inside the donwload/unzip is
still there.

I was reading in some mailinglists that there is/was an issue
with append/merge config from different locations, but what
happens to mapping definiteions locate in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(roger-2) #9

Hi Shay, Nick

Sorry I was a littlebit unclear in my initial mail.
I'm not using the *.bat script. I'm developing a
python doctest stub server lib which starts the
ES server from a python subproces. This lib allows
to configure the ES server.

I'm not able to set the path.conf in any way.
My code looks something like right now:


# setup java options and friends
libPath = os.path.join(server, 'lib')
sigarPath = os.path.join(libPath, 'sigar')
jarPath = os.path.join(libPath, 'elasticsearch-0.15.2.jar')
JAVA_HOME = os.environ['JAVA_HOME']
JAVA_BIN = os.path.join(JAVA_HOME, 'bin', 'java')
ES_JAVA_OPTS = '-Delasticsearch'
ES_JAVA_OPTS += ' -Des-foreground=yes'
ES_JAVA_OPTS += ' -Des.path.home="%s"' % server
ES_JAVA_OPTS += ' -Des.path.conf="%s"' % config
ES_JAVA_OPTS += ' -Des.path.logs="%s"' % logs

CLASSPATH = os.environ['CLASSPATH']
ES_CLASSPATH = "%s;%s/*;%s/*;%s" % (jarPath, libPath, sigarPath,
    CLASSPATH)

env = dict(os.environ)
env['ES_JAVA_OPTS'] = ES_JAVA_OPTS
env['ES_CLASSPATH'] = ES_CLASSPATH

cmd = [JAVA_BIN,
       "-Xms128m",
       "-Xmx128m",
       "-Djline.enabled=false",
       "-XX:+AggressiveOpts",
       "-XX:+UseParNewGC",
       "-XX:+UseConcMarkSweepGC",
       "-XX:+CMSParallelRemarkEnabled",
       "-XX:+HeapDumpOnOutOfMemoryError",
       ES_JAVA_OPTS,
       "-cp", ES_CLASSPATH, "org.elasticsearch.bootstrap.Bootstrap"]

# start the elasticsearch stub server
try:
    p = subprocess.Popen(cmd, shell=False, env=env)
except Exception, e:
    raise Exception("Subprocess error: %s" % e)

Regards
Roger Ineichen

Betreff: Re: AW: -Des.path.conf issue?

Sorry, ES_JAVA_OPTS is there in teh script

On Wednesday, March 23, 2011 at 2:41 PM, Shay Banon wrote:

Using: elasticsearch.bat -Des.... will not work, since
it only get parsed in the elasticsearch.sh file.

For windows, I see that I did not add the option to add
ES_JAVA_OPTS. But if you look at the file, you can see that
JAVA_OPTS sets some default values, you can add to that some
more -D parameters.

I need to improve that batch shell...

On Wednesday, March 23, 2011 at 1:32 PM, Roger wrote:

  Hi Shay 
  
  

  	Betreff: Re: -Des.path.conf issue?
  	
  	The -D parameter does not work on 

windows, only on *nix systems.

  Thanks for the quick response!
  But now I'm confused. The bin/elasticsearch.bat script 
  defines the following start parameter:
  
  ---
  set

ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSI
ON@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch
-Des-foreground=yes -Des.path.home="%ES_HOME%"

  "%JAVA_HOME%\bin\java" %JAVA_OPTS% 

%ES_JAVA_OPTS% %ES_PARAMS% -cp
"%ES_CLASSPATH%"
"org.elasticsearch.bootstrap.ElasticSearch"
---

  This means a -D parameter is used e.g. 

-Des.path.home="%ES_HOME%"
Is this working or not? Is only the
-Des.path.conf not working?

  Regards
  Roger Ineichen
  
  

  	On Wednesday, March 23, 2011 at 4:30 

AM, Roger wrote:

  	Hi together
  	
  	I'm working on an elasticsearch server 

stub for python using
a startup via python subprocess.

  	Everything is working fine except the 

es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was
testing with windows and
nix config path e.g. \ and /.

  	I'm trying to use a plain ES 

download/unzip and pointing
to an external config folder which
contains some mappings.
The default config folder inside the
donwload/unzip is
still there.

  	I was reading in some mailinglists that 

there is/was an issue
with append/merge config from different
locations, but what
happens to mapping definiteions locate
in config/mappings in
such a configuration?

  	Is this an issue?
  	
  	Regards
  	Roger Ineichen
  	_____________________________
  	END OF MESSAGE

(Shay Banon) #10

If you fork the Java process, then you need to pass each -D option by itself, the same way you pass -Xmx, -Xms, and so on (your cmd var). Also, remove the AgressiveOpts from there, I am updating the batch script to not include it as well.
On Wednesday, March 23, 2011 at 3:18 PM, Roger wrote:

Hi Shay, Nick

Sorry I was a littlebit unclear in my initial mail.
I'm not using the *.bat script. I'm developing a
python doctest stub server lib which starts the
ES server from a python subproces. This lib allows
to configure the ES server.

I'm not able to set the path.conf in any way.
My code looks something like right now:


setup java options and friends

libPath = os.path.join(server, 'lib')
sigarPath = os.path.join(libPath, 'sigar')
jarPath = os.path.join(libPath, 'elasticsearch-0.15.2.jar')
JAVA_HOME = os.environ['JAVA_HOME']
JAVA_BIN = os.path.join(JAVA_HOME, 'bin', 'java')
ES_JAVA_OPTS = '-Delasticsearch'
ES_JAVA_OPTS += ' -Des-foreground=yes'
ES_JAVA_OPTS += ' -Des.path.home="%s"' % server
ES_JAVA_OPTS += ' -Des.path.conf="%s"' % config
ES_JAVA_OPTS += ' -Des.path.logs="%s"' % logs

CLASSPATH = os.environ['CLASSPATH']
ES_CLASSPATH = "%s;%s/;%s/;%s" % (jarPath, libPath, sigarPath,
CLASSPATH)

env = dict(os.environ)
env['ES_JAVA_OPTS'] = ES_JAVA_OPTS
env['ES_CLASSPATH'] = ES_CLASSPATH

cmd = [JAVA_BIN,
"-Xms128m",
"-Xmx128m",
"-Djline.enabled=false",
"-XX:+AggressiveOpts",
"-XX:+UseParNewGC",
"-XX:+UseConcMarkSweepGC",
"-XX:+CMSParallelRemarkEnabled",
"-XX:+HeapDumpOnOutOfMemoryError",
ES_JAVA_OPTS,
"-cp", ES_CLASSPATH, "org.elasticsearch.bootstrap.Bootstrap"]

start the elasticsearch stub server

try:
p = subprocess.Popen(cmd, shell=False, env=env)
except Exception, e:
raise Exception("Subprocess error: %s" % e)


Regards
Roger Ineichen

Betreff: Re: AW: -Des.path.conf issue?

Sorry, ES_JAVA_OPTS is there in teh script

On Wednesday, March 23, 2011 at 2:41 PM, Shay Banon wrote:

Using: elasticsearch.bat -Des.... will not work, since
it only get parsed in the elasticsearch.sh file.

For windows, I see that I did not add the option to add
ES_JAVA_OPTS. But if you look at the file, you can see that
JAVA_OPTS sets some default values, you can add to that some
more -D parameters.

I need to improve that batch shell...

On Wednesday, March 23, 2011 at 1:32 PM, Roger wrote:

Hi Shay

Betreff: Re: -Des.path.conf issue?

The -D parameter does not work on
windows, only on *nix systems.

Thanks for the quick response!
But now I'm confused. The bin/elasticsearch.bat script
defines the following start parameter:


set

ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSI
ON@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch
-Des-foreground=yes -Des.path.home="%ES_HOME%"

"%JAVA_HOME%\bin\java" %JAVA_OPTS%
%ES_JAVA_OPTS% %ES_PARAMS% -cp
"%ES_CLASSPATH%"
"org.elasticsearch.bootstrap.ElasticSearch"

This means a -D parameter is used e.g.
-Des.path.home="%ES_HOME%"
Is this working or not? Is only the
-Des.path.conf not working?

Regards
Roger Ineichen

On Wednesday, March 23, 2011 at 4:30
AM, Roger wrote:

Hi together

I'm working on an elasticsearch server
stub for python using
a startup via python subprocess.

Everything is working fine except the
es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was
testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES
download/unzip and pointing
to an external config folder which
contains some mappings.
The default config folder inside the
donwload/unzip is
still there.

I was reading in some mailinglists that
there is/was an issue
with append/merge config from different
locations, but what
happens to mapping definiteions locate
in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(Administrator-2) #11

Oh, yeah, that definitely does not work.

However, it’s easy enough to patch the batch file to take in the first parameter and then attach that to the end of the parameter list that is used to run ES.

  • Nick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: Wednesday, March 23, 2011 9:03 AM
To: users@elasticsearch.com
Subject: Re: RE: -Des.path.conf issue?

I am talking about executing the shell in this manner:

elasticsearch.bat -Des.path.conf=path/to/conf

This does not work.

You can set the -D parameters in the ES_JAVA_OPS env var.

On Wednesday, March 23, 2011 at 2:57 PM, Administrator wrote:

I'm confused in the same way, I've been using this batch script to run on
Windows, and it works just fine when I use the -Des.* parameters on the
command line.

I've also been able to get it to work on the command-line in Azure, when
dynamically mounting drives for data and logs.

This most definitely does work on Windows machines.

  • Nick

-----Original Message-----
From: Roger [mailto:dev@projekt01.ch]
Sent: Wednesday, March 23, 2011 7:32 AM
To: users@elasticsearch.com
Subject: AW: -Des.path.conf issue?

Hi Shay

Betreff: Re: -Des.path.conf issue?

The -D parameter does not work on windows, only on *nix systems.

Thanks for the quick response!
But now I'm confused. The bin/elasticsearch.bat script
defines the following start parameter:


set
ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSION@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch -Des-foreground=yes -Des.path.home="%ES_HOME%"

"%JAVA_HOME%\bin\java" %JAVA_OPTS% %ES_JAVA_OPTS% %ES_PARAMS% -cp
"%ES_CLASSPATH%" "org.elasticsearch.bootstrap.ElasticSearch"

This means a -D parameter is used e.g. -Des.path.home="%ES_HOME%"
Is this working or not? Is only the -Des.path.conf not working?

Regards
Roger Ineichen

On Wednesday, March 23, 2011 at 4:30 AM, Roger wrote:

Hi together

I'm working on an elasticsearch server stub for python using
a startup via python subprocess.

Everything is working fine except the es.path.conf property.
It seems that the -Des.path.conf= is not
working on a windows system. I was testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES download/unzip and pointing
to an external config folder which contains some mappings.
The default config folder inside the donwload/unzip is
still there.

I was reading in some mailinglists that there is/was an issue
with append/merge config from different locations, but what
happens to mapping definiteions locate in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(roger-2) #12

hi Shay, Nick

You are right I need a list of -D options in the cmd.
But my real problem was the following:

' -Des.path.conf="%s"' % config

This ends in a path like:

"C:\foobar"/elasticsearch.yml

This doesn't work because of the "" in the path string, eeeek

Thanks a lot for your help.

btw,
I will soon release a python package which
offers a stub server setup which can be used in
python doc tests. This will (for the first time)
download an elasticserver release and start the server
for testing against with something like:


def doctestSetUp(test):
# use default elasticsearch stub with our server dir
here = os.path.dirname(file)
serverDir = os.path.join(here, 'server')
confDir = os.path.join(here, 'config')
es01.stub.testing.startElasticSearchServer(serverDir=serverDir,
confDir=confDir)

def doctestTearDown(test):
es01.stub.testing.stopElasticSearchServer()

def test_suite():
return unittest.TestSuite((
doctest.DocFileSuite('README.txt',
setUp=doctestSetUp,
tearDown=doctestTearDown,
optionflags=doctest.NORMALIZE_WHITESPACE|doctest.ELLIPSIS,
encoding='utf-8'),
))

if name == 'main':
unittest.main(defaultTest='test_suite')

Regards
Roger Ineichen

-----Ursprüngliche Nachricht-----
Von: Shay Banon [mailto:shay.banon@elasticsearch.com]
Gesendet: Mittwoch, 23. März 2011 14:22
An: users@elasticsearch.com
Betreff: Re: -Des.path.conf issue?

If you fork the Java process, then you need to pass each -D
option by itself, the same way you pass -Xmx, -Xms, and so on
(your cmd var). Also, remove the AgressiveOpts from there, I
am updating the batch script to not include it as well.

On Wednesday, March 23, 2011 at 3:18 PM, Roger wrote:

Hi Shay, Nick

Sorry I was a littlebit unclear in my initial mail.
I'm not using the *.bat script. I'm developing a
python doctest stub server lib which starts the
ES server from a python subproces. This lib allows
to configure the ES server.

I'm not able to set the path.conf in any way.
My code looks something like right now:


setup java options and friends

libPath = os.path.join(server, 'lib')
sigarPath = os.path.join(libPath, 'sigar')
jarPath = os.path.join(libPath, 'elasticsearch-0.15.2.jar')
JAVA_HOME = os.environ['JAVA_HOME']
JAVA_BIN = os.path.join(JAVA_HOME, 'bin', 'java')
ES_JAVA_OPTS = '-Delasticsearch'
ES_JAVA_OPTS += ' -Des-foreground=yes'
ES_JAVA_OPTS += ' -Des.path.home="%s"' % server
ES_JAVA_OPTS += ' -Des.path.conf="%s"' % config
ES_JAVA_OPTS += ' -Des.path.logs="%s"' % logs

CLASSPATH = os.environ['CLASSPATH']
ES_CLASSPATH = "%s;%s/;%s/;%s" % (jarPath, libPath, sigarPath,
CLASSPATH)

env =D dict(os.environ)
env['ES_JAVA_OPTS'] = ES_JAVA_OPTS
env['ES_CLASSPATH'] = ES_CLASSPATH

cmd = [JAVA_BIN,
"-Xms128m",
"-Xmx128m",
"-Djline.enabled=false",
"-XX:+AggressiveOpts",
"-XX:+UseParNewGC",
"-XX:+UseConcMarkSweepGC",
"-XX:+CMSParallelRemarkEnabled",
"-XX:+HeapDumpOnOutOfMemoryError",
ES_JAVA_OPTS,
"-cp", ES_CLASSPATH, "org.elasticsearch.bootstrap.Bootstrap"]

start the elasticsearch stub server

try:
p = subprocess.Popen(cmd, shell=False, env=env)
except Exception, e:
raise Exception("Subprocess error: %s" % e)


Regards
Roger Ineichen

  Betreff: Re: AW: -Des.path.conf issue?
  
  Sorry, ES_JAVA_OPTS is there in teh script
  
  
  On Wednesday, March 23, 2011 at 2:41 PM, Shay 

Banon wrote:

  Using: elasticsearch.bat -Des.... will not work, since 
  it only get parsed in the elasticsearch.sh file.
  
  
  For windows, I see that I did not add the option to add 
  ES_JAVA_OPTS. But if you look at the file, you 

can see that
JAVA_OPTS sets some default values, you can add
to that some
more -D parameters.

  I need to improve that batch shell...
  
  
  On Wednesday, March 23, 2011 at 1:32 PM, Roger wrote:
  
  
  Hi Shay 
  
  
  
  Betreff: Re: -Des.path.conf issue?
  
  The -D parameter does not work on 
  windows, only on *nix systems.
  
  
  
  Thanks for the quick response!
  But now I'm confused. The bin/elasticsearch.bat script 
  defines the following start parameter:
  
  ---
  set

ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSI
ON@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch
-Des-foreground=yes -Des.path.home="%ES_HOME%"

  "%JAVA_HOME%\bin\java" %JAVA_OPTS% 
  %ES_JAVA_OPTS% %ES_PARAMS% -cp
  "%ES_CLASSPATH%" 
  "org.elasticsearch.bootstrap.ElasticSearch"
  ---
  
  This means a -D parameter is used e.g. 
  -Des.path.home="%ES_HOME%"
  Is this working or not? Is only the 
  -Des.path.conf not working?
  
  Regards
  Roger Ineichen
  
  
  
  On Wednesday, March 23, 2011 at 4:30 
  AM, Roger wrote:
  
  
  Hi together
  
  I'm working on an elasticsearch server 
  stub for python using
  a startup via python subprocess.
  
  Everything is working fine except the 
  es.path.conf property. It seems that the 

-Des.path.conf= is not
working on a windows system. I was
testing with windows and
nix config path e.g. \ and /.

  I'm trying to use a plain ES 
  download/unzip and pointing
  to an external config folder which 
  contains some mappings.
  The default config folder inside the 
  donwload/unzip is
  still there.
  
  I was reading in some mailinglists that 
  there is/was an issue
  with append/merge config from different 
  locations, but what 
  happens to mapping definiteions locate 
  in config/mappings in
  such a configuration?
  
  Is this an issue?
  
  Regards
  Roger Ineichen
  ______________________________
  END OF MESSAGE

(Shay Banon) #13

Thats great!, does it make sense to integrate it with pyes maybe? Something like a test module in pyes (I think there is one now in ElasticSearch.pm).
On Wednesday, March 23, 2011 at 4:39 PM, Roger wrote:

hi Shay, Nick

You are right I need a list of -D options in the cmd.
But my real problem was the following:

' -Des.path.conf="%s"' % config

This ends in a path like:

"C:\foobar"/elasticsearch.yml

This doesn't work because of the "" in the path string, eeeek

Thanks a lot for your help.

btw,
I will soon release a python package which
offers a stub server setup which can be used in
python doc tests. This will (for the first time)
download an elasticserver release and start the server
for testing against with something like:


def doctestSetUp(test):

use default elasticsearch stub with our server dir

here = os.path.dirname(file)
serverDir = os.path.join(here, 'server')
confDir = os.path.join(here, 'config')
es01.stub.testing.startElasticSearchServer(serverDir=serverDir,
confDir=confDir)

def doctestTearDown(test):
es01.stub.testing.stopElasticSearchServer()

def test_suite():
return unittest.TestSuite((
doctest.DocFileSuite('README.txt',
setUp=doctestSetUp,
tearDown=doctestTearDown,
optionflags=doctest.NORMALIZE_WHITESPACE|doctest.ELLIPSIS,
encoding='utf-8'),
))

if name == 'main':
unittest.main(defaultTest='test_suite')

Regards
Roger Ineichen

-----Ursprüngliche Nachricht-----
Von: Shay Banon [mailto:shay.banon@elasticsearch.com]
Gesendet: Mittwoch, 23. März 2011 14:22
An: users@elasticsearch.com
Betreff: Re: -Des.path.conf issue?

If you fork the Java process, then you need to pass each -D
option by itself, the same way you pass -Xmx, -Xms, and so on
(your cmd var). Also, remove the AgressiveOpts from there, I
am updating the batch script to not include it as well.

On Wednesday, March 23, 2011 at 3:18 PM, Roger wrote:

Hi Shay, Nick

Sorry I was a littlebit unclear in my initial mail.
I'm not using the *.bat script. I'm developing a
python doctest stub server lib which starts the
ES server from a python subproces. This lib allows
to configure the ES server.

I'm not able to set the path.conf in any way.
My code looks something like right now:


setup java options and friends

libPath = os.path.join(server, 'lib')
sigarPath = os.path.join(libPath, 'sigar')
jarPath = os.path.join(libPath, 'elasticsearch-0.15.2.jar')
JAVA_HOME = os.environ['JAVA_HOME']
JAVA_BIN = os.path.join(JAVA_HOME, 'bin', 'java')
ES_JAVA_OPTS = '-Delasticsearch'
ES_JAVA_OPTS += ' -Des-foreground=yes'
ES_JAVA_OPTS += ' -Des.path.home="%s"' % server
ES_JAVA_OPTS += ' -Des.path.conf="%s"' % config
ES_JAVA_OPTS += ' -Des.path.logs="%s"' % logs

CLASSPATH = os.environ['CLASSPATH']
ES_CLASSPATH = "%s;%s/;%s/;%s" % (jarPath, libPath, sigarPath,
CLASSPATH)

env =D dict(os.environ)
env['ES_JAVA_OPTS'] = ES_JAVA_OPTS
env['ES_CLASSPATH'] = ES_CLASSPATH

cmd = [JAVA_BIN,
"-Xms128m",
"-Xmx128m",
"-Djline.enabled=false",
"-XX:+AggressiveOpts",
"-XX:+UseParNewGC",
"-XX:+UseConcMarkSweepGC",
"-XX:+CMSParallelRemarkEnabled",
"-XX:+HeapDumpOnOutOfMemoryError",
ES_JAVA_OPTS,
"-cp", ES_CLASSPATH, "org.elasticsearch.bootstrap.Bootstrap"]

start the elasticsearch stub server

try:
p = subprocess.Popen(cmd, shell=False, env=env)
except Exception, e:
raise Exception("Subprocess error: %s" % e)


Regards
Roger Ineichen

Betreff: Re: AW: -Des.path.conf issue?

Sorry, ES_JAVA_OPTS is there in teh script

On Wednesday, March 23, 2011 at 2:41 PM, Shay
Banon wrote:

Using: elasticsearch.bat -Des.... will not work, since
it only get parsed in the elasticsearch.sh file.

For windows, I see that I did not add the option to add
ES_JAVA_OPTS. But if you look at the file, you
can see that
JAVA_OPTS sets some default values, you can add
to that some
more -D parameters.

I need to improve that batch shell...

On Wednesday, March 23, 2011 at 1:32 PM, Roger wrote:

Hi Shay

Betreff: Re: -Des.path.conf issue?

The -D parameter does not work on
windows, only on *nix systems.

Thanks for the quick response!
But now I'm confused. The bin/elasticsearch.bat script
defines the following start parameter:


set

ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSI
ON@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch
-Des-foreground=yes -Des.path.home="%ES_HOME%"

"%JAVA_HOME%\bin\java" %JAVA_OPTS%
%ES_JAVA_OPTS% %ES_PARAMS% -cp
"%ES_CLASSPATH%"
"org.elasticsearch.bootstrap.ElasticSearch"

This means a -D parameter is used e.g.
-Des.path.home="%ES_HOME%"
Is this working or not? Is only the
-Des.path.conf not working?

Regards
Roger Ineichen

On Wednesday, March 23, 2011 at 4:30
AM, Roger wrote:

Hi together

I'm working on an elasticsearch server
stub for python using
a startup via python subprocess.

Everything is working fine except the
es.path.conf property. It seems that the
-Des.path.conf= is not
working on a windows system. I was
testing with windows and
nix config path e.g. \ and /.

I'm trying to use a plain ES
download/unzip and pointing
to an external config folder which
contains some mappings.
The default config folder inside the
donwload/unzip is
still there.

I was reading in some mailinglists that
there is/was an issue
with append/merge config from different
locations, but what
happens to mapping definiteions locate
in config/mappings in
such a configuration?

Is this an issue?

Regards
Roger Ineichen


END OF MESSAGE


(roger-2) #14

Hi Shay

Betreff: Re: AW: -Des.path.conf issue?

Thats great!, does it make sense to integrate it with pyes
maybe? Something like a test module in pyes (I think there is
one now in ElasticSearch.pm).

Yes, why not. I'll post a message if the package is ready to use.
Let's then see how this will fit for pyes testing.

Regards
Roger Ineichen

On Wednesday, March 23, 2011 at 4:39 PM, Roger wrote:

hi Shay, Nick

You are right I need a list of -D options in the cmd.
But my real problem was the following:

' -Des.path.conf="%s"' % config

This ends in a path like:

"C:\foobar"/elasticsearch.yml

This doesn't work because of the "" in the path string, eeeek

Thanks a lot for your help.

btw,
I will soon release a python package which
offers a stub server setup which can be used in
python doc tests. This will (for the first time)
download an elasticserver release and start the server
for testing against with something like:


def doctestSetUp(test):

use default elasticsearch stub with our server dir

here = os.path.dirname(file)
serverDir = os.path.join(here, 'server')
confDir = os.path.join(here, 'config')
es01.stub.testing.startElasticSearchServer(serverDir=serverDir,
confDir=confDir)

def doctestTearDown(test):
es01.stub.testing.stopElasticSearchServer()

def test_suite():
return unittest.TestSuite((
doctest.DocFileSuite('README.txt',
setUp=doctestSetUp,
tearDown=DdoctestTearDown,
optionflags=doctest.NORMALIZE_WHITESPACE|doctest.ELLIPSIS,
encoding='utf-8'),
))

if name == 'main':
unittest.main(defaultTest='test_suite')

Regards
Roger Ineichen

  -----Ursprüngliche Nachricht-----
  Von: Shay Banon [mailto:shay.banon@elasticsearch.com] 
  Gesendet: Mittwoch, 23. März 2011 14:22
  An: users@elasticsearch.com
  Betreff: Re: -Des.path.conf issue?
  
  If you fork the Java process, then you need to 

pass each -D
option by itself, the same way you pass -Xmx,
-Xms, and so on
(your cmd var). Also, remove the AgressiveOpts
from there, I
am updating the batch script to not include it as well.

  On Wednesday, March 23, 2011 at 3:18 PM, Roger wrote:
  
  
  Hi Shay, Nick 
  
  
  Sorry I was a littlebit unclear in my initial mail.
  I'm not using the *.bat script. I'm developing a 
  python doctest stub server lib which starts the
  ES server from a python subproces. This lib allows
  to configure the ES server.
  
  I'm not able to set the path.conf in any way.
  My code looks something like right now:
  
  ---
  # setup java options and friends
  libPath = os.path.join(server, 'lib')
  sigarPath = os.path.join(libPath, 'sigar')
  jarPath = os.path.join(libPath, 

'elasticsearch-0.15.2.jar')
JAVA_HOME = os.environ['JAVA_HOME']
JAVA_BIN = os.path.join(JAVA_HOME, 'bin', 'java')
ES_JAVA_OPTS = '-Delasticsearch'
ES_JAVA_OPTS += ' -Des-foreground=yes'
ES_JAVA_OPTS += ' -Des.path.home="%s"' % server
ES_JAVA_OPTS += ' -Des.path.conf="%s"' % config
ES_JAVA_OPTS += ' -Des.path.logs="%s"' % logs

  CLASSPATH = os.environ['CLASSPATH']
  ES_CLASSPATH = "%s;%s/*;%s/*;%s" % (jarPath, 

libPath, sigarPath,
CLASSPATH)

  env =D dict(os.environ)
  env['ES_JAVA_OPTS'] = ES_JAVA_OPTS
  env['ES_CLASSPATH'] = ES_CLASSPATH
  
  cmd = [JAVA_BIN,
  "-Xms128m",
  "-Xmx128m",
  "-Djline.enabled=false",
  "-XX:+AggressiveOpts",
  "-XX:+UseParNewGC",
  "-XX:+UseConcMarkSweepGC",
  "-XX:+CMSParallelRemarkEnabled",
  "-XX:+HeapDumpOnOutOfMemoryError",
  ES_JAVA_OPTS,
  "-cp", ES_CLASSPATH, 

"org.elasticsearch.bootstrap.Bootstrap"]

  ## start the elasticsearch stub server
  try:
  p = subprocess.Popen(cmd, shell=False, env=env)
  except Exception, e:
  raise Exception("Subprocess error: %s" % e)
  
  ---
  
  Regards
  Roger Ineichen
  
  
  
  Betreff: Re: AW: -Des.path.conf issue?
  
  Sorry, ES_JAVA_OPTS is there in teh script
  
  
  On Wednesday, March 23, 2011 at 2:41 PM, Shay 
  Banon wrote:
  
  
  Using: elasticsearch.bat -Des.... will not work, since 
  it only get parsed in the elasticsearch.sh file.
  
  
  For windows, I see that I did not add the option to add 
  ES_JAVA_OPTS. But if you look at the file, you 
  can see that 
  JAVA_OPTS sets some default values, you can add 
  to that some 
  more -D parameters.
  
  
  I need to improve that batch shell...
  
  
  On Wednesday, March 23, 2011 at 1:32 PM, Roger wrote:
  
  
  Hi Shay 
  
  
  
  Betreff: Re: -Des.path.conf issue?
  
  The -D parameter does not work on 
  windows, only on *nix systems.
  
  
  
  Thanks for the quick response!
  But now I'm confused. The bin/elasticsearch.bat script 
  defines the following start parameter:
  
  ---
  set

ES_CLASSPATH=%CLASSPATH%;%ES_HOME%/lib/elasticsearch-@ES_VERSI
ON@.jar;%ES_HO
ME%/lib/;%ES_HOME%/lib/sigar/
set ES_PARAMS=-Delasticsearch
-Des-foreground=yes -Des.path.home="%ES_HOME%"

  "%JAVA_HOME%\bin\java" %JAVA_OPTS% 
  %ES_JAVA_OPTS% %ES_PARAMS% -cp
  "%ES_CLASSPATH%" 
  "org.elasticsearch.bootstrap.ElasticSearch"
  ---
  
  This means a -D parameter is used e.g. 
  -Des.path.home="%ES_HOME%"
  Is this working or not? Is only the 
  -Des.path.conf not working?
  
  Regards
  Roger Ineichen
  
  
  
  On Wednesday, March 23, 2011 at 4:30 
  AM, Roger wrote: 
  
  Hi together
  
  I'm working on an elasticsearch server 
  stub for python using
  a startup via python subprocess.
  
  Everything is working fine except the 
  es.path.conf property. It seems that the 
  -Des.path.conf=<path 
  outside ES_HOME> is not
  working on a windows system. I was 
  testing with windows and
  nix config path e.g. \\ and /.
  
  I'm trying to use a plain ES 
  download/unzip and pointing
  to an external config folder which 
  contains some mappings.
  The default config folder inside the 
  donwload/unzip is
  still there.
  
  I was reading in some mailinglists that 
  there is/was an issue
  with append/merge config from different 
  locations, but what 
  happens to mapping definiteions locate 
  in config/mappings in
  such a configuration?
  
  Is this an issue?
  
  Regards
  Roger Ineichen
  ______________________________
  END OF MESSAGE

(system) #15