Connect to external instance inside app with embedded node

Hello,

I am attempting to connect to an external elasticsearch cluster inside an
application that now embeds elasticsearch and creates its own cluster. I
am using the TransportClient but cannot get it to connect to this external
cluster, it always only finds the embedded cluster. If I use unicast and
point directly to my external es cluster it only finds the embedded cluster
and then rejects my operations saying that I am not part of the cluster -
which makes perfect sense. Is there any way of doing this? Also, I have
tried to use the sniff and it only finds the embedded instance. I am
pointing my TransportClient directly to my external instance, but it does
not find it. Any suggestions?

--

Bump, anyone?

Shouldn't there be a way to hit an external cluster within a jvm that
already has a cluster embedded? I thought that maybe the problem was the
TransportClient picking up the config from the classpath but that didn't
turn out any different results.

Just FYI, this code was working before ElasticSearch was embedded into the
app I am building plugins for. It uses the nodeBuilder() to create its
embedded nodes. Any ideas on how I can break free from the inner cluster?

Thanks for the help!

On Thursday, September 27, 2012 9:24:51 AM UTC-6, Nicholas Padilla wrote:

Hello,

I am attempting to connect to an external elasticsearch cluster inside an
application that now embeds elasticsearch and creates its own cluster. I
am using the TransportClient but cannot get it to connect to this external
cluster, it always only finds the embedded cluster. If I use unicast and
point directly to my external es cluster it only finds the embedded cluster
and then rejects my operations saying that I am not part of the cluster -
which makes perfect sense. Is there any way of doing this? Also, I have
tried to use the sniff and it only finds the embedded instance. I am
pointing my TransportClient directly to my external instance, but it does
not find it. Any suggestions?

--

Hello,

I have recently been playing around with the new version of dotcms, v2.2.
In v1.9 they didn't use ES and so I wrote a plugin to handle some indexing
to a remote cluster, using the TransportClient. Now that dotcms has
embedded an ES node into the application for handling their internal
indices I cannot connect to my remote cluster. I have tried everything,
with no success. I finally started to look into the codebase to see if
there was anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties. With
some testing I found that this was indeed the problem. I have been working
with 0.19 branch and have some small changes that will provide this
functionality without breaking the contract with other users. Should I
commit this to both the 0.19 and master or just master?

Also, I was hoping to be able to have this in the next release - if
possible. I'd rather use your releases to my custom build ones. Please
advise.

Thanks!

On Thursday, September 27, 2012 9:24:51 AM UTC-6, Nicholas Padilla wrote:

Hello,

I am attempting to connect to an external elasticsearch cluster inside an
application that now embeds elasticsearch and creates its own cluster. I
am using the TransportClient but cannot get it to connect to this external
cluster, it always only finds the embedded cluster. If I use unicast and
point directly to my external es cluster it only finds the embedded cluster
and then rejects my operations saying that I am not part of the cluster -
which makes perfect sense. Is there any way of doing this? Also, I have
tried to use the sniff and it only finds the embedded instance. I am
pointing my TransportClient directly to my external instance, but it does
not find it. Any suggestions?

--

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties.
With some testing I found that this was indeed the problem. I have
been working with 0.19 branch and have some small changes that will
provide this functionality without breaking the contract with other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a patch
we can't discuss what you're proposing to change.

-Drew

--

are you sure you have 2 different Client object instances (each with
different properties)? I would have thought this should work, as long as
you have distinct Client instances for your 2 different purposes?

On 7 October 2012 05:16, Nicholas Padilla
nicholas@monstersoftwarellc.comwrote:

Hello,

I have recently been playing around with the new version of dotcms, v2.2.
In v1.9 they didn't use ES and so I wrote a plugin to handle some indexing
to a remote cluster, using the TransportClient. Now that dotcms has
embedded an ES node into the application for handling their internal
indices I cannot connect to my remote cluster. I have tried everything,
with no success. I finally started to look into the codebase to see if
there was anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties. With
some testing I found that this was indeed the problem. I have been working
with 0.19 branch and have some small changes that will provide this
functionality without breaking the contract with other users. Should I
commit this to both the 0.19 and master or just master?

Also, I was hoping to be able to have this in the next release - if
possible. I'd rather use your releases to my custom build ones. Please
advise.

Thanks!

On Thursday, September 27, 2012 9:24:51 AM UTC-6, Nicholas Padilla wrote:

Hello,

I am attempting to connect to an external elasticsearch cluster inside an
application that now embeds elasticsearch and creates its own cluster. I
am using the TransportClient but cannot get it to connect to this external
cluster, it always only finds the embedded cluster. If I use unicast and
point directly to my external es cluster it only finds the embedded cluster
and then rejects my operations saying that I am not part of the cluster -
which makes perfect sense. Is there any way of doing this? Also, I have
tried to use the sniff and it only finds the embedded instance. I am
pointing my TransportClient directly to my external instance, but it does
not find it. Any suggestions?

--

--

I figured the same thing at the start of all this, figured I was just
overlooking a configuration option. After going over everything, and
stepping over the code, I can see that it is distinctly loading System
Properties that override the ones provided by the ImmutableSettings passed
in. There is a parameter for the TransportClient that allows control over
the if settings from the Classpath gets loaded, but not for the System
properties. I have found that everything does indeed work after the small
change, I just want to ensure I commit my changes according to how ES would
like it.

Are there any guidelines doc or web page for the ES project? If not I can
just commit the code to master and send a pull request for that change. I
was hoping it could be added to 0.19 branch - so I could use it in
production. I just wanted to make life easier for anyone administering the
project, follow their workflow.

Either way, thanks for the response! Let me know if you have any other
questions, or have any other info for me.

Thanks!

On Monday, October 8, 2012 4:43:25 PM UTC-6, tallpsmith wrote:

are you sure you have 2 different Client object instances (each with
different properties)? I would have thought this should work, as long as
you have distinct Client instances for your 2 different purposes?

On 7 October 2012 05:16, Nicholas Padilla <nich...@monstersoftwarellc.com<javascript:>

wrote:

Hello,

I have recently been playing around with the new version of dotcms, v2.2.
In v1.9 they didn't use ES and so I wrote a plugin to handle some indexing
to a remote cluster, using the TransportClient. Now that dotcms has
embedded an ES node into the application for handling their internal
indices I cannot connect to my remote cluster. I have tried everything,
with no success. I finally started to look into the codebase to see if
there was anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties. With
some testing I found that this was indeed the problem. I have been working
with 0.19 branch and have some small changes that will provide this
functionality without breaking the contract with other users. Should I
commit this to both the 0.19 and master or just master?

Also, I was hoping to be able to have this in the next release - if
possible. I'd rather use your releases to my custom build ones. Please
advise.

Thanks!

On Thursday, September 27, 2012 9:24:51 AM UTC-6, Nicholas Padilla wrote:

Hello,

I am attempting to connect to an external elasticsearch cluster inside
an application that now embeds elasticsearch and creates its own cluster.
I am using the TransportClient but cannot get it to connect to this
external cluster, it always only finds the embedded cluster. If I use
unicast and point directly to my external es cluster it only finds the
embedded cluster and then rejects my operations saying that I am not part
of the cluster - which makes perfect sense. Is there any way of doing
this? Also, I have tried to use the sniff and it only finds the embedded
instance. I am pointing my TransportClient directly to my external
instance, but it does not find it. Any suggestions?

--

--

On 9 October 2012 09:58, Nicholas Padilla
nicholas@monstersoftwarellc.comwrote:

I figured the same thing at the start of all this, figured I was just
overlooking a configuration option. After going over everything, and
stepping over the code, I can see that it is distinctly loading System
Properties that override the ones provided by the ImmutableSettings passed
in. There is a parameter for the TransportClient that allows control over
the if settings from the Classpath gets loaded, but not for the System
properties. I have found that everything does indeed work after the small
change, I just want to ensure I commit my changes according to how ES would
like it.

ok I see now, so you have some 'rogue' System properties you can't control?
(as in you're required to set them for this other 3rd party system that
uses ES?) that's a bummer.

Are there any guidelines doc or web page for the ES project? If not I can
just commit the code to master and send a pull request for that change. I
was hoping it could be added to 0.19 branch - so I could use it in
production. I just wanted to make life easier for anyone administering the
project, follow their workflow.

yes, I'd commit to your fork of ES Master, and then submit a pull request
with details of the why etc. It's an interesting problem.

Paul

--

Hello,

I have submitted a pull request here:

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties.
With some testing I found that this was indeed the problem. I have
been working with 0.19 branch and have some small changes that will
provide this functionality without breaking the contract with other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a patch
we can't discuss what you're proposing to change.

-Drew

--

Nicholas,

Here is what you could try:

Move all elasticsearch properties from
dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for
example, myes.properties

Replace all elasticsearch properties
in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following
single line that points to the myes.properties file.

es.config=path/to/myes.properties 

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(settingsBuilder, false);

I am not sure how dotcms instantiates embedded elasticsearch node, but if
they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote:

Hello,

I have submitted a pull request here:
https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties.
With some testing I found that this was indeed the problem. I have
been working with 0.19 branch and have some small changes that will
provide this functionality without breaking the contract with other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a patch
we can't discuss what you're proposing to change.

-Drew

--

Hey Igor!

Thanks for the reply! I had thought about that but after looking at how
dotcms loads it properties I found I couldn't move this direction. They
are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla

“Problems cannot be solved by the same level of thinking that created them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop
questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov imotov@gmail.com wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from
dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for
example, myes.properties

Replace all elasticsearch properties
in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following
single line that points to the myes.properties file.

es.config=path/to/myes.properties

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(settingsBuilder, false);

I am not sure how dotcms instantiates embedded elasticsearch node, but if
they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote:

Hello,

I have submitted a pull request here: https://github.com/**
elasticsearch/elasticsearch/**pull/2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties.
With some testing I found that this was indeed the problem. I have
been working with 0.19 branch and have some small changes that will
provide this functionality without breaking the contract with other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a patch
we can't discuss what you're proposing to change.

-Drew

--

--

But where are they adding settings to the System Properties from? Isn't it
dotCMS/WEB-INF/classes/dotmarketing-config.properties? So, if you will move
properties from this file to an external file, as I described above, the
only property that they will add to System Properties will be es.config=
which points to the location of the config file. As a result, the embedded
server will load all properties that it needs from the config file, while
your client will not simply ignore this config file and will use only
properties that you specified.

On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla <
nicholas@monstersoftwarellc.com> wrote:

Hey Igor!

Thanks for the reply! I had thought about that but after looking at how
dotcms loads it properties I found I couldn't move this direction. They
are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov imotov@gmail.com wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from
dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for
example, myes.properties

Replace all elasticsearch properties
in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following
single line that points to the myes.properties file.

es.config=path/to/myes.properties

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(settingsBuilder, false);

I am not sure how dotcms instantiates embedded elasticsearch node, but if
they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote:

Hello,

I have submitted a pull request here: https://github.com/**
elasticsearch/elasticsearch/**pull/2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties.
With some testing I found that this was indeed the problem. I have
been working with 0.19 branch and have some small changes that will
provide this functionality without breaking the contract with other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a patch
we can't discuss what you're proposing to change.

-Drew

--

--

--

Hey Igor,

Hmmm.. your right. Let me take a look to see if they are doing anything
that would cause this idea not to work. Thanks for the idea!

Thanks!

Nicholas Padilla

“Problems cannot be solved by the same level of thinking that created them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop
questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 1:10 PM, Igor Motov igor@motovs.org wrote:

But where are they adding settings to the System Properties from? Isn't it
dotCMS/WEB-INF/classes/dotmarketing-config.properties? So, if you will
move properties from this file to an external file, as I described above,
the only property that they will add to System Properties will be
es.config= which points to the location of the config file. As a result,
the embedded server will load all properties that it needs from the config
file, while your client will not simply ignore this config file and will
use only properties that you specified.

On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla <
nicholas@monstersoftwarellc.com> wrote:

Hey Igor!

Thanks for the reply! I had thought about that but after looking at how
dotcms loads it properties I found I couldn't move this direction. They
are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov imotov@gmail.com wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from
dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for
example, myes.properties

Replace all elasticsearch properties
in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following
single line that points to the myes.properties file.

es.config=path/to/myes.properties

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(settingsBuilder, false);

I am not sure how dotcms instantiates embedded elasticsearch node, but
if they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote:

Hello,

I have submitted a pull request here: https://github.com/**
elasticsearch/elasticsearch/**pull/2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties.
With some testing I found that this was indeed the problem. I have
been working with 0.19 branch and have some small changes that will
provide this functionality without breaking the contract with other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a patch
we can't discuss what you're proposing to change.

-Drew

--

--

--

--

Hey Igor,

I have found that this does work, however the .properties needs to be a
.yml and in the YAML format. This is a workaround and not a final
solution, as it makes upgrading more difficult due to the changing of the
core .properties file. As a dotcms best practice, everything should be
done to customize the application in a config plugin. This is so the
dotcms team can make updates to these files as needed, bug fixes or new
features. This could be a solution if the parent architecture were a bit
different. I have to pass this along with some notes as to how to upgrade,
when you start throwing more cut and paste stuff in there the more room for
error there is.

Thanks for all your help on this matter! We at least have a stop gap
measure in case the code adjustment I added takes a long time to get
committed.

Thanks again!

On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote:

But where are they adding settings to the System Properties from? Isn't it
dotCMS/WEB-INF/classes/dotmarketing-config.properties? So, if you will
move properties from this file to an external file, as I described above,
the only property that they will add to System Properties will be
es.config= which points to the location of the config file. As a result,
the embedded server will load all properties that it needs from the config
file, while your client will not simply ignore this config file and will
use only properties that you specified.

On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla <
nich...@monstersoftwarellc.com <javascript:>> wrote:

Hey Igor!

Thanks for the reply! I had thought about that but after looking at how
dotcms loads it properties I found I couldn't move this direction. They
are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov <imo...@gmail.com<javascript:>

wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from
dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for
example, myes.properties

Replace all elasticsearch properties
in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following
single line that points to the myes.properties file.

es.config=path/to/myes.properties 

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(settingsBuilder, false);

I am not sure how dotcms instantiates embedded elasticsearch node, but
if they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote:

Hello,

I have submitted a pull request here: https://github.com/**
elasticsearch/elasticsearch/**pull/2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties.
With some testing I found that this was indeed the problem. I have
been working with 0.19 branch and have some small changes that will
provide this functionality without breaking the contract with other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a patch
we can't discuss what you're proposing to change.

-Drew

--

--

--

Hi Nicholas,

I agree that it sounds more like a workaround. What I am still struggling
with is figuring out if this is a workaround for an elasticsearch issue or
a dotcms issue. It's easy to add this feature to elasticsearch. But before
we do that, we should ask ourselves a question: Are we solving the right
problem? Elasticsearch already provides several different ways of supplying
configuration. The dotcms application is using the most global way that
overrides settings for all es instances running in the dotcms process and
that affects your use case. The problem can be fixed on the dotcms side by
changing the default configuration files that dotcms is shipping with, or
even better by switching to a config file base configuration instead of
system properties base configuration. So, would it make more sense to fix
it in dotcms rather than add a new feature to elasticsearch? I am not very
familiar with dotcms, so I might be missing something important here. What
do you think?

Igor

On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote:

Hey Igor,

I have found that this does work, however the .properties needs to be a
.yml and in the YAML format. This is a workaround and not a final
solution, as it makes upgrading more difficult due to the changing of the
core .properties file. As a dotcms best practice, everything should be
done to customize the application in a config plugin. This is so the
dotcms team can make updates to these files as needed, bug fixes or new
features. This could be a solution if the parent architecture were a bit
different. I have to pass this along with some notes as to how to upgrade,
when you start throwing more cut and paste stuff in there the more room for
error there is.

Thanks for all your help on this matter! We at least have a stop gap
measure in case the code adjustment I added takes a long time to get
committed.

Thanks again!

On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote:

But where are they adding settings to the System Properties from? Isn't
it dotCMS/WEB-INF/classes/dotmarketing-config.properties? So, if you
will move properties from this file to an external file, as I described
above, the only property that they will add to System Properties will be
es.config= which points to the location of the config file. As a result,
the embedded server will load all properties that it needs from the config
file, while your client will not simply ignore this config file and will
use only properties that you specified.

On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla <
nich...@monstersoftwarellc.com> wrote:

Hey Igor!

Thanks for the reply! I had thought about that but after looking at
how dotcms loads it properties I found I couldn't move this direction.
They are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov imo...@gmail.com wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from
dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for
example, myes.properties

Replace all elasticsearch properties
in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following
single line that points to the myes.properties file.

es.config=path/to/myes.properties 

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(settingsBuilder, false);

I am not sure how dotcms instantiates embedded elasticsearch node, but
if they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote:

Hello,

I have submitted a pull request here: https://github.com/**
elasticsearch/elasticsearch/**pull/2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System Properties.
With some testing I found that this was indeed the problem. I have
been working with 0.19 branch and have some small changes that will
provide this functionality without breaking the contract with other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a patch
we can't discuss what you're proposing to change.

-Drew

--

--

--

Hey Igor,

Yeah, I can agree with that. I think the biggest problem is that the
dotcms codebase isn't as put together as ES. So fixing it this way
affected the least amount of code. I could attempt to restructure their
properties stuff, but there is quite a bit of code that would need to
change - I don't think they would go for it.

Honestly, the upgrade would take far too much work for my client to be
willing to spend the time - especially since we have a work around and a
possible fix already. Dotcms would have to do the work to change that, but
they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to
say that those things don't make sense for them, but that it takes forever
for them to do anything large - the codebase just isn't set up for rapid
development. Also, there is a very large plugin community that could be
affected - further lengthening the time for a fix - if in fact systemic
changes are required. We were hoping to be able to have an instance set up
in December sometime.

So to sum up, I agree that their properties framework needs work. I also
agree that it doesn't make sense to try and put a fix in place that is out
of place. However, having said all that - I also think that there are
plenty of environments that coders have to deal with that limit integration
options - out of lack of foresight or other issues. Giving an ES
TransportClient the option to not load ANY configurations outside of the
ones passed in the Settings object is, I think, reasonable. There are
just too many legacy and/or crappy applications to go around fixing all of
them to be what you or I would expect or want. Our ES plugin predates the
use of ES internally and as a consultant, it is hard to justify that sort
of additional expenditure. Please let me know if you have any questions or
additional concerns.

Thanks for your time!
Nick

Thanks!

Nicholas Padilla

“Problems cannot be solved by the same level of thinking that created them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop
questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov imotov@gmail.com wrote:

Hi Nicholas,

I agree that it sounds more like a workaround. What I am still struggling
with is figuring out if this is a workaround for an elasticsearch issue or
a dotcms issue. It's easy to add this feature to elasticsearch. But before
we do that, we should ask ourselves a question: Are we solving the right
problem? Elasticsearch already provides several different ways of supplying
configuration. The dotcms application is using the most global way that
overrides settings for all es instances running in the dotcms process and
that affects your use case. The problem can be fixed on the dotcms side by
changing the default configuration files that dotcms is shipping with, or
even better by switching to a config file base configuration instead of
system properties base configuration. So, would it make more sense to fix
it in dotcms rather than add a new feature to elasticsearch? I am not very
familiar with dotcms, so I might be missing something important here. What
do you think?

Igor

On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote:

Hey Igor,

I have found that this does work, however the .properties needs to be a
.yml and in the YAML format. This is a workaround and not a final
solution, as it makes upgrading more difficult due to the changing of the
core .properties file. As a dotcms best practice, everything should be
done to customize the application in a config plugin. This is so the
dotcms team can make updates to these files as needed, bug fixes or new
features. This could be a solution if the parent architecture were a bit
different. I have to pass this along with some notes as to how to upgrade,
when you start throwing more cut and paste stuff in there the more room for
error there is.

Thanks for all your help on this matter! We at least have a stop gap
measure in case the code adjustment I added takes a long time to get
committed.

Thanks again!

On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote:

But where are they adding settings to the System Properties from? Isn't
it dotCMS/WEB-INF/classes/dotm**arketing-config.properties? So, if you
will move properties from this file to an external file, as I described
above, the only property that they will add to System Properties will be
es.config= which points to the location of the config file. As a result,
the embedded server will load all properties that it needs from the config
file, while your client will not simply ignore this config file and will
use only properties that you specified.

On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla <
nich...@monstersoftwarellc.**com> wrote:

Hey Igor!

Thanks for the reply! I had thought about that but after looking at
how dotcms loads it properties I found I couldn't move this direction.
They are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The
important thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov imo...@gmail.com wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from dotCMS/WEB-INF/classes/**dotmarketing-config.properties
to another file, for example, myes.properties

Replace all elasticsearch properties in dotCMS/WEB-INF/classes/**dotmarketing-config.properties
with the following single line that points to the myes.properties file.

es.config=path/to/myes.**properties

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(**settingsBuilder, false);

I am not sure how dotcms instantiates embedded **elasticsearch node,
but if they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote:

Hello,

I have submitted a pull request here: https://github.com/elast
icsearch/elasticsearch/**pull/**2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System
Properties.
With some testing I found that this was indeed the problem. I
have
been working with 0.19 branch and have some small changes that
will
provide this functionality without breaking the contract with
other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a patch
we can't discuss what you're proposing to change.

-Drew

--

--

--

--

Hey Igor,

I want to update my client on this in the next few days and wanted to see
what we wanted to do on this. If the code changes will not be integrated
then I will need to talk to my client about what will need to be done to
get this to work with dotcms. If we make changes to the dotcms codebase,
dotcms will no longer support the application - so doing this ourselves is
not going to be possible. We cannot wait for dotcms to decide to work on a
change for a single customer, and currently single use case. I have two
possible solutions to get this working, if in fact we cannot get the change
merged into ES:

  1. Use the work around and suffer the upgrade/maintenance penalties.
  2. Build an application that offers some web services that can take in data
    and index it to the remote(or local) ES instance.

Having to recommend a path that costs more for my client isn't a great
place to be in. I was hoping this small change would be something that
could be processed quickly, if not quickly then at least processed. Either
way I have to figure out what is going to have to happen to get my client
upgraded and running the external index. I just thought of another
solution, rebuild the plugin to use another index provider - solr or
something similar. I prefer ES to anything else so having to do that would
be painful. Please advise.

Thanks!

Nick

On Monday, November 26, 2012 11:11:05 AM UTC-7, Nicholas Padilla wrote:

Hey Igor,

Yeah, I can agree with that. I think the biggest problem is that the
dotcms codebase isn't as put together as ES. So fixing it this way
affected the least amount of code. I could attempt to restructure their
properties stuff, but there is quite a bit of code that would need to
change - I don't think they would go for it.

Honestly, the upgrade would take far too much work for my client to be
willing to spend the time - especially since we have a work around and a
possible fix already. Dotcms would have to do the work to change that, but
they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to
say that those things don't make sense for them, but that it takes forever
for them to do anything large - the codebase just isn't set up for rapid
development. Also, there is a very large plugin community that could be
affected - further lengthening the time for a fix - if in fact systemic
changes are required. We were hoping to be able to have an instance set up
in December sometime.

So to sum up, I agree that their properties framework needs work. I also
agree that it doesn't make sense to try and put a fix in place that is out
of place. However, having said all that - I also think that there are
plenty of environments that coders have to deal with that limit integration
options - out of lack of foresight or other issues. Giving an ES
TransportClient the option to not load ANY configurations outside of the
ones passed in the Settings object is, I think, reasonable. There are
just too many legacy and/or crappy applications to go around fixing all of
them to be what you or I would expect or want. Our ES plugin predates the
use of ES internally and as a consultant, it is hard to justify that sort
of additional expenditure. Please let me know if you have any questions or
additional concerns.

Thanks for your time!
Nick

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov imotov@gmail.com wrote:

Hi Nicholas,

I agree that it sounds more like a workaround. What I am still struggling
with is figuring out if this is a workaround for an elasticsearch issue or
a dotcms issue. It's easy to add this feature to elasticsearch. But before
we do that, we should ask ourselves a question: Are we solving the right
problem? Elasticsearch already provides several different ways of supplying
configuration. The dotcms application is using the most global way that
overrides settings for all es instances running in the dotcms process and
that affects your use case. The problem can be fixed on the dotcms side by
changing the default configuration files that dotcms is shipping with, or
even better by switching to a config file base configuration instead of
system properties base configuration. So, would it make more sense to fix
it in dotcms rather than add a new feature to elasticsearch? I am not very
familiar with dotcms, so I might be missing something important here. What
do you think?

Igor

On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote:

Hey Igor,

I have found that this does work, however the .properties needs to be a
.yml and in the YAML format. This is a workaround and not a final
solution, as it makes upgrading more difficult due to the changing of the
core .properties file. As a dotcms best practice, everything should be
done to customize the application in a config plugin. This is so the
dotcms team can make updates to these files as needed, bug fixes or new
features. This could be a solution if the parent architecture were a bit
different. I have to pass this along with some notes as to how to upgrade,
when you start throwing more cut and paste stuff in there the more room for
error there is.

Thanks for all your help on this matter! We at least have a stop gap
measure in case the code adjustment I added takes a long time to get
committed.

Thanks again!

On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote:

But where are they adding settings to the System Properties from? Isn't
it dotCMS/WEB-INF/classes/dotm**arketing-config.properties? So, if you
will move properties from this file to an external file, as I described
above, the only property that they will add to System Properties will be
es.config= which points to the location of the config file. As a result,
the embedded server will load all properties that it needs from the config
file, while your client will not simply ignore this config file and will
use only properties that you specified.

On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla <
nich...@monstersoftwarellc.**com> wrote:

Hey Igor!

Thanks for the reply! I had thought about that but after looking at
how dotcms loads it properties I found I couldn't move this direction.
They are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The
important thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov imo...@gmail.com wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from dotCMS/WEB-INF/classes/**dotmarketing-config.properties
to another file, for example, myes.properties

Replace all elasticsearch properties in dotCMS/WEB-INF/classes/**dotmarketing-config.properties
with the following single line that points to the myes.properties file.

es.config=path/to/myes.**properties 

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(**settingsBuilder, false);

I am not sure how dotcms instantiates embedded **elasticsearch node,
but if they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote:

Hello,

I have submitted a pull request here: https://github.com/elast
icsearch/elasticsearch/**pull/**2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System
Properties.
With some testing I found that this was indeed the problem. I
have
been working with 0.19 branch and have some small changes that
will
provide this functionality without breaking the contract with
other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a
patch
we can't discuss what you're proposing to change.

-Drew

--

--

--

--

Hey Igor,

I wanted to check in on this and see what the verdict was. If you have any
questions or additional concerns please let me know.

Thanks!
Nick

On Thursday, November 29, 2012 12:30:40 PM UTC-7, Nicholas Padilla wrote:

Hey Igor,

I want to update my client on this in the next few days and wanted to see
what we wanted to do on this. If the code changes will not be integrated
then I will need to talk to my client about what will need to be done to
get this to work with dotcms. If we make changes to the dotcms codebase,
dotcms will no longer support the application - so doing this ourselves is
not going to be possible. We cannot wait for dotcms to decide to work on a
change for a single customer, and currently single use case. I have two
possible solutions to get this working, if in fact we cannot get the change
merged into ES:

  1. Use the work around and suffer the upgrade/maintenance penalties.
  2. Build an application that offers some web services that can take in
    data and index it to the remote(or local) ES instance.

Having to recommend a path that costs more for my client isn't a great
place to be in. I was hoping this small change would be something that
could be processed quickly, if not quickly then at least processed. Either
way I have to figure out what is going to have to happen to get my client
upgraded and running the external index. I just thought of another
solution, rebuild the plugin to use another index provider - solr or
something similar. I prefer ES to anything else so having to do that would
be painful. Please advise.

Thanks!

Nick

On Monday, November 26, 2012 11:11:05 AM UTC-7, Nicholas Padilla wrote:

Hey Igor,

Yeah, I can agree with that. I think the biggest problem is that the
dotcms codebase isn't as put together as ES. So fixing it this way
affected the least amount of code. I could attempt to restructure their
properties stuff, but there is quite a bit of code that would need to
change - I don't think they would go for it.

Honestly, the upgrade would take far too much work for my client to be
willing to spend the time - especially since we have a work around and a
possible fix already. Dotcms would have to do the work to change that, but
they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to
say that those things don't make sense for them, but that it takes forever
for them to do anything large - the codebase just isn't set up for rapid
development. Also, there is a very large plugin community that could be
affected - further lengthening the time for a fix - if in fact systemic
changes are required. We were hoping to be able to have an instance set up
in December sometime.

So to sum up, I agree that their properties framework needs work. I also
agree that it doesn't make sense to try and put a fix in place that is out
of place. However, having said all that - I also think that there are
plenty of environments that coders have to deal with that limit integration
options - out of lack of foresight or other issues. Giving an ES
TransportClient the option to not load ANY configurations outside of the
ones passed in the Settings object is, I think, reasonable. There are
just too many legacy and/or crappy applications to go around fixing all of
them to be what you or I would expect or want. Our ES plugin predates the
use of ES internally and as a consultant, it is hard to justify that sort
of additional expenditure. Please let me know if you have any questions or
additional concerns.

Thanks for your time!
Nick

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov wrote:

Hi Nicholas,

I agree that it sounds more like a workaround. What I am still
struggling with is figuring out if this is a workaround for an
elasticsearch issue or a dotcms issue. It's easy to add this feature to
elasticsearch. But before we do that, we should ask ourselves a question:
Are we solving the right problem? Elasticsearch already provides several
different ways of supplying configuration. The dotcms application is using
the most global way that overrides settings for all es instances running in
the dotcms process and that affects your use case. The problem can be fixed
on the dotcms side by changing the default configuration files that dotcms
is shipping with, or even better by switching to a config file base
configuration instead of system properties base configuration. So, would it
make more sense to fix it in dotcms rather than add a new feature to
elasticsearch? I am not very familiar with dotcms, so I might be missing
something important here. What do you think?

Igor

On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote:

Hey Igor,

I have found that this does work, however the .properties needs to be a
.yml and in the YAML format. This is a workaround and not a final
solution, as it makes upgrading more difficult due to the changing of the
core .properties file. As a dotcms best practice, everything should be
done to customize the application in a config plugin. This is so the
dotcms team can make updates to these files as needed, bug fixes or new
features. This could be a solution if the parent architecture were a bit
different. I have to pass this along with some notes as to how to upgrade,
when you start throwing more cut and paste stuff in there the more room for
error there is.

Thanks for all your help on this matter! We at least have a stop gap
measure in case the code adjustment I added takes a long time to get
committed.

Thanks again!

On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote:

But where are they adding settings to the System Properties from?
Isn't it dotCMS/WEB-INF/classes/dotm**arketing-config.properties? So,
if you will move properties from this file to an external file, as I
described above, the only property that they will add to System Properties
will be es.config= which points to the location of the config file. As a
result, the embedded server will load all properties that it needs from the
config file, while your client will not simply ignore this config file and
will use only properties that you specified.

On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla wrote:

Hey Igor!

Thanks for the reply! I had thought about that but after looking at
how dotcms loads it properties I found I couldn't move this direction.
They are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The
important thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from dotCMS/WEB-INF/classes/**dotmarketing-config.properties
to another file, for example, myes.properties

Replace all elasticsearch properties in dotCMS/WEB-INF/classes/**dotmarketing-config.properties
with the following single line that points to the myes.properties file.

es.config=path/to/myes.**properties 

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(**settingsBuilder, false);

I am not sure how dotcms instantiates embedded **elasticsearch
node, but if they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote:

Hello,

I have submitted a pull request here: https://github.com/elast
icsearch/elasticsearch/**pull/**2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System
Properties.
With some testing I found that this was indeed the problem. I
have
been working with 0.19 branch and have some small changes that
will
provide this functionality without breaking the contract with
other
users. Should I commit this to both the 0.19 and master or just
master?

Commit it to master and create a pull request. Until we see a
patch
we can't discuss what you're proposing to change.

-Drew

--

--

--

--

Hi Nick,

I just spoke to Shay about this issue an hour ago. We have a slightly
different solution that I think will work for you just as well. I will post
PR for it in a few hours.

Igor

On Wednesday, December 5, 2012 10:40:27 AM UTC-5, Nicholas Padilla wrote:

Hey Igor,

I wanted to check in on this and see what the verdict was. If you have
any questions or additional concerns please let me know.

Thanks!
Nick

On Thursday, November 29, 2012 12:30:40 PM UTC-7, Nicholas Padilla wrote:

Hey Igor,

I want to update my client on this in the next few days and wanted to see
what we wanted to do on this. If the code changes will not be integrated
then I will need to talk to my client about what will need to be done to
get this to work with dotcms. If we make changes to the dotcms codebase,
dotcms will no longer support the application - so doing this ourselves is
not going to be possible. We cannot wait for dotcms to decide to work on a
change for a single customer, and currently single use case. I have two
possible solutions to get this working, if in fact we cannot get the change
merged into ES:

  1. Use the work around and suffer the upgrade/maintenance penalties.
  2. Build an application that offers some web services that can take in
    data and index it to the remote(or local) ES instance.

Having to recommend a path that costs more for my client isn't a great
place to be in. I was hoping this small change would be something that
could be processed quickly, if not quickly then at least processed. Either
way I have to figure out what is going to have to happen to get my client
upgraded and running the external index. I just thought of another
solution, rebuild the plugin to use another index provider - solr or
something similar. I prefer ES to anything else so having to do that would
be painful. Please advise.

Thanks!

Nick

On Monday, November 26, 2012 11:11:05 AM UTC-7, Nicholas Padilla wrote:

Hey Igor,

Yeah, I can agree with that. I think the biggest problem is that the
dotcms codebase isn't as put together as ES. So fixing it this way
affected the least amount of code. I could attempt to restructure their
properties stuff, but there is quite a bit of code that would need to
change - I don't think they would go for it.

Honestly, the upgrade would take far too much work for my client to be
willing to spend the time - especially since we have a work around and a
possible fix already. Dotcms would have to do the work to change that, but
they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to
say that those things don't make sense for them, but that it takes forever
for them to do anything large - the codebase just isn't set up for rapid
development. Also, there is a very large plugin community that could be
affected - further lengthening the time for a fix - if in fact systemic
changes are required. We were hoping to be able to have an instance set up
in December sometime.

So to sum up, I agree that their properties framework needs work. I
also agree that it doesn't make sense to try and put a fix in place that is
out of place. However, having said all that - I also think that there are
plenty of environments that coders have to deal with that limit integration
options - out of lack of foresight or other issues. Giving an ES
TransportClient the option to not load ANY configurations outside of the
ones passed in the Settings object is, I think, reasonable. There are
just too many legacy and/or crappy applications to go around fixing all of
them to be what you or I would expect or want. Our ES plugin predates the
use of ES internally and as a consultant, it is hard to justify that sort
of additional expenditure. Please let me know if you have any questions or
additional concerns.

Thanks for your time!
Nick

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov wrote:

Hi Nicholas,

I agree that it sounds more like a workaround. What I am still
struggling with is figuring out if this is a workaround for an
elasticsearch issue or a dotcms issue. It's easy to add this feature to
elasticsearch. But before we do that, we should ask ourselves a question:
Are we solving the right problem? Elasticsearch already provides several
different ways of supplying configuration. The dotcms application is using
the most global way that overrides settings for all es instances running in
the dotcms process and that affects your use case. The problem can be fixed
on the dotcms side by changing the default configuration files that dotcms
is shipping with, or even better by switching to a config file base
configuration instead of system properties base configuration. So, would it
make more sense to fix it in dotcms rather than add a new feature to
elasticsearch? I am not very familiar with dotcms, so I might be missing
something important here. What do you think?

Igor

On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote:

Hey Igor,

I have found that this does work, however the .properties needs to be
a .yml and in the YAML format. This is a workaround and not a final
solution, as it makes upgrading more difficult due to the changing of the
core .properties file. As a dotcms best practice, everything should be
done to customize the application in a config plugin. This is so the
dotcms team can make updates to these files as needed, bug fixes or new
features. This could be a solution if the parent architecture were a bit
different. I have to pass this along with some notes as to how to upgrade,
when you start throwing more cut and paste stuff in there the more room for
error there is.

Thanks for all your help on this matter! We at least have a stop gap
measure in case the code adjustment I added takes a long time to get
committed.

Thanks again!

On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote:

But where are they adding settings to the System Properties from?
Isn't it dotCMS/WEB-INF/classes/dotm**arketing-config.properties?
So, if you will move properties from this file to an external file, as I
described above, the only property that they will add to System Properties
will be es.config= which points to the location of the config file. As a
result, the embedded server will load all properties that it needs from the
config file, while your client will not simply ignore this config file and
will use only properties that you specified.

On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla wrote:

Hey Igor!

Thanks for the reply! I had thought about that but after looking
at how dotcms loads it properties I found I couldn't move this direction.
They are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that
created them.”
“Learn from yesterday, live for today, hope for tomorrow. The
important thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from dotCMS/WEB-INF/classes/**dotmarketing-config.properties
to another file, for example, myes.properties

Replace all elasticsearch properties in dotCMS/WEB-INF/classes/**dotmarketing-config.properties
with the following single line that points to the myes.properties file.

es.config=path/to/myes.**properties 

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(**settingsBuilder, false);

I am not sure how dotcms instantiates embedded **elasticsearch
node, but if they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla
wrote:

Hello,

I have submitted a pull request here: https://github.com/elast
icsearch/elasticsearch/**pull/**2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System
Properties.
With some testing I found that this was indeed the problem. I
have
been working with 0.19 branch and have some small changes that
will
provide this functionality without breaking the contract with
other
users. Should I commit this to both the 0.19 and master or
just
master?

Commit it to master and create a pull request. Until we see a
patch
we can't discuss what you're proposing to change.

-Drew

--

--

--

--

Hey Igor,

That works! Thanks a million for working with me on this, I really
appreciate it! Please let me know if there is anything I can help with.

Thanks!

Nicholas Padilla

“Problems cannot be solved by the same level of thinking that created them.”
“Learn from yesterday, live for today, hope for tomorrow. The important
thing is not to stop
questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Wed, Dec 5, 2012 at 8:57 AM, Igor Motov imotov@gmail.com wrote:

Hi Nick,

I just spoke to Shay about this issue an hour ago. We have a slightly
different solution that I think will work for you just as well. I will post
PR for it in a few hours.

Igor

On Wednesday, December 5, 2012 10:40:27 AM UTC-5, Nicholas Padilla wrote:

Hey Igor,

I wanted to check in on this and see what the verdict was. If you have
any questions or additional concerns please let me know.

Thanks!
Nick

On Thursday, November 29, 2012 12:30:40 PM UTC-7, Nicholas Padilla wrote:

Hey Igor,

I want to update my client on this in the next few days and wanted to
see what we wanted to do on this. If the code changes will not be
integrated then I will need to talk to my client about what will need to be
done to get this to work with dotcms. If we make changes to the dotcms
codebase, dotcms will no longer support the application - so doing this
ourselves is not going to be possible. We cannot wait for dotcms to decide
to work on a change for a single customer, and currently single use case.
I have two possible solutions to get this working, if in fact we cannot
get the change merged into ES:

  1. Use the work around and suffer the upgrade/maintenance penalties.
  2. Build an application that offers some web services that can take in
    data and index it to the remote(or local) ES instance.

Having to recommend a path that costs more for my client isn't a great
place to be in. I was hoping this small change would be something that
could be processed quickly, if not quickly then at least processed. Either
way I have to figure out what is going to have to happen to get my client
upgraded and running the external index. I just thought of another
solution, rebuild the plugin to use another index provider - solr or
something similar. I prefer ES to anything else so having to do that would
be painful. Please advise.

Thanks!

Nick

On Monday, November 26, 2012 11:11:05 AM UTC-7, Nicholas Padilla wrote:

Hey Igor,

Yeah, I can agree with that. I think the biggest problem is that the
dotcms codebase isn't as put together as ES. So fixing it this way
affected the least amount of code. I could attempt to restructure their
properties stuff, but there is quite a bit of code that would need to
change - I don't think they would go for it.

Honestly, the upgrade would take far too much work for my client to be
willing to spend the time - especially since we have a work around and a
possible fix already. Dotcms would have to do the work to change that, but
they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to
say that those things don't make sense for them, but that it takes forever
for them to do anything large - the codebase just isn't set up for rapid
development. Also, there is a very large plugin community that could be
affected - further lengthening the time for a fix - if in fact systemic
changes are required. We were hoping to be able to have an instance set up
in December sometime.

So to sum up, I agree that their properties framework needs work. I
also agree that it doesn't make sense to try and put a fix in place that is
out of place. However, having said all that - I also think that there are
plenty of environments that coders have to deal with that limit integration
options - out of lack of foresight or other issues. Giving an ES
TransportClient the option to not load ANY configurations outside of the
ones passed in the Settings object is, I think, reasonable. There are
just too many legacy and/or crappy applications to go around fixing all of
them to be what you or I would expect or want. Our ES plugin predates the
use of ES internally and as a consultant, it is hard to justify that sort
of additional expenditure. Please let me know if you have any questions or
additional concerns.

Thanks for your time!
Nick

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that created
them.”
“Learn from yesterday, live for today, hope for tomorrow. The
important thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov wrote:

Hi Nicholas,

I agree that it sounds more like a workaround. What I am still
struggling with is figuring out if this is a workaround for an
elasticsearch issue or a dotcms issue. It's easy to add this feature to
elasticsearch. But before we do that, we should ask ourselves a question:
Are we solving the right problem? Elasticsearch already provides several
different ways of supplying configuration. The dotcms application is using
the most global way that overrides settings for all es instances running in
the dotcms process and that affects your use case. The problem can be fixed
on the dotcms side by changing the default configuration files that dotcms
is shipping with, or even better by switching to a config file base
configuration instead of system properties base configuration. So, would it
make more sense to fix it in dotcms rather than add a new feature to
elasticsearch? I am not very familiar with dotcms, so I might be missing
something important here. What do you think?

Igor

On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote:

Hey Igor,

I have found that this does work, however the .properties needs to be
a .yml and in the YAML format. This is a workaround and not a final
solution, as it makes upgrading more difficult due to the changing of the
core .properties file. As a dotcms best practice, everything should be
done to customize the application in a config plugin. This is so the
dotcms team can make updates to these files as needed, bug fixes or new
features. This could be a solution if the parent architecture were a bit
different. I have to pass this along with some notes as to how to upgrade,
when you start throwing more cut and paste stuff in there the more room for
error there is.

Thanks for all your help on this matter! We at least have a stop gap
measure in case the code adjustment I added takes a long time to get
committed.

Thanks again!

On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote:

But where are they adding settings to the System Properties from?
Isn't it dotCMS/WEB-INF/classes/dotm****arketing-config.properties?
So, if you will move properties from this file to an external file, as I
described above, the only property that they will add to System Properties
will be es.config= which points to the location of the config file. As a
result, the embedded server will load all properties that it needs from the
config file, while your client will not simply ignore this config file and
will use only properties that you specified.

On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla wrote:

Hey Igor!

Thanks for the reply! I had thought about that but after looking
at how dotcms loads it properties I found I couldn't move this direction.
They are adding all properties to the System Properties. Creating a new
TransportClient using the constructor you referenced only restricts from
loading settings from the Classpath. They are adding all settings to the
System Properties and that the TransportClient, by default, loads and
overrides my Settings object from the System Properties. This is where
the problem lies, there isn't a way to tell the TransportClient to NOT load
properties from the System Properties at this point. That is what my patch
does, allows you to say to ONLY use settings applied from the Settings
object.

Thanks!

Nicholas Padilla
www.monstersoftwarellc.com

“Problems cannot be solved by the same level of thinking that
created them.”
“Learn from yesterday, live for today, hope for tomorrow. The
important thing is not to stop questioning.http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html

Albert Einstein http://thinkexist.com/quotes/albert_einstein/

On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov wrote:

Nicholas,

Here is what you could try:

Move all elasticsearch properties from dotCMS/WEB-INF/classes/**
dotmark**eting-config.properties to another file, for example,
myes.properties

Replace all elasticsearch properties in dotCMS/WEB-INF/classes/**
dotm**arketing-config.properties with the following single line
that points to the myes.properties file.

es.config=path/to/myes.**propert**ies

Start your client with loadConfigSettings set to false:

This way embedded elasticsearch in dotcms should get its settings
from myes.properties, but the client will ignore them if you will set
loadConfigSettings to false:

... = new TransportClient(**settingsBuilde**r, false);

I am not sure how dotcms instantiates embedded elasticsearch
node, but if they didn't change defaults, it might work.

On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla
wrote:

Hello,

I have submitted a pull request here: https://github.com/*elast
***icsearch/elasticsearch/**pull/2312https://github.com/elasticsearch/elasticsearch/pull/2312

Thanks!

On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote:

Nicholas Padilla wrote:

I finally started to look into the codebase to see if there
was
anything stood out. I found that the Settings provided to the
TransportClient constructor can be overridden by System
Properties.
With some testing I found that this was indeed the problem. I
have
been working with 0.19 branch and have some small changes that
will
provide this functionality without breaking the contract with
other
users. Should I commit this to both the 0.19 and master or
just
master?

Commit it to master and create a pull request. Until we see a
patch
we can't discuss what you're proposing to change.

-Drew

--

--

--

--

--