# Connect to external instance inside app with embedded node **URL:** \ **Category:** Elasticsearch\ **Created:** [September 27, 2012, 3:24pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171 "2012-09-27T15:24:50Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [September 27, 2012, 3:24pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/1 "2012-09-27T15:24:50Z")
Hello, I am attempting to connect to an external elasticsearch cluster inside an application that now embeds elasticsearch and creates its own cluster. I am using the TransportClient but cannot get it to connect to this external cluster, it always only finds the embedded cluster. If I use unicast and point directly to my external es cluster it only finds the embedded cluster and then rejects my operations saying that I am not part of the cluster - which makes perfect sense. Is there any way of doing this? Also, I have tried to use the sniff and it only finds the embedded instance. I am pointing my TransportClient directly to my external instance, but it does not find it. Any suggestions? -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [October 1, 2012, 7:39pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/2 "2012-10-01T19:39:49Z")
Bump, anyone? Shouldn't there be a way to hit an external cluster within a jvm that already has a cluster embedded? I thought that maybe the problem was the TransportClient picking up the config from the classpath but that didn't turn out any different results. Just FYI, this code was working before Elasticsearch was embedded into the app I am building plugins for. It uses the nodeBuilder() to create its embedded nodes. Any ideas on how I can break free from the inner cluster? Thanks for the help! On Thursday, September 27, 2012 9:24:51 AM UTC-6, Nicholas Padilla wrote: > Hello, > > I am attempting to connect to an external elasticsearch cluster inside an > application that now embeds elasticsearch and creates its own cluster. I > am using the TransportClient but cannot get it to connect to this external > cluster, it always only finds the embedded cluster. If I use unicast and > point directly to my external es cluster it only finds the embedded cluster > and then rejects my operations saying that I am not part of the cluster - > which makes perfect sense. Is there any way of doing this? Also, I have > tried to use the sniff and it only finds the embedded instance. I am > pointing my TransportClient directly to my external instance, but it does > not find it. Any suggestions? -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [October 6, 2012, 6:16pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/3 "2012-10-06T18:16:27Z")
Hello, I have recently been playing around with the new version of dotcms, v2.2. In v1.9 they didn't use ES and so I wrote a plugin to handle some indexing to a remote cluster, using the TransportClient. Now that dotcms has embedded an ES node into the application for handling their internal indices I cannot connect to my remote cluster. I have tried everything, with no success. I finally started to look into the codebase to see if there was anything stood out. I found that the Settings provided to the TransportClient constructor can be overridden by System Properties. With some testing I found that this was indeed the problem. I have been working with 0.19 branch and have some small changes that will provide this functionality without breaking the contract with other users. Should I commit this to both the 0.19 and master or just master? Also, I was hoping to be able to have this in the next release - if possible. I'd rather use your releases to my custom build ones. Please advise. Thanks! On Thursday, September 27, 2012 9:24:51 AM UTC-6, Nicholas Padilla wrote: > Hello, > > I am attempting to connect to an external elasticsearch cluster inside an > application that now embeds elasticsearch and creates its own cluster. I > am using the TransportClient but cannot get it to connect to this external > cluster, it always only finds the embedded cluster. If I use unicast and > point directly to my external es cluster it only finds the embedded cluster > and then rejects my operations saying that I am not part of the cluster - > which makes perfect sense. Is there any way of doing this? Also, I have > tried to use the sniff and it only finds the embedded instance. I am > pointing my TransportClient directly to my external instance, but it does > not find it. Any suggestions? -- ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [October 8, 2012, 2:49pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/4 "2012-10-08T14:49:07Z")
Nicholas Padilla wrote: > I finally started to look into the codebase to see if there was > anything stood out. I found that the Settings provided to the > TransportClient constructor can be overridden by System Properties. > With some testing I found that this was indeed the problem. I have > been working with 0.19 branch and have some small changes that will > provide this functionality without breaking the contract with other > users. Should I commit this to both the 0.19 and master or just > master? Commit it to master and create a pull request. Until we see a patch we can't discuss what you're proposing to change. -Drew -- ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [October 8, 2012, 10:43pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/5 "2012-10-08T22:43:22Z")
are you sure you have 2 different Client object instances (each with different properties)? I would have thought this should work, as long as you have distinct Client instances for your 2 different purposes? On 7 October 2012 05:16, Nicholas Padilla [nicholas@monstersoftwarellc.com](mailto:nicholas@monstersoftwarellc.com)wrote: > Hello, > > I have recently been playing around with the new version of dotcms, v2.2. > In v1.9 they didn't use ES and so I wrote a plugin to handle some indexing > to a remote cluster, using the TransportClient. Now that dotcms has > embedded an ES node into the application for handling their internal > indices I cannot connect to my remote cluster. I have tried everything, > with no success. I finally started to look into the codebase to see if > there was anything stood out. I found that the Settings provided to the > TransportClient constructor can be overridden by System Properties. With > some testing I found that this was indeed the problem. I have been working > with 0.19 branch and have some small changes that will provide this > functionality without breaking the contract with other users. Should I > commit this to both the 0.19 and master or just master? > > Also, I was hoping to be able to have this in the next release - if > possible. I'd rather use your releases to my custom build ones. Please > advise. > > Thanks! > > On Thursday, September 27, 2012 9:24:51 AM UTC-6, Nicholas Padilla wrote: > > > Hello, > > > > I am attempting to connect to an external elasticsearch cluster inside an > > application that now embeds elasticsearch and creates its own cluster. I > > am using the TransportClient but cannot get it to connect to this external > > cluster, it always only finds the embedded cluster. If I use unicast and > > point directly to my external es cluster it only finds the embedded cluster > > and then rejects my operations saying that I am not part of the cluster - > > which makes perfect sense. Is there any way of doing this? Also, I have > > tried to use the sniff and it only finds the embedded instance. I am > > pointing my TransportClient directly to my external instance, but it does > > not find it. Any suggestions? > > -- -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [October 8, 2012, 10:58pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/6 "2012-10-08T22:58:40Z")
I figured the same thing at the start of all this, figured I was just overlooking a configuration option. After going over everything, and stepping over the code, I can see that it is distinctly loading System Properties that override the ones provided by the ImmutableSettings passed in. There is a parameter for the TransportClient that allows control over the if settings from the Classpath gets loaded, but not for the System properties. I have found that everything does indeed work after the small change, I just want to ensure I commit my changes according to how ES would like it. Are there any guidelines doc or web page for the ES project? If not I can just commit the code to master and send a pull request for that change. I was hoping it could be added to 0.19 branch - so I could use it in production. I just wanted to make life easier for anyone administering the project, follow their workflow. Either way, thanks for the response! Let me know if you have any other questions, or have any other info for me. Thanks! On Monday, October 8, 2012 4:43:25 PM UTC-6, tallpsmith wrote: > are you sure you have 2 different Client object instances (each with > different properties)? I would have thought this should work, as long as > you have distinct Client instances for your 2 different purposes? > > On 7 October 2012 05:16, Nicholas Padilla \<[nich...@monstersoftwarellc.com](mailto:nich...@monstersoftwarellc.com)\ > > > wrote: > > > Hello, > > > > I have recently been playing around with the new version of dotcms, v2.2. > > In v1.9 they didn't use ES and so I wrote a plugin to handle some indexing > > to a remote cluster, using the TransportClient. Now that dotcms has > > embedded an ES node into the application for handling their internal > > indices I cannot connect to my remote cluster. I have tried everything, > > with no success. I finally started to look into the codebase to see if > > there was anything stood out. I found that the Settings provided to the > > TransportClient constructor can be overridden by System Properties. With > > some testing I found that this was indeed the problem. I have been working > > with 0.19 branch and have some small changes that will provide this > > functionality without breaking the contract with other users. Should I > > commit this to both the 0.19 and master or just master? > > > > Also, I was hoping to be able to have this in the next release - if > > possible. I'd rather use your releases to my custom build ones. Please > > advise. > > > > Thanks! > > > > On Thursday, September 27, 2012 9:24:51 AM UTC-6, Nicholas Padilla wrote: > > > > > Hello, > > > > > > I am attempting to connect to an external elasticsearch cluster inside > > > an application that now embeds elasticsearch and creates its own cluster. > > > I am using the TransportClient but cannot get it to connect to this > > > external cluster, it always only finds the embedded cluster. If I use > > > unicast and point directly to my external es cluster it only finds the > > > embedded cluster and then rejects my operations saying that I am not part > > > of the cluster - which makes perfect sense. Is there any way of doing > > > this? Also, I have tried to use the sniff and it only finds the embedded > > > instance. I am pointing my TransportClient directly to my external > > > instance, but it does not find it. Any suggestions? > > > > -- -- ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [October 8, 2012, 11:10pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/7 "2012-10-08T23:10:41Z")
On 9 October 2012 09:58, Nicholas Padilla [nicholas@monstersoftwarellc.com](mailto:nicholas@monstersoftwarellc.com)wrote: > I figured the same thing at the start of all this, figured I was just > overlooking a configuration option. After going over everything, and > stepping over the code, I can see that it is distinctly loading System > Properties that override the ones provided by the ImmutableSettings passed > in. There is a parameter for the TransportClient that allows control over > the if settings from the Classpath gets loaded, but not for the System > properties. I have found that everything does indeed work after the small > change, I just want to ensure I commit my changes according to how ES would > like it. ok I see now, so you have some 'rogue' System properties you can't control? (as in you're required to set them for this other 3rd party system that uses ES?) that's a bummer. > Are there any guidelines doc or web page for the ES project? If not I can > just commit the code to master and send a pull request for that change. I > was hoping it could be added to 0.19 branch - so I could use it in > production. I just wanted to make life easier for anyone administering the > project, follow their workflow. yes, I'd commit to your fork of ES Master, and then submit a pull request with details of the why etc. It's an interesting problem. Paul -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [October 9, 2012, 9:32pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/8 "2012-10-09T21:32:02Z")
Hello, I have submitted a pull request here: > > > I had problems when trying to use the TransportClient inside a java application …plugin - that now uses an internal ES node. They are setting all the ES settings via the System Properties, so my settings get overridden. I am now just allowing a bit more control over how the settings get loaded. Thanks! On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > Nicholas Padilla wrote: > > > I finally started to look into the codebase to see if there was > > anything stood out. I found that the Settings provided to the > > TransportClient constructor can be overridden by System Properties. > > With some testing I found that this was indeed the problem. I have > > been working with 0.19 branch and have some small changes that will > > provide this functionality without breaking the contract with other > > users. Should I commit this to both the 0.19 and master or just > > master? > > Commit it to master and create a pull request. Until we see a patch > we can't discuss what you're proposing to change. > > -Drew -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [October 26, 2012, 6:57pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/9 "2012-10-26T18:57:40Z")
Nicholas, Here is what you could try: Move all elasticsearch properties from dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for example, myes.properties Replace all elasticsearch properties in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following single line that points to the myes.properties file. ``` es.config=path/to/myes.properties ``` Start your client with loadConfigSettings set to false: This way embedded elasticsearch in dotcms should get its settings from myes.properties, but the client will ignore them if you will set loadConfigSettings to false: ``` ... = new TransportClient(settingsBuilder, false); ``` I am not sure how dotcms instantiates embedded elasticsearch node, but if they didn't change defaults, it might work. On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote: > Hello, > > I have submitted a pull request here: > [Allow control over if we load settings from System Properties when using TransportClient. by NickPadilla · Pull Request #2312 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/2312) > > Thanks! > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > Nicholas Padilla wrote: > > > > > I finally started to look into the codebase to see if there was > > > anything stood out. I found that the Settings provided to the > > > TransportClient constructor can be overridden by System Properties. > > > With some testing I found that this was indeed the problem. I have > > > been working with 0.19 branch and have some small changes that will > > > provide this functionality without breaking the contract with other > > > users. Should I commit this to both the 0.19 and master or just > > > master? > > > > Commit it to master and create a pull request. Until we see a patch > > we can't discuss what you're proposing to change. > > > > -Drew -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [October 26, 2012, 7:04pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/10 "2012-10-26T19:04:50Z")
Hey Igor! Thanks for the reply! I had thought about that but after looking at how dotcms loads it properties I found I couldn't move this direction. They are adding all properties to the System Properties. Creating a new TransportClient using the constructor you referenced only restricts from loading settings from the Classpath. They are adding all settings to the System Properties and that the TransportClient, by default, loads and overrides my Settings object from the System Properties. This is where the problem lies, there isn't a way to tell the TransportClient to NOT load properties from the System Properties at this point. That is what my patch does, allows you to say to ONLY use settings applied from the Settings object. Thanks! Nicholas Padilla [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) “Problems cannot be solved by the same level of thinking that created them.” “Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) ” Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote: > Nicholas, > > Here is what you could try: > > Move all elasticsearch properties from > dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for > example, myes.properties > > Replace all elasticsearch properties > in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following > single line that points to the myes.properties file. > > ``` > es.config=path/to/myes.properties > > ``` > > Start your client with loadConfigSettings set to false: > > This way embedded elasticsearch in dotcms should get its settings > from myes.properties, but the client will ignore them if you will set > loadConfigSettings to false: > > ``` > ... = new TransportClient(settingsBuilder, false); > > ``` > > I am not sure how dotcms instantiates embedded elasticsearch node, but if > they didn't change defaults, it might work. > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote: > > > Hello, > > > > I have submitted a pull request here: [https://github.com/](https://github.com/)\*\* > > elasticsearch/elasticsearch/\*\*pull/2312[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > Thanks! > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > Nicholas Padilla wrote: > > > > > > > I finally started to look into the codebase to see if there was > > > > anything stood out. I found that the Settings provided to the > > > > TransportClient constructor can be overridden by System Properties. > > > > With some testing I found that this was indeed the problem. I have > > > > been working with 0.19 branch and have some small changes that will > > > > provide this functionality without breaking the contract with other > > > > users. Should I commit this to both the 0.19 and master or just > > > > master? > > > > > > Commit it to master and create a pull request. Until we see a patch > > > we can't discuss what you're proposing to change. > > > > > > -Drew > > > > -- -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [October 26, 2012, 7:10pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/11 "2012-10-26T19:10:32Z")
But where are they adding settings to the System Properties from? Isn't it dotCMS/WEB-INF/classes/dotmarketing-config.properties? So, if you will move properties from this file to an external file, as I described above, the only property that they will add to System Properties will be es.config= which points to the location of the config file. As a result, the embedded server will load all properties that it needs from the config file, while your client will not simply ignore this config file and will use only properties that you specified. On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla \< [nicholas@monstersoftwarellc.com](mailto:nicholas@monstersoftwarellc.com)\> wrote: > Hey Igor! > > Thanks for the reply! I had thought about that but after looking at how > dotcms loads it properties I found I couldn't move this direction. They > are adding all properties to the System Properties. Creating a new > TransportClient using the constructor you referenced only restricts from > loading settings from the Classpath. They are adding all settings to the > System Properties and that the TransportClient, by default, loads and > overrides my Settings object from the System Properties. This is where > the problem lies, there isn't a way to tell the TransportClient to NOT load > properties from the System Properties at this point. That is what my patch > does, allows you to say to ONLY use settings applied from the Settings > object. > > Thanks! > > Nicholas Padilla > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > “Problems cannot be solved by the same level of thinking that created > them.” > “Learn from yesterday, live for today, hope for tomorrow. The important > thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > ” > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote: > > > Nicholas, > > > > Here is what you could try: > > > > Move all elasticsearch properties from > > dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for > > example, myes.properties > > > > Replace all elasticsearch properties > > in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following > > single line that points to the myes.properties file. > > > > ``` > > es.config=path/to/myes.properties > > > > ``` > > > > Start your client with loadConfigSettings set to false: > > > > This way embedded elasticsearch in dotcms should get its settings > > from myes.properties, but the client will ignore them if you will set > > loadConfigSettings to false: > > > > ``` > > ... = new TransportClient(settingsBuilder, false); > > > > ``` > > > > I am not sure how dotcms instantiates embedded elasticsearch node, but if > > they didn't change defaults, it might work. > > > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote: > > > > > Hello, > > > > > > I have submitted a pull request here: [https://github.com/](https://github.com/)\*\* > > > elasticsearch/elasticsearch/\*\*pull/2312[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > > > Thanks! > > > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > > > Nicholas Padilla wrote: > > > > > > > > > I finally started to look into the codebase to see if there was > > > > > anything stood out. I found that the Settings provided to the > > > > > TransportClient constructor can be overridden by System Properties. > > > > > With some testing I found that this was indeed the problem. I have > > > > > been working with 0.19 branch and have some small changes that will > > > > > provide this functionality without breaking the contract with other > > > > > users. Should I commit this to both the 0.19 and master or just > > > > > master? > > > > > > > > Commit it to master and create a pull request. Until we see a patch > > > > we can't discuss what you're proposing to change. > > > > > > > > -Drew > > > > > > -- > > -- -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [October 26, 2012, 7:15pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/12 "2012-10-26T19:15:56Z")
Hey Igor, Hmmm.. your right. Let me take a look to see if they are doing anything that would cause this idea not to work. Thanks for the idea! Thanks! Nicholas Padilla [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) “Problems cannot be solved by the same level of thinking that created them.” “Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) ” Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) On Fri, Oct 26, 2012 at 1:10 PM, Igor Motov [igor@motovs.org](mailto:igor@motovs.org) wrote: > But where are they adding settings to the System Properties from? Isn't it > dotCMS/WEB-INF/classes/dotmarketing-config.properties? So, if you will > move properties from this file to an external file, as I described above, > the only property that they will add to System Properties will be > es.config= which points to the location of the config file. As a result, > the embedded server will load all properties that it needs from the config > file, while your client will not simply ignore this config file and will > use only properties that you specified. > > On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla \< > [nicholas@monstersoftwarellc.com](mailto:nicholas@monstersoftwarellc.com)\> wrote: > > > Hey Igor! > > > > Thanks for the reply! I had thought about that but after looking at how > > dotcms loads it properties I found I couldn't move this direction. They > > are adding all properties to the System Properties. Creating a new > > TransportClient using the constructor you referenced only restricts from > > loading settings from the Classpath. They are adding all settings to the > > System Properties and that the TransportClient, by default, loads and > > overrides my Settings object from the System Properties. This is where > > the problem lies, there isn't a way to tell the TransportClient to NOT load > > properties from the System Properties at this point. That is what my patch > > does, allows you to say to ONLY use settings applied from the Settings > > object. > > > > Thanks! > > > > Nicholas Padilla > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > “Problems cannot be solved by the same level of thinking that created > > them.” > > “Learn from yesterday, live for today, hope for tomorrow. The important > > thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > ” > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote: > > > > > Nicholas, > > > > > > Here is what you could try: > > > > > > Move all elasticsearch properties from > > > dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for > > > example, myes.properties > > > > > > Replace all elasticsearch properties > > > in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following > > > single line that points to the myes.properties file. > > > > > > ``` > > > es.config=path/to/myes.properties > > > > > > ``` > > > > > > Start your client with loadConfigSettings set to false: > > > > > > This way embedded elasticsearch in dotcms should get its settings > > > from myes.properties, but the client will ignore them if you will set > > > loadConfigSettings to false: > > > > > > ``` > > > ... = new TransportClient(settingsBuilder, false); > > > > > > ``` > > > > > > I am not sure how dotcms instantiates embedded elasticsearch node, but > > > if they didn't change defaults, it might work. > > > > > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote: > > > > > > > Hello, > > > > > > > > I have submitted a pull request here: [https://github.com/](https://github.com/)\*\* > > > > elasticsearch/elasticsearch/\*\*pull/2312[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > > > > > Thanks! > > > > > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > > > > > Nicholas Padilla wrote: > > > > > > > > > > > I finally started to look into the codebase to see if there was > > > > > > anything stood out. I found that the Settings provided to the > > > > > > TransportClient constructor can be overridden by System Properties. > > > > > > With some testing I found that this was indeed the problem. I have > > > > > > been working with 0.19 branch and have some small changes that will > > > > > > provide this functionality without breaking the contract with other > > > > > > users. Should I commit this to both the 0.19 and master or just > > > > > > master? > > > > > > > > > > Commit it to master and create a pull request. Until we see a patch > > > > > we can't discuss what you're proposing to change. > > > > > > > > > > -Drew > > > > > > > > -- > > > > -- > > -- -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [November 26, 2012, 4:02pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/13 "2012-11-26T16:02:39Z")
Hey Igor, I have found that this does work, however the .properties needs to be a .yml and in the YAML format. This is a workaround and not a final solution, as it makes upgrading more difficult due to the changing of the core .properties file. As a dotcms best practice, everything should be done to customize the application in a config plugin. This is so the dotcms team can make updates to these files as needed, bug fixes or new features. This could be a solution if the parent architecture were a bit different. I have to pass this along with some notes as to how to upgrade, when you start throwing more cut and paste stuff in there the more room for error there is. Thanks for all your help on this matter! We at least have a stop gap measure in case the code adjustment I added takes a long time to get committed. Thanks again! On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote: > But where are they adding settings to the System Properties from? Isn't it > dotCMS/WEB-INF/classes/dotmarketing-config.properties? So, if you will > move properties from this file to an external file, as I described above, > the only property that they will add to System Properties will be > es.config= which points to the location of the config file. As a result, > the embedded server will load all properties that it needs from the config > file, while your client will not simply ignore this config file and will > use only properties that you specified. > > On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla \< > [nich...@monstersoftwarellc.com](mailto:nich...@monstersoftwarellc.com) \\> wrote: > > > Hey Igor! > > > > Thanks for the reply! I had thought about that but after looking at how > > dotcms loads it properties I found I couldn't move this direction. They > > are adding all properties to the System Properties. Creating a new > > TransportClient using the constructor you referenced only restricts from > > loading settings from the Classpath. They are adding all settings to the > > System Properties and that the TransportClient, by default, loads and > > overrides my Settings object from the System Properties. This is where > > the problem lies, there isn't a way to tell the TransportClient to NOT load > > properties from the System Properties at this point. That is what my patch > > does, allows you to say to ONLY use settings applied from the Settings > > object. > > > > Thanks! > > > > Nicholas Padilla > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > “Problems cannot be solved by the same level of thinking that created > > them.” > > “Learn from yesterday, live for today, hope for tomorrow. The important > > thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > ” > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov \<[imo...@gmail.com](mailto:imo...@gmail.com)\ > > > > > wrote: > > > > > Nicholas, > > > > > > Here is what you could try: > > > > > > Move all elasticsearch properties from > > > dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for > > > example, myes.properties > > > > > > Replace all elasticsearch properties > > > in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following > > > single line that points to the myes.properties file. > > > > > > ``` > > > es.config=path/to/myes.properties > > > > > > ``` > > > > > > Start your client with loadConfigSettings set to false: > > > > > > This way embedded elasticsearch in dotcms should get its settings > > > from myes.properties, but the client will ignore them if you will set > > > loadConfigSettings to false: > > > > > > ``` > > > ... = new TransportClient(settingsBuilder, false); > > > > > > ``` > > > > > > I am not sure how dotcms instantiates embedded elasticsearch node, but > > > if they didn't change defaults, it might work. > > > > > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote: > > > > > > > Hello, > > > > > > > > I have submitted a pull request here: [https://github.com/](https://github.com/)\*\* > > > > elasticsearch/elasticsearch/\*\*pull/2312[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > > > > > Thanks! > > > > > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > > > > > Nicholas Padilla wrote: > > > > > > > > > > > I finally started to look into the codebase to see if there was > > > > > > anything stood out. I found that the Settings provided to the > > > > > > TransportClient constructor can be overridden by System Properties. > > > > > > With some testing I found that this was indeed the problem. I have > > > > > > been working with 0.19 branch and have some small changes that will > > > > > > provide this functionality without breaking the contract with other > > > > > > users. Should I commit this to both the 0.19 and master or just > > > > > > master? > > > > > > > > > > Commit it to master and create a pull request. Until we see a patch > > > > > we can't discuss what you're proposing to change. > > > > > > > > > > -Drew > > > > > > > > -- > > > > -- -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [November 26, 2012, 5:42pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/14 "2012-11-26T17:42:33Z")
Hi Nicholas, I agree that it sounds more like a workaround. What I am still struggling with is figuring out if this is a workaround for an elasticsearch issue or a dotcms issue. It's easy to add this feature to elasticsearch. But before we do that, we should ask ourselves a question: Are we solving the right problem? Elasticsearch already provides several different ways of supplying configuration. The dotcms application is using the most global way that overrides settings for all es instances running in the dotcms process and that affects your use case. The problem can be fixed on the dotcms side by changing the default configuration files that dotcms is shipping with, or even better by switching to a config file base configuration instead of system properties base configuration. So, would it make more sense to fix it in dotcms rather than add a new feature to elasticsearch? I am not very familiar with dotcms, so I might be missing something important here. What do you think? Igor On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote: > Hey Igor, > > I have found that this does work, however the .properties needs to be a > .yml and in the YAML format. This is a workaround and not a final > solution, as it makes upgrading more difficult due to the changing of the > core .properties file. As a dotcms best practice, everything should be > done to customize the application in a config plugin. This is so the > dotcms team can make updates to these files as needed, bug fixes or new > features. This could be a solution if the parent architecture were a bit > different. I have to pass this along with some notes as to how to upgrade, > when you start throwing more cut and paste stuff in there the more room for > error there is. > > Thanks for all your help on this matter! We at least have a stop gap > measure in case the code adjustment I added takes a long time to get > committed. > > Thanks again! > > On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote: > > > But where are they adding settings to the System Properties from? Isn't > > it dotCMS/WEB-INF/classes/dotmarketing-config.properties? So, if you > > will move properties from this file to an external file, as I described > > above, the only property that they will add to System Properties will be > > es.config= which points to the location of the config file. As a result, > > the embedded server will load all properties that it needs from the config > > file, while your client will not simply ignore this config file and will > > use only properties that you specified. > > > > On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla \< > > [nich...@monstersoftwarellc.com](mailto:nich...@monstersoftwarellc.com)\> wrote: > > > > > Hey Igor! > > > > > > Thanks for the reply! I had thought about that but after looking at > > > how dotcms loads it properties I found I couldn't move this direction. > > > They are adding all properties to the System Properties. Creating a new > > > TransportClient using the constructor you referenced only restricts from > > > loading settings from the Classpath. They are adding all settings to the > > > System Properties and that the TransportClient, by default, loads and > > > overrides my Settings object from the System Properties. This is where > > > the problem lies, there isn't a way to tell the TransportClient to NOT load > > > properties from the System Properties at this point. That is what my patch > > > does, allows you to say to ONLY use settings applied from the Settings > > > object. > > > > > > Thanks! > > > > > > Nicholas Padilla > > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > > > “Problems cannot be solved by the same level of thinking that created > > > them.” > > > “Learn from yesterday, live for today, hope for tomorrow. The important > > > thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > > ” > > > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > > > On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov [imo...@gmail.com](mailto:imo...@gmail.com) wrote: > > > > > > > Nicholas, > > > > > > > > Here is what you could try: > > > > > > > > Move all elasticsearch properties from > > > > dotCMS/WEB-INF/classes/dotmarketing-config.properties to another file, for > > > > example, myes.properties > > > > > > > > Replace all elasticsearch properties > > > > in dotCMS/WEB-INF/classes/dotmarketing-config.properties with the following > > > > single line that points to the myes.properties file. > > > > > > > > ``` > > > > es.config=path/to/myes.properties > > > > > > > > ``` > > > > > > > > Start your client with loadConfigSettings set to false: > > > > > > > > This way embedded elasticsearch in dotcms should get its settings > > > > from myes.properties, but the client will ignore them if you will set > > > > loadConfigSettings to false: > > > > > > > > ``` > > > > ... = new TransportClient(settingsBuilder, false); > > > > > > > > ``` > > > > > > > > I am not sure how dotcms instantiates embedded elasticsearch node, but > > > > if they didn't change defaults, it might work. > > > > > > > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote: > > > > > > > > > Hello, > > > > > > > > > > I have submitted a pull request here: [https://github.com/](https://github.com/)\*\* > > > > > elasticsearch/elasticsearch/\*\*pull/2312[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > > > > > > > Thanks! > > > > > > > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > > > > > > > Nicholas Padilla wrote: > > > > > > > > > > > > > I finally started to look into the codebase to see if there was > > > > > > > anything stood out. I found that the Settings provided to the > > > > > > > TransportClient constructor can be overridden by System Properties. > > > > > > > With some testing I found that this was indeed the problem. I have > > > > > > > been working with 0.19 branch and have some small changes that will > > > > > > > provide this functionality without breaking the contract with other > > > > > > > users. Should I commit this to both the 0.19 and master or just > > > > > > > master? > > > > > > > > > > > > Commit it to master and create a pull request. Until we see a patch > > > > > > we can't discuss what you're proposing to change. > > > > > > > > > > > > -Drew > > > > > > > > > > -- > > > > > > -- -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [November 26, 2012, 6:11pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/15 "2012-11-26T18:11:00Z")
Hey Igor, Yeah, I can agree with that. I think the biggest problem is that the dotcms codebase isn't as put together as ES. So fixing it this way affected the least amount of code. I could attempt to restructure their properties stuff, but there is quite a bit of code that would need to change - I don't think they would go for it. Honestly, the upgrade would take far too much work for my client to be willing to spend the time - especially since we have a work around and a possible fix already. Dotcms would have to do the work to change that, but they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to say that those things don't make sense for them, but that it takes forever for them to do anything large - the codebase just isn't set up for rapid development. Also, there is a very large plugin community that could be affected - further lengthening the time for a fix - if in fact systemic changes are required. We were hoping to be able to have an instance set up in December sometime. So to sum up, I agree that their properties framework needs work. I also agree that it doesn't make sense to try and put a fix in place that is out of place. However, having said all that - I also think that there are plenty of environments that coders have to deal with that limit integration options - out of lack of foresight or other issues. Giving an ES TransportClient the option to not load ANY configurations outside of the ones passed in the Settings object is, I think, reasonable. There are just too many legacy and/or crappy applications to go around fixing all of them to be what you or I would expect or want. Our ES plugin predates the use of ES internally and as a consultant, it is hard to justify that sort of additional expenditure. Please let me know if you have any questions or additional concerns. Thanks for your time! Nick Thanks! Nicholas Padilla [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) “Problems cannot be solved by the same level of thinking that created them.” “Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) ” Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote: > Hi Nicholas, > > I agree that it sounds more like a workaround. What I am still struggling > with is figuring out if this is a workaround for an elasticsearch issue or > a dotcms issue. It's easy to add this feature to elasticsearch. But before > we do that, we should ask ourselves a question: Are we solving the right > problem? Elasticsearch already provides several different ways of supplying > configuration. The dotcms application is using the most global way that > overrides settings for all es instances running in the dotcms process and > that affects your use case. The problem can be fixed on the dotcms side by > changing the default configuration files that dotcms is shipping with, or > even better by switching to a config file base configuration instead of > system properties base configuration. So, would it make more sense to fix > it in dotcms rather than add a new feature to elasticsearch? I am not very > familiar with dotcms, so I might be missing something important here. What > do you think? > > Igor > > On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote: > > > Hey Igor, > > > > I have found that this does work, however the .properties needs to be a > > .yml and in the YAML format. This is a workaround and not a final > > solution, as it makes upgrading more difficult due to the changing of the > > core .properties file. As a dotcms best practice, everything should be > > done to customize the application in a config plugin. This is so the > > dotcms team can make updates to these files as needed, bug fixes or new > > features. This could be a solution if the parent architecture were a bit > > different. I have to pass this along with some notes as to how to upgrade, > > when you start throwing more cut and paste stuff in there the more room for > > error there is. > > > > Thanks for all your help on this matter! We at least have a stop gap > > measure in case the code adjustment I added takes a long time to get > > committed. > > > > Thanks again! > > > > On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote: > > > > > But where are they adding settings to the System Properties from? Isn't > > > it dotCMS/WEB-INF/classes/dotm\*\*arketing-config.properties? So, if you > > > will move properties from this file to an external file, as I described > > > above, the only property that they will add to System Properties will be > > > es.config= which points to the location of the config file. As a result, > > > the embedded server will load all properties that it needs from the config > > > file, while your client will not simply ignore this config file and will > > > use only properties that you specified. > > > > > > On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla \< > > > nich...@monstersoftwarellc.\*\*com\> wrote: > > > > > > > Hey Igor! > > > > > > > > Thanks for the reply! I had thought about that but after looking at > > > > how dotcms loads it properties I found I couldn't move this direction. > > > > They are adding all properties to the System Properties. Creating a new > > > > TransportClient using the constructor you referenced only restricts from > > > > loading settings from the Classpath. They are adding all settings to the > > > > System Properties and that the TransportClient, by default, loads and > > > > overrides my Settings object from the System Properties. This is where > > > > the problem lies, there isn't a way to tell the TransportClient to NOT load > > > > properties from the System Properties at this point. That is what my patch > > > > does, allows you to say to ONLY use settings applied from the Settings > > > > object. > > > > > > > > Thanks! > > > > > > > > Nicholas Padilla > > > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > > > > > “Problems cannot be solved by the same level of thinking that created > > > > them.” > > > > “Learn from yesterday, live for today, hope for tomorrow. The > > > > important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > > > ” > > > > > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > > > > > On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov [imo...@gmail.com](mailto:imo...@gmail.com) wrote: > > > > > > > > > Nicholas, > > > > > > > > > > Here is what you could try: > > > > > > > > > > Move all elasticsearch properties from dotCMS/WEB-INF/classes/\*\*dotmarketing-config.properties > > > > > to another file, for example, myes.properties > > > > > > > > > > Replace all elasticsearch properties in dotCMS/WEB-INF/classes/\*\*dotmarketing-config.properties > > > > > with the following single line that points to the myes.properties file. > > > > > > > > > > ``` > > > > > es.config=path/to/myes.**properties > > > > > > > > > > ``` > > > > > > > > > > Start your client with loadConfigSettings set to false: > > > > > > > > > > This way embedded elasticsearch in dotcms should get its settings > > > > > from myes.properties, but the client will ignore them if you will set > > > > > loadConfigSettings to false: > > > > > > > > > > ``` > > > > > ... = new TransportClient(**settingsBuilder, false); > > > > > > > > > > ``` > > > > > > > > > > I am not sure how dotcms instantiates embedded \*\*elasticsearch node, > > > > > but if they didn't change defaults, it might work. > > > > > > > > > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > I have submitted a pull request here: [https://github.com/\*\*elast](https://github.com/**elast)\*\* > > > > > > icsearch/elasticsearch/\*\*pull/\*\*2312[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > > > > > > > > > Thanks! > > > > > > > > > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > > > > > > > > > Nicholas Padilla wrote: > > > > > > > > > > > > > > > I finally started to look into the codebase to see if there was > > > > > > > > anything stood out. I found that the Settings provided to the > > > > > > > > TransportClient constructor can be overridden by System > > > > > > > > Properties. > > > > > > > > With some testing I found that this was indeed the problem. I > > > > > > > > have > > > > > > > > been working with 0.19 branch and have some small changes that > > > > > > > > will > > > > > > > > provide this functionality without breaking the contract with > > > > > > > > other > > > > > > > > users. Should I commit this to both the 0.19 and master or just > > > > > > > > master? > > > > > > > > > > > > > > Commit it to master and create a pull request. Until we see a patch > > > > > > > we can't discuss what you're proposing to change. > > > > > > > > > > > > > > -Drew > > > > > > > > > > > > -- > > > > > > > > -- > > > > > > -- -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [November 29, 2012, 7:30pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/16 "2012-11-29T19:30:40Z")
Hey Igor, I want to update my client on this in the next few days and wanted to see what we wanted to do on this. If the code changes will not be integrated then I will need to talk to my client about what will need to be done to get this to work with dotcms. If we make changes to the dotcms codebase, dotcms will no longer support the application - so doing this ourselves is not going to be possible. We cannot wait for dotcms to decide to work on a change for a single customer, and currently single use case. I have two possible solutions to get this working, if in fact we cannot get the change merged into ES: 1. Use the work around and suffer the upgrade/maintenance penalties. 2. Build an application that offers some web services that can take in data and index it to the remote(or local) ES instance. Having to recommend a path that costs more for my client isn't a great place to be in. I was hoping this small change would be something that could be processed quickly, if not quickly then at least processed. Either way I have to figure out what is going to have to happen to get my client upgraded and running the external index. I just thought of another solution, rebuild the plugin to use another index provider - solr or something similar. I prefer ES to anything else so having to do that would be painful. Please advise. Thanks! Nick On Monday, November 26, 2012 11:11:05 AM UTC-7, Nicholas Padilla wrote: > Hey Igor, > > Yeah, I can agree with that. I think the biggest problem is that the > dotcms codebase isn't as put together as ES. So fixing it this way > affected the least amount of code. I could attempt to restructure their > properties stuff, but there is quite a bit of code that would need to > change - I don't think they would go for it. > > Honestly, the upgrade would take far too much work for my client to be > willing to spend the time - especially since we have a work around and a > possible fix already. Dotcms would have to do the work to change that, but > they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to > say that those things don't make sense for them, but that it takes forever > for them to do anything large - the codebase just isn't set up for rapid > development. Also, there is a very large plugin community that could be > affected - further lengthening the time for a fix - if in fact systemic > changes are required. We were hoping to be able to have an instance set up > in December sometime. > > So to sum up, I agree that their properties framework needs work. I also > agree that it doesn't make sense to try and put a fix in place that is out > of place. However, having said all that - I also think that there are > plenty of environments that coders have to deal with that limit integration > options - out of lack of foresight or other issues. Giving an ES > TransportClient the option to not load ANY configurations outside of the > ones passed in the Settings object is, I think, reasonable. There are > just too many legacy and/or crappy applications to go around fixing all of > them to be what you or I would expect or want. Our ES plugin predates the > use of ES internally and as a consultant, it is hard to justify that sort > of additional expenditure. Please let me know if you have any questions or > additional concerns. > > Thanks for your time! > Nick > > Thanks! > > Nicholas Padilla > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > “Problems cannot be solved by the same level of thinking that created > them.” > “Learn from yesterday, live for today, hope for tomorrow. The important > thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > ” > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote: > > > Hi Nicholas, > > > > I agree that it sounds more like a workaround. What I am still struggling > > with is figuring out if this is a workaround for an elasticsearch issue or > > a dotcms issue. It's easy to add this feature to elasticsearch. But before > > we do that, we should ask ourselves a question: Are we solving the right > > problem? Elasticsearch already provides several different ways of supplying > > configuration. The dotcms application is using the most global way that > > overrides settings for all es instances running in the dotcms process and > > that affects your use case. The problem can be fixed on the dotcms side by > > changing the default configuration files that dotcms is shipping with, or > > even better by switching to a config file base configuration instead of > > system properties base configuration. So, would it make more sense to fix > > it in dotcms rather than add a new feature to elasticsearch? I am not very > > familiar with dotcms, so I might be missing something important here. What > > do you think? > > > > Igor > > > > On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote: > > > > > Hey Igor, > > > > > > I have found that this does work, however the .properties needs to be a > > > .yml and in the YAML format. This is a workaround and not a final > > > solution, as it makes upgrading more difficult due to the changing of the > > > core .properties file. As a dotcms best practice, everything should be > > > done to customize the application in a config plugin. This is so the > > > dotcms team can make updates to these files as needed, bug fixes or new > > > features. This could be a solution if the parent architecture were a bit > > > different. I have to pass this along with some notes as to how to upgrade, > > > when you start throwing more cut and paste stuff in there the more room for > > > error there is. > > > > > > Thanks for all your help on this matter! We at least have a stop gap > > > measure in case the code adjustment I added takes a long time to get > > > committed. > > > > > > Thanks again! > > > > > > On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote: > > > > > > > But where are they adding settings to the System Properties from? Isn't > > > > it dotCMS/WEB-INF/classes/dotm\*\*arketing-config.properties? So, if you > > > > will move properties from this file to an external file, as I described > > > > above, the only property that they will add to System Properties will be > > > > es.config= which points to the location of the config file. As a result, > > > > the embedded server will load all properties that it needs from the config > > > > file, while your client will not simply ignore this config file and will > > > > use only properties that you specified. > > > > > > > > On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla \< > > > > nich...@monstersoftwarellc.\*\*com\> wrote: > > > > > > > > > Hey Igor! > > > > > > > > > > Thanks for the reply! I had thought about that but after looking at > > > > > how dotcms loads it properties I found I couldn't move this direction. > > > > > They are adding all properties to the System Properties. Creating a new > > > > > TransportClient using the constructor you referenced only restricts from > > > > > loading settings from the Classpath. They are adding all settings to the > > > > > System Properties and that the TransportClient, by default, loads and > > > > > overrides my Settings object from the System Properties. This is where > > > > > the problem lies, there isn't a way to tell the TransportClient to NOT load > > > > > properties from the System Properties at this point. That is what my patch > > > > > does, allows you to say to ONLY use settings applied from the Settings > > > > > object. > > > > > > > > > > Thanks! > > > > > > > > > > Nicholas Padilla > > > > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > > > > > > > “Problems cannot be solved by the same level of thinking that created > > > > > them.” > > > > > “Learn from yesterday, live for today, hope for tomorrow. The > > > > > important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > > > > ” > > > > > > > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > > > > > > > On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov [imo...@gmail.com](mailto:imo...@gmail.com) wrote: > > > > > > > > > > > Nicholas, > > > > > > > > > > > > Here is what you could try: > > > > > > > > > > > > Move all elasticsearch properties from dotCMS/WEB-INF/classes/\*\*dotmarketing-config.properties > > > > > > to another file, for example, myes.properties > > > > > > > > > > > > Replace all elasticsearch properties in dotCMS/WEB-INF/classes/\*\*dotmarketing-config.properties > > > > > > with the following single line that points to the myes.properties file. > > > > > > > > > > > > ``` > > > > > > es.config=path/to/myes.**properties > > > > > > > > > > > > ``` > > > > > > > > > > > > Start your client with loadConfigSettings set to false: > > > > > > > > > > > > This way embedded elasticsearch in dotcms should get its settings > > > > > > from myes.properties, but the client will ignore them if you will set > > > > > > loadConfigSettings to false: > > > > > > > > > > > > ``` > > > > > > ... = new TransportClient(**settingsBuilder, false); > > > > > > > > > > > > ``` > > > > > > > > > > > > I am not sure how dotcms instantiates embedded \*\*elasticsearch node, > > > > > > but if they didn't change defaults, it might work. > > > > > > > > > > > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > I have submitted a pull request here: [https://github.com/\*\*elast](https://github.com/**elast)\*\* > > > > > > > icsearch/elasticsearch/\*\*pull/\*\*2312[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > > > > > > > > > > > Nicholas Padilla wrote: > > > > > > > > > > > > > > > > > I finally started to look into the codebase to see if there was > > > > > > > > > anything stood out. I found that the Settings provided to the > > > > > > > > > TransportClient constructor can be overridden by System > > > > > > > > > Properties. > > > > > > > > > With some testing I found that this was indeed the problem. I > > > > > > > > > have > > > > > > > > > been working with 0.19 branch and have some small changes that > > > > > > > > > will > > > > > > > > > provide this functionality without breaking the contract with > > > > > > > > > other > > > > > > > > > users. Should I commit this to both the 0.19 and master or just > > > > > > > > > master? > > > > > > > > > > > > > > > > Commit it to master and create a pull request. Until we see a > > > > > > > > patch > > > > > > > > we can't discuss what you're proposing to change. > > > > > > > > > > > > > > > > -Drew > > > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > -- -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [December 5, 2012, 3:40pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/17 "2012-12-05T15:40:27Z")
Hey Igor, I wanted to check in on this and see what the verdict was. If you have any questions or additional concerns please let me know. Thanks! Nick On Thursday, November 29, 2012 12:30:40 PM UTC-7, Nicholas Padilla wrote: > Hey Igor, > > I want to update my client on this in the next few days and wanted to see > what we wanted to do on this. If the code changes will not be integrated > then I will need to talk to my client about what will need to be done to > get this to work with dotcms. If we make changes to the dotcms codebase, > dotcms will no longer support the application - so doing this ourselves is > not going to be possible. We cannot wait for dotcms to decide to work on a > change for a single customer, and currently single use case. I have two > possible solutions to get this working, if in fact we cannot get the change > merged into ES: > > 1. Use the work around and suffer the upgrade/maintenance penalties. > 2. Build an application that offers some web services that can take in > data and index it to the remote(or local) ES instance. > > Having to recommend a path that costs more for my client isn't a great > place to be in. I was hoping this small change would be something that > could be processed quickly, if not quickly then at least processed. Either > way I have to figure out what is going to have to happen to get my client > upgraded and running the external index. I just thought of another > solution, rebuild the plugin to use another index provider - solr or > something similar. I prefer ES to anything else so having to do that would > be painful. Please advise. > > Thanks! > > Nick > > On Monday, November 26, 2012 11:11:05 AM UTC-7, Nicholas Padilla wrote: > > > Hey Igor, > > > > Yeah, I can agree with that. I think the biggest problem is that the > > dotcms codebase isn't as put together as ES. So fixing it this way > > affected the least amount of code. I could attempt to restructure their > > properties stuff, but there is quite a bit of code that would need to > > change - I don't think they would go for it. > > > > Honestly, the upgrade would take far too much work for my client to be > > willing to spend the time - especially since we have a work around and a > > possible fix already. Dotcms would have to do the work to change that, but > > they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to > > say that those things don't make sense for them, but that it takes forever > > for them to do anything large - the codebase just isn't set up for rapid > > development. Also, there is a very large plugin community that could be > > affected - further lengthening the time for a fix - if in fact systemic > > changes are required. We were hoping to be able to have an instance set up > > in December sometime. > > > > So to sum up, I agree that their properties framework needs work. I also > > agree that it doesn't make sense to try and put a fix in place that is out > > of place. However, having said all that - I also think that there are > > plenty of environments that coders have to deal with that limit integration > > options - out of lack of foresight or other issues. Giving an ES > > TransportClient the option to not load ANY configurations outside of the > > ones passed in the Settings object is, I think, reasonable. There are > > just too many legacy and/or crappy applications to go around fixing all of > > them to be what you or I would expect or want. Our ES plugin predates the > > use of ES internally and as a consultant, it is hard to justify that sort > > of additional expenditure. Please let me know if you have any questions or > > additional concerns. > > > > Thanks for your time! > > Nick > > > > Thanks! > > > > Nicholas Padilla > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > “Problems cannot be solved by the same level of thinking that created > > them.” > > “Learn from yesterday, live for today, hope for tomorrow. The important > > thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > ” > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov wrote: > > > > > Hi Nicholas, > > > > > > I agree that it sounds more like a workaround. What I am still > > > struggling with is figuring out if this is a workaround for an > > > elasticsearch issue or a dotcms issue. It's easy to add this feature to > > > elasticsearch. But before we do that, we should ask ourselves a question: > > > Are we solving the right problem? Elasticsearch already provides several > > > different ways of supplying configuration. The dotcms application is using > > > the most global way that overrides settings for all es instances running in > > > the dotcms process and that affects your use case. The problem can be fixed > > > on the dotcms side by changing the default configuration files that dotcms > > > is shipping with, or even better by switching to a config file base > > > configuration instead of system properties base configuration. So, would it > > > make more sense to fix it in dotcms rather than add a new feature to > > > elasticsearch? I am not very familiar with dotcms, so I might be missing > > > something important here. What do you think? > > > > > > Igor > > > > > > On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote: > > > > > > > Hey Igor, > > > > > > > > I have found that this does work, however the .properties needs to be a > > > > .yml and in the YAML format. This is a workaround and not a final > > > > solution, as it makes upgrading more difficult due to the changing of the > > > > core .properties file. As a dotcms best practice, everything should be > > > > done to customize the application in a config plugin. This is so the > > > > dotcms team can make updates to these files as needed, bug fixes or new > > > > features. This could be a solution if the parent architecture were a bit > > > > different. I have to pass this along with some notes as to how to upgrade, > > > > when you start throwing more cut and paste stuff in there the more room for > > > > error there is. > > > > > > > > Thanks for all your help on this matter! We at least have a stop gap > > > > measure in case the code adjustment I added takes a long time to get > > > > committed. > > > > > > > > Thanks again! > > > > > > > > On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote: > > > > > > > > > But where are they adding settings to the System Properties from? > > > > > Isn't it dotCMS/WEB-INF/classes/dotm\*\*arketing-config.properties? So, > > > > > if you will move properties from this file to an external file, as I > > > > > described above, the only property that they will add to System Properties > > > > > will be es.config= which points to the location of the config file. As a > > > > > result, the embedded server will load all properties that it needs from the > > > > > config file, while your client will not simply ignore this config file and > > > > > will use only properties that you specified. > > > > > > > > > > On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla wrote: > > > > > > > > > > > Hey Igor! > > > > > > > > > > > > Thanks for the reply! I had thought about that but after looking at > > > > > > how dotcms loads it properties I found I couldn't move this direction. > > > > > > They are adding all properties to the System Properties. Creating a new > > > > > > TransportClient using the constructor you referenced only restricts from > > > > > > loading settings from the Classpath. They are adding all settings to the > > > > > > System Properties and that the TransportClient, by default, loads and > > > > > > overrides my Settings object from the System Properties. This is where > > > > > > the problem lies, there isn't a way to tell the TransportClient to NOT load > > > > > > properties from the System Properties at this point. That is what my patch > > > > > > does, allows you to say to ONLY use settings applied from the Settings > > > > > > object. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Nicholas Padilla > > > > > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > > > > > > > > > “Problems cannot be solved by the same level of thinking that created > > > > > > them.” > > > > > > “Learn from yesterday, live for today, hope for tomorrow. The > > > > > > important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > > > > > ” > > > > > > > > > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > > > > > > > > > On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov wrote: > > > > > > > > > > > > > Nicholas, > > > > > > > > > > > > > > Here is what you could try: > > > > > > > > > > > > > > Move all elasticsearch properties from dotCMS/WEB-INF/classes/\*\*dotmarketing-config.properties > > > > > > > to another file, for example, myes.properties > > > > > > > > > > > > > > Replace all elasticsearch properties in dotCMS/WEB-INF/classes/\*\*dotmarketing-config.properties > > > > > > > with the following single line that points to the myes.properties file. > > > > > > > > > > > > > > ``` > > > > > > > es.config=path/to/myes.**properties > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > Start your client with loadConfigSettings set to false: > > > > > > > > > > > > > > This way embedded elasticsearch in dotcms should get its settings > > > > > > > from myes.properties, but the client will ignore them if you will set > > > > > > > loadConfigSettings to false: > > > > > > > > > > > > > > ``` > > > > > > > ... = new TransportClient(**settingsBuilder, false); > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > I am not sure how dotcms instantiates embedded \*\*elasticsearch > > > > > > > node, but if they didn't change defaults, it might work. > > > > > > > > > > > > > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla wrote: > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > I have submitted a pull request here: [https://github.com/\*\*elast](https://github.com/**elast)\*\* > > > > > > > > icsearch/elasticsearch/\*\*pull/\*\*2312[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > > > > > > > > > > > > > Nicholas Padilla wrote: > > > > > > > > > > > > > > > > > > > I finally started to look into the codebase to see if there was > > > > > > > > > > anything stood out. I found that the Settings provided to the > > > > > > > > > > TransportClient constructor can be overridden by System > > > > > > > > > > Properties. > > > > > > > > > > With some testing I found that this was indeed the problem. I > > > > > > > > > > have > > > > > > > > > > been working with 0.19 branch and have some small changes that > > > > > > > > > > will > > > > > > > > > > provide this functionality without breaking the contract with > > > > > > > > > > other > > > > > > > > > > users. Should I commit this to both the 0.19 and master or just > > > > > > > > > > master? > > > > > > > > > > > > > > > > > > Commit it to master and create a pull request. Until we see a > > > > > > > > > patch > > > > > > > > > we can't discuss what you're proposing to change. > > > > > > > > > > > > > > > > > > -Drew > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > -- > > > > > > > > > > -- -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [December 5, 2012, 3:57pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/18 "2012-12-05T15:57:37Z")
Hi Nick, I just spoke to Shay about this issue an hour ago. We have a slightly different solution that I think will work for you just as well. I will post PR for it in a few hours. Igor On Wednesday, December 5, 2012 10:40:27 AM UTC-5, Nicholas Padilla wrote: > Hey Igor, > > I wanted to check in on this and see what the verdict was. If you have > any questions or additional concerns please let me know. > > Thanks! > Nick > > On Thursday, November 29, 2012 12:30:40 PM UTC-7, Nicholas Padilla wrote: > > > Hey Igor, > > > > I want to update my client on this in the next few days and wanted to see > > what we wanted to do on this. If the code changes will not be integrated > > then I will need to talk to my client about what will need to be done to > > get this to work with dotcms. If we make changes to the dotcms codebase, > > dotcms will no longer support the application - so doing this ourselves is > > not going to be possible. We cannot wait for dotcms to decide to work on a > > change for a single customer, and currently single use case. I have two > > possible solutions to get this working, if in fact we cannot get the change > > merged into ES: > > > > 1. Use the work around and suffer the upgrade/maintenance penalties. > > 2. Build an application that offers some web services that can take in > > data and index it to the remote(or local) ES instance. > > > > Having to recommend a path that costs more for my client isn't a great > > place to be in. I was hoping this small change would be something that > > could be processed quickly, if not quickly then at least processed. Either > > way I have to figure out what is going to have to happen to get my client > > upgraded and running the external index. I just thought of another > > solution, rebuild the plugin to use another index provider - solr or > > something similar. I prefer ES to anything else so having to do that would > > be painful. Please advise. > > > > Thanks! > > > > Nick > > > > On Monday, November 26, 2012 11:11:05 AM UTC-7, Nicholas Padilla wrote: > > > > > Hey Igor, > > > > > > Yeah, I can agree with that. I think the biggest problem is that the > > > dotcms codebase isn't as put together as ES. So fixing it this way > > > affected the least amount of code. I could attempt to restructure their > > > properties stuff, but there is quite a bit of code that would need to > > > change - I don't think they would go for it. > > > > > > Honestly, the upgrade would take far too much work for my client to be > > > willing to spend the time - especially since we have a work around and a > > > possible fix already. Dotcms would have to do the work to change that, but > > > they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to > > > say that those things don't make sense for them, but that it takes forever > > > for them to do anything large - the codebase just isn't set up for rapid > > > development. Also, there is a very large plugin community that could be > > > affected - further lengthening the time for a fix - if in fact systemic > > > changes are required. We were hoping to be able to have an instance set up > > > in December sometime. > > > > > > So to sum up, I agree that their properties framework needs work. I > > > also agree that it doesn't make sense to try and put a fix in place that is > > > out of place. However, having said all that - I also think that there are > > > plenty of environments that coders have to deal with that limit integration > > > options - out of lack of foresight or other issues. Giving an ES > > > TransportClient the option to not load ANY configurations outside of the > > > ones passed in the Settings object is, I think, reasonable. There are > > > just too many legacy and/or crappy applications to go around fixing all of > > > them to be what you or I would expect or want. Our ES plugin predates the > > > use of ES internally and as a consultant, it is hard to justify that sort > > > of additional expenditure. Please let me know if you have any questions or > > > additional concerns. > > > > > > Thanks for your time! > > > Nick > > > > > > Thanks! > > > > > > Nicholas Padilla > > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > > > “Problems cannot be solved by the same level of thinking that created > > > them.” > > > “Learn from yesterday, live for today, hope for tomorrow. The important > > > thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > > ” > > > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > > > On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov wrote: > > > > > > > Hi Nicholas, > > > > > > > > I agree that it sounds more like a workaround. What I am still > > > > struggling with is figuring out if this is a workaround for an > > > > elasticsearch issue or a dotcms issue. It's easy to add this feature to > > > > elasticsearch. But before we do that, we should ask ourselves a question: > > > > Are we solving the right problem? Elasticsearch already provides several > > > > different ways of supplying configuration. The dotcms application is using > > > > the most global way that overrides settings for all es instances running in > > > > the dotcms process and that affects your use case. The problem can be fixed > > > > on the dotcms side by changing the default configuration files that dotcms > > > > is shipping with, or even better by switching to a config file base > > > > configuration instead of system properties base configuration. So, would it > > > > make more sense to fix it in dotcms rather than add a new feature to > > > > elasticsearch? I am not very familiar with dotcms, so I might be missing > > > > something important here. What do you think? > > > > > > > > Igor > > > > > > > > On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote: > > > > > > > > > Hey Igor, > > > > > > > > > > I have found that this does work, however the .properties needs to be > > > > > a .yml and in the YAML format. This is a workaround and not a final > > > > > solution, as it makes upgrading more difficult due to the changing of the > > > > > core .properties file. As a dotcms best practice, everything should be > > > > > done to customize the application in a config plugin. This is so the > > > > > dotcms team can make updates to these files as needed, bug fixes or new > > > > > features. This could be a solution if the parent architecture were a bit > > > > > different. I have to pass this along with some notes as to how to upgrade, > > > > > when you start throwing more cut and paste stuff in there the more room for > > > > > error there is. > > > > > > > > > > Thanks for all your help on this matter! We at least have a stop gap > > > > > measure in case the code adjustment I added takes a long time to get > > > > > committed. > > > > > > > > > > Thanks again! > > > > > > > > > > On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote: > > > > > > > > > > > But where are they adding settings to the System Properties from? > > > > > > Isn't it dotCMS/WEB-INF/classes/dotm\*\*arketing-config.properties? > > > > > > So, if you will move properties from this file to an external file, as I > > > > > > described above, the only property that they will add to System Properties > > > > > > will be es.config= which points to the location of the config file. As a > > > > > > result, the embedded server will load all properties that it needs from the > > > > > > config file, while your client will not simply ignore this config file and > > > > > > will use only properties that you specified. > > > > > > > > > > > > On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla wrote: > > > > > > > > > > > > > Hey Igor! > > > > > > > > > > > > > > Thanks for the reply! I had thought about that but after looking > > > > > > > at how dotcms loads it properties I found I couldn't move this direction. > > > > > > > They are adding all properties to the System Properties. Creating a new > > > > > > > TransportClient using the constructor you referenced only restricts from > > > > > > > loading settings from the Classpath. They are adding all settings to the > > > > > > > System Properties and that the TransportClient, by default, loads and > > > > > > > overrides my Settings object from the System Properties. This is where > > > > > > > the problem lies, there isn't a way to tell the TransportClient to NOT load > > > > > > > properties from the System Properties at this point. That is what my patch > > > > > > > does, allows you to say to ONLY use settings applied from the Settings > > > > > > > object. > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Nicholas Padilla > > > > > > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > > > > > > > > > > > “Problems cannot be solved by the same level of thinking that > > > > > > > created them.” > > > > > > > “Learn from yesterday, live for today, hope for tomorrow. The > > > > > > > important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > > > > > > ” > > > > > > > > > > > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > > > > > > > > > > > On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov wrote: > > > > > > > > > > > > > > > Nicholas, > > > > > > > > > > > > > > > > Here is what you could try: > > > > > > > > > > > > > > > > Move all elasticsearch properties from dotCMS/WEB-INF/classes/\*\*dotmarketing-config.properties > > > > > > > > to another file, for example, myes.properties > > > > > > > > > > > > > > > > Replace all elasticsearch properties in dotCMS/WEB-INF/classes/\*\*dotmarketing-config.properties > > > > > > > > with the following single line that points to the myes.properties file. > > > > > > > > > > > > > > > > ``` > > > > > > > > es.config=path/to/myes.**properties > > > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > Start your client with loadConfigSettings set to false: > > > > > > > > > > > > > > > > This way embedded elasticsearch in dotcms should get its settings > > > > > > > > from myes.properties, but the client will ignore them if you will set > > > > > > > > loadConfigSettings to false: > > > > > > > > > > > > > > > > ``` > > > > > > > > ... = new TransportClient(**settingsBuilder, false); > > > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > I am not sure how dotcms instantiates embedded \*\*elasticsearch > > > > > > > > node, but if they didn't change defaults, it might work. > > > > > > > > > > > > > > > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > I have submitted a pull request here: [https://github.com/\*\*elast](https://github.com/**elast)\*\* > > > > > > > > > icsearch/elasticsearch/\*\*pull/\*\*2312[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > > > > > > > > > > > > > > > Nicholas Padilla wrote: > > > > > > > > > > > > > > > > > > > > > I finally started to look into the codebase to see if there was > > > > > > > > > > > anything stood out. I found that the Settings provided to the > > > > > > > > > > > TransportClient constructor can be overridden by System > > > > > > > > > > > Properties. > > > > > > > > > > > With some testing I found that this was indeed the problem. I > > > > > > > > > > > have > > > > > > > > > > > been working with 0.19 branch and have some small changes that > > > > > > > > > > > will > > > > > > > > > > > provide this functionality without breaking the contract with > > > > > > > > > > > other > > > > > > > > > > > users. Should I commit this to both the 0.19 and master or > > > > > > > > > > > just > > > > > > > > > > > master? > > > > > > > > > > > > > > > > > > > > Commit it to master and create a pull request. Until we see a > > > > > > > > > > patch > > > > > > > > > > we can't discuss what you're proposing to change. > > > > > > > > > > > > > > > > > > > > -Drew > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > -- > > > > > > > > > > > > -- -- ---
**Author:** ![Nicholas\_Padilla](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@Nicholas\_Padilla](https://discuss.elastic.co/u/Nicholas_Padilla)\ **Post date:** [December 5, 2012, 4:07pm UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/19 "2012-12-05T16:07:37Z")
Hey Igor, That works! Thanks a million for working with me on this, I really appreciate it! Please let me know if there is anything I can help with. Thanks! Nicholas Padilla [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) “Problems cannot be solved by the same level of thinking that created them.” “Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) ” Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) On Wed, Dec 5, 2012 at 8:57 AM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote: > Hi Nick, > > I just spoke to Shay about this issue an hour ago. We have a slightly > different solution that I think will work for you just as well. I will post > PR for it in a few hours. > > Igor > > On Wednesday, December 5, 2012 10:40:27 AM UTC-5, Nicholas Padilla wrote: > > > Hey Igor, > > > > I wanted to check in on this and see what the verdict was. If you have > > any questions or additional concerns please let me know. > > > > Thanks! > > Nick > > > > On Thursday, November 29, 2012 12:30:40 PM UTC-7, Nicholas Padilla wrote: > > > > > Hey Igor, > > > > > > I want to update my client on this in the next few days and wanted to > > > see what we wanted to do on this. If the code changes will not be > > > integrated then I will need to talk to my client about what will need to be > > > done to get this to work with dotcms. If we make changes to the dotcms > > > codebase, dotcms will no longer support the application - so doing this > > > ourselves is not going to be possible. We cannot wait for dotcms to decide > > > to work on a change for a single customer, and currently single use case. > > > I have two possible solutions to get this working, if in fact we cannot > > > get the change merged into ES: > > > > > > 1. Use the work around and suffer the upgrade/maintenance penalties. > > > 2. Build an application that offers some web services that can take in > > > data and index it to the remote(or local) ES instance. > > > > > > Having to recommend a path that costs more for my client isn't a great > > > place to be in. I was hoping this small change would be something that > > > could be processed quickly, if not quickly then at least processed. Either > > > way I have to figure out what is going to have to happen to get my client > > > upgraded and running the external index. I just thought of another > > > solution, rebuild the plugin to use another index provider - solr or > > > something similar. I prefer ES to anything else so having to do that would > > > be painful. Please advise. > > > > > > Thanks! > > > > > > Nick > > > > > > On Monday, November 26, 2012 11:11:05 AM UTC-7, Nicholas Padilla wrote: > > > > > > > Hey Igor, > > > > > > > > Yeah, I can agree with that. I think the biggest problem is that the > > > > dotcms codebase isn't as put together as ES. So fixing it this way > > > > affected the least amount of code. I could attempt to restructure their > > > > properties stuff, but there is quite a bit of code that would need to > > > > change - I don't think they would go for it. > > > > > > > > Honestly, the upgrade would take far too much work for my client to be > > > > willing to spend the time - especially since we have a work around and a > > > > possible fix already. Dotcms would have to do the work to change that, but > > > > they are still using Struts 1 and JUST upgraded to Spring 3 from 2. Not to > > > > say that those things don't make sense for them, but that it takes forever > > > > for them to do anything large - the codebase just isn't set up for rapid > > > > development. Also, there is a very large plugin community that could be > > > > affected - further lengthening the time for a fix - if in fact systemic > > > > changes are required. We were hoping to be able to have an instance set up > > > > in December sometime. > > > > > > > > So to sum up, I agree that their properties framework needs work. I > > > > also agree that it doesn't make sense to try and put a fix in place that is > > > > out of place. However, having said all that - I also think that there are > > > > plenty of environments that coders have to deal with that limit integration > > > > options - out of lack of foresight or other issues. Giving an ES > > > > TransportClient the option to not load ANY configurations outside of the > > > > ones passed in the Settings object is, I think, reasonable. There are > > > > just too many legacy and/or crappy applications to go around fixing all of > > > > them to be what you or I would expect or want. Our ES plugin predates the > > > > use of ES internally and as a consultant, it is hard to justify that sort > > > > of additional expenditure. Please let me know if you have any questions or > > > > additional concerns. > > > > > > > > Thanks for your time! > > > > Nick > > > > > > > > Thanks! > > > > > > > > Nicholas Padilla > > > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > > > > > “Problems cannot be solved by the same level of thinking that created > > > > them.” > > > > “Learn from yesterday, live for today, hope for tomorrow. The > > > > important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > > > ” > > > > > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > > > > > On Mon, Nov 26, 2012 at 10:42 AM, Igor Motov wrote: > > > > > > > > > Hi Nicholas, > > > > > > > > > > I agree that it sounds more like a workaround. What I am still > > > > > struggling with is figuring out if this is a workaround for an > > > > > elasticsearch issue or a dotcms issue. It's easy to add this feature to > > > > > elasticsearch. But before we do that, we should ask ourselves a question: > > > > > Are we solving the right problem? Elasticsearch already provides several > > > > > different ways of supplying configuration. The dotcms application is using > > > > > the most global way that overrides settings for all es instances running in > > > > > the dotcms process and that affects your use case. The problem can be fixed > > > > > on the dotcms side by changing the default configuration files that dotcms > > > > > is shipping with, or even better by switching to a config file base > > > > > configuration instead of system properties base configuration. So, would it > > > > > make more sense to fix it in dotcms rather than add a new feature to > > > > > elasticsearch? I am not very familiar with dotcms, so I might be missing > > > > > something important here. What do you think? > > > > > > > > > > Igor > > > > > > > > > > On Monday, November 26, 2012 11:02:39 AM UTC-5, Nicholas Padilla wrote: > > > > > > > > > > > Hey Igor, > > > > > > > > > > > > I have found that this does work, however the .properties needs to be > > > > > > a .yml and in the YAML format. This is a workaround and not a final > > > > > > solution, as it makes upgrading more difficult due to the changing of the > > > > > > core .properties file. As a dotcms best practice, everything should be > > > > > > done to customize the application in a config plugin. This is so the > > > > > > dotcms team can make updates to these files as needed, bug fixes or new > > > > > > features. This could be a solution if the parent architecture were a bit > > > > > > different. I have to pass this along with some notes as to how to upgrade, > > > > > > when you start throwing more cut and paste stuff in there the more room for > > > > > > error there is. > > > > > > > > > > > > Thanks for all your help on this matter! We at least have a stop gap > > > > > > measure in case the code adjustment I added takes a long time to get > > > > > > committed. > > > > > > > > > > > > Thanks again! > > > > > > > > > > > > On Friday, October 26, 2012 1:10:35 PM UTC-6, Igor Motov wrote: > > > > > > > > > > > > > But where are they adding settings to the System Properties from? > > > > > > > Isn't it dotCMS/WEB-INF/classes/dotm\*\*\*\*arketing-config.properties? > > > > > > > So, if you will move properties from this file to an external file, as I > > > > > > > described above, the only property that they will add to System Properties > > > > > > > will be es.config= which points to the location of the config file. As a > > > > > > > result, the embedded server will load all properties that it needs from the > > > > > > > config file, while your client will not simply ignore this config file and > > > > > > > will use only properties that you specified. > > > > > > > > > > > > > > On Fri, Oct 26, 2012 at 3:04 PM, Nicholas Padilla wrote: > > > > > > > > > > > > > > > Hey Igor! > > > > > > > > > > > > > > > > Thanks for the reply! I had thought about that but after looking > > > > > > > > at how dotcms loads it properties I found I couldn't move this direction. > > > > > > > > They are adding all properties to the System Properties. Creating a new > > > > > > > > TransportClient using the constructor you referenced only restricts from > > > > > > > > loading settings from the Classpath. They are adding all settings to the > > > > > > > > System Properties and that the TransportClient, by default, loads and > > > > > > > > overrides my Settings object from the System Properties. This is where > > > > > > > > the problem lies, there isn't a way to tell the TransportClient to NOT load > > > > > > > > properties from the System Properties at this point. That is what my patch > > > > > > > > does, allows you to say to ONLY use settings applied from the Settings > > > > > > > > object. > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > Nicholas Padilla > > > > > > > > [www.monstersoftwarellc.com](http://www.monstersoftwarellc.com) > > > > > > > > > > > > > > > > “Problems cannot be solved by the same level of thinking that > > > > > > > > created them.” > > > > > > > > “Learn from yesterday, live for today, hope for tomorrow. The > > > > > > > > important thing is not to stop questioning.[http://thinkexist.com/quotation/learn\_from\_yesterday-live\_for\_today-hope\_for/222120.html](http://thinkexist.com/quotation/learn_from_yesterday-live_for_today-hope_for/222120.html) > > > > > > > > ” > > > > > > > > > > > > > > > > Albert Einstein [http://thinkexist.com/quotes/albert\_einstein/](http://thinkexist.com/quotes/albert_einstein/) > > > > > > > > > > > > > > > > On Fri, Oct 26, 2012 at 12:57 PM, Igor Motov wrote: > > > > > > > > > > > > > > > > > Nicholas, > > > > > > > > > > > > > > > > > > Here is what you could try: > > > > > > > > > > > > > > > > > > Move all elasticsearch properties from dotCMS/WEB-INF/classes/\*\* > > > > > > > > > dotmark\*\*eting-config.properties to another file, for example, > > > > > > > > > myes.properties > > > > > > > > > > > > > > > > > > Replace all elasticsearch properties in dotCMS/WEB-INF/classes/\*\* > > > > > > > > > dotm\*\*arketing-config.properties with the following single line > > > > > > > > > that points to the myes.properties file. > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > es.config=path/to/myes. **propert** ies > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > Start your client with loadConfigSettings set to false: > > > > > > > > > > > > > > > > > > This way embedded elasticsearch in dotcms should get its settings > > > > > > > > > from myes.properties, but the client will ignore them if you will set > > > > > > > > > loadConfigSettings to false: > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > ... = new TransportClient( **settingsBuilde** r, false); > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > I am not sure how dotcms instantiates embedded **e** lasticsearch > > > > > > > > > node, but if they didn't change defaults, it might work. > > > > > > > > > > > > > > > > > > On Tuesday, October 9, 2012 5:32:02 PM UTC-4, Nicholas Padilla > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > I have submitted a pull request here: [https://github.com/\*\*elast](https://github.com/**elast)\* > > > > > > > > > > \*\*\*icsearch/elasticsearch/\*\*pull/ **23** 12[https://github.com/elasticsearch/elasticsearch/pull/2312](https://github.com/elasticsearch/elasticsearch/pull/2312) > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > On Monday, October 8, 2012 8:49:07 AM UTC-6, Drew Raines wrote: > > > > > > > > > > > > > > > > > > > > > Nicholas Padilla wrote: > > > > > > > > > > > > > > > > > > > > > > > I finally started to look into the codebase to see if there > > > > > > > > > > > > was > > > > > > > > > > > > anything stood out. I found that the Settings provided to the > > > > > > > > > > > > TransportClient constructor can be overridden by System > > > > > > > > > > > > Properties. > > > > > > > > > > > > With some testing I found that this was indeed the problem. I > > > > > > > > > > > > have > > > > > > > > > > > > been working with 0.19 branch and have some small changes that > > > > > > > > > > > > will > > > > > > > > > > > > provide this functionality without breaking the contract with > > > > > > > > > > > > other > > > > > > > > > > > > users. Should I commit this to both the 0.19 and master or > > > > > > > > > > > > just > > > > > > > > > > > > master? > > > > > > > > > > > > > > > > > > > > > > Commit it to master and create a pull request. Until we see a > > > > > > > > > > > patch > > > > > > > > > > > we can't discuss what you're proposing to change. > > > > > > > > > > > > > > > > > > > > > > -Drew > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > -- > > > > > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:01am UTC](https://discuss.elastic.co/t/connect-to-external-instance-inside-app-with-embedded-node/9171/20 "2017-07-06T03:01:14Z")