Optimized Rest requests in case of embedded elastic search

Elastic Stack Elasticsearch
1

Hi guys,

I intend to use the elasticsearch embedded in my webapplication.
Is there a kind of optimization for rest calls (for indexing a document for
example) to not generate http traffic, when the elasticsearch is running in
same virtual machine as the service caller? (Instead of generate an http
request, just use java objects)
Is there a way to index a document without using the rest api, in case the
elasticsearch is embedded in the same webapplication?
I want to index the log messages from my application as soon they are
produced, but we have a lot of messages, should I be concerned of the http
loading on the server?

Thank you very much,
Doru

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/81f4070a-7528-43a0-8c7e-af23c7699c41%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi Doru,
Based on what I've seen (but of course may be criticized by others)

I don't know if there is a sufficient reason to use the REST API and
use something else instead.
What is your concern? Network traffic? AFAIK if localhost is bound to lo or
something similar, that isn't an issue.

As for your question about indexing your data before sending into ES, IMO
that is what the Logstash examples illustrate.

The little 5-node ES cluster lab cluster I setup running ES, Redis,
logstash and netcat as described in the Logstash 10-minute walkthrough was
interesting re your question about separating load on to different apps
running in different consoles. I could observe the apache data as it
progressed through each app (because logstash was configured with a stdout
output). I could see if there was a pause here and there, possibly
indicating unused capacity.

I recommend you look at that tutorial and see if it answers your questions
as well.

IMO,
Tony

On Monday, February 10, 2014 7:58:59 AM UTC-8, Doru Sular wrote:

Hi guys,

I intend to use the elasticsearch embedded in my webapplication.
Is there a kind of optimization for rest calls (for indexing a document
for example) to not generate http traffic, when the elasticsearch is
running in same virtual machine as the service caller? (Instead of generate
an http request, just use java objects)
Is there a way to index a document without using the rest api, in case the
elasticsearch is embedded in the same webapplication?
I want to index the log messages from my application as soon they are
produced, but we have a lot of messages, should I be concerned of the http
loading on the server?

Thank you very much,
Doru

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4c976cf9-1526-4e24-ac5f-187bb7e7e0d1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi Tony,
yes, network traffic might be a problem. I hope I can use easily the bulk
option, otherweise it will be a "no go" for using elasticsearch.
Still I have a question. What means "localhost is bound to lo or something
similar" ?
Thank you very much,
Doru

On Monday, February 10, 2014 10:52:14 PM UTC+1, Tony Su wrote:

Hi Doru,
Based on what I've seen (but of course may be criticized by others)

I don't know if there is a sufficient reason to use the REST API and
use something else instead.
What is your concern? Network traffic? AFAIK if localhost is bound to lo
or something similar, that isn't an issue.

As for your question about indexing your data before sending into ES, IMO
that is what the Logstash examples illustrate.

The little 5-node ES cluster lab cluster I setup running ES, Redis,
logstash and netcat as described in the Logstash 10-minute walkthrough was
interesting re your question about separating load on to different apps
running in different consoles. I could observe the apache data as it
progressed through each app (because logstash was configured with a stdout
output). I could see if there was a pause here and there, possibly
indicating unused capacity.

I recommend you look at that tutorial and see if it answers your questions
as well.

IMO,
Tony

On Monday, February 10, 2014 7:58:59 AM UTC-8, Doru Sular wrote:

Hi guys,

I intend to use the elasticsearch embedded in my webapplication.
Is there a kind of optimization for rest calls (for indexing a document
for example) to not generate http traffic, when the elasticsearch is
running in same virtual machine as the service caller? (Instead of generate
an http request, just use java objects)
Is there a way to index a document without using the rest api, in case
the elasticsearch is embedded in the same webapplication?
I want to index the log messages from my application as soon they are
produced, but we have a lot of messages, should I be concerned of the http
loading on the server?

Thank you very much,
Doru

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/709363dc-72e3-4ec2-ba9e-80349af3790a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Look at the Java APIhttp://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/.
From my look through elasticsearch code, it does not generate HTTP requests
but routes them through services.

How many logs do you generate on average? Do your servers have enough
resources to handle indexing and searching at the same time? I think you
should worry about these questions more than HTTP loading on your server.

Additionally logging messages from a web application to elasticsearch in
the same webapp requires some thought into implementation; you can wind up
in an endless loop if a class that saves your logs to elasticsearch also
logs a message. Another item to think about is how to index the logs and
how long to retain them.

On Monday, February 10, 2014 10:58:59 AM UTC-5, Doru Sular wrote:

Hi guys,

I intend to use the elasticsearch embedded in my webapplication.
Is there a kind of optimization for rest calls (for indexing a document
for example) to not generate http traffic, when the elasticsearch is
running in same virtual machine as the service caller? (Instead of generate
an http request, just use java objects)
Is there a way to index a document without using the rest api, in case the
elasticsearch is embedded in the same webapplication?
I want to index the log messages from my application as soon they are
produced, but we have a lot of messages, should I be concerned of the http
loading on the server?

Thank you very much,
Doru

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c0f60c04-a939-4ea8-aade-e3781a54cafb%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

I wrap up my Elasticsearch apps in a Wildfly web app with JAX-RS API, using
the native java transport protocol.

You do not need to use HTTP with Java app logs. Just write a log4j appender
or something like this and connect this to a bulk indexer.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHbCLfwrZ9dwXnV3%3D%2B_sZj%3DeNgEA2XkF3751iddAB5_wg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Hi Jörg,
Can you please provide the name of the elasticsearch classes needed to
implement a bulk indexer?
Thank you very much,
Doru

On Wednesday, February 12, 2014 10:10:59 AM UTC+1, Jörg Prante wrote:

I wrap up my Elasticsearch apps in a Wildfly web app with JAX-RS API,
using the native java transport protocol.

You do not need to use HTTP with Java app logs. Just write a log4j
appender or something like this and connect this to a bulk indexer.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/77cfe567-a3cb-4058-b918-31fdf4bb8611%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.