I have verified the index repository by querying, so said unable to index
even though there are no error
On Monday, April 28, 2014 12:35:12 PM UTC+5:30, Costin Leau wrote:
If ES is not running, getting an exception is expected since one will get
a connectivity error.
As for logging, make sure you properly configure log4j for your Hadoop
environment - it depends on what version you are
using and what libraries.
If the job is complete, you can always tests the results by querying ES
for the data that was just indexed.On 4/28/14 9:20 AM, Chetana wrote:
I am using now ES 1.1.1 and as before es-hadoop 1.3.0.M3. But even with
latest ES I am unable to index.
I don't see any log pertaining to es-hadoop in any of the log file and
also not on console. Also there is no exception
while running the job and the job completes successfully
But if elasticsearch server is not running, the job throws an exception
and does not run
/hadoop-yarn/staging/user1/.staging/job_1398663730568_0001
14/04/28 11:18:27 ERROR security.UserGroupInformation:
PriviledgedActionException as:user1 (auth:SIMPLE)
cause:java.io.IOException: Out of nodes and retries; caught exception
Exception in thread "main" java.io.IOException: Out of nodes and
retries; caught exception
at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:81)
at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:221)
at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:205)
at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:209)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:103)
Please someone suggest me how to troubleshoot this issue. After this
excise I need to index user log content to ES
Thanks,On Thursday, April 24, 2014 3:38:38 PM UTC+5:30, Costin Leau wrote:
Try es-hadoop 1.3.0.M3 - if you enable logging as indicated in thedocs I've mentioned you should be seeing all the
activity - namely the connection being made, the data beingtransmitted, the reply, etc....
On 4/24/14 12:21 PM, Chetana wrote: > Yes, I am able to search but not able to index. I am using1.3.0.M2. The json file size is just 28bytes.
> I corrected port/host and some log setting. With all thesechnages I am to run the application, but the indexing is
> still not happening > I dont see any error messages in log files. > I am using a single node cluster and running both elastic searchand hadoop on the same system.
> > On Thursday, April 24, 2014 10:30:22 AM UTC+5:30, Costin Leauwrote:
> > Hi, > > If I understand correctly, you can read data from ES (throughes-hadoop) but you cannot write to it - am I correct? Can
> you confirm that you are using the latest es-hadoop, namely1.3.0.M3?
> How big is the JSON file you are trying to index? Do you seeany activity in the console?
> > There are various ways in which you can monitor activity - inES you can monitor the console or use Marvel [1], in
> ES-hadoop you can enable logging [3] and see how the jobprogresses. Try starting with a small file to have a short
> feedback loop and once things get ironed out, try your actualdesired file.
> > > A few notes: > - In general we recommend using the latest stable version ofElasticsearch - 0.90.2 is quite old and unless you have a
> strong reason to stay on it, I highly recommend upgrading to1.1.1 or, in the worst case scenario, ES 0.90.13.
> - you have repeating code - you set "es.nodes" and "es.port"twice
> - you are copying the local file to the destination filesystem(presumably HDFS) which is okay but typically this is
> done outside the job launch > - if you are using Hadoop MRv2, consider switching to MRv1.es-hadoop supports both modes but the latter is easier to
> use and still the one the vendors recommend by default > - since you are using Hadoop 2 from Hortonworks, you mightwant to upgrade to their latest HDP (2.1) release.
> > Hope this helps, > > [1]http://www.elasticsearch.org/overview/marvel/ <Elasticsearch Platform — Find real-time answers at scale | Elastic>
<http://www.elasticsearch.org/overview/marvel/ <Elasticsearch Platform — Find real-time answers at scale | Elastic>>
> [2]Elasticsearch Platform — Find real-time answers at scale | Elastic
<Elasticsearch Platform — Find real-time answers at scale | Elastic>
> <Elasticsearch Platform — Find real-time answers at scale | Elastic
<Elasticsearch Platform — Find real-time answers at scale | Elastic>>
> > On 4/24/14 7:30 AM, Chetana wrote: > > I am using elasticsearch-hadoop plugin (GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop <
GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop><https://github.com/elasticsearch/elasticsearch-hadoop <> trying to index some > > documents. Iam using elasticsearch version 0.90.2 and HadoopHortonworks 2.2.0. The search functionality works fine, but
> > while indexing application hangs > >https://github.com/elasticsearch/elasticsearch-hadoop <<https://github.com/elasticsearch/elasticsearch-hadoop <> > The json file location is passed as a command line arguemntand below is the indexing code snippet
> > Configuration conf = new Configuration(); > > conf.setBoolean("mapred.map.tasks.speculative.execution",false);
> >conf.setBoolean("mapred.reduce.tasks.speculative.execution", false);
> > conf.setInt("mapred.min.split.size",40); > > conf.set("es.resource", "test/test"); > > conf.set("es.nodes", "localhost"); > > conf.set("es.port", "9200"); > > conf.set("es.input.json", "yes"); > > conf.set("es.nodes", "localhost"); > > conf.set("es.port", "9200"); > > > > Job job = Job.getInstance(conf); > > job.setMapperClass(Mapper.class); > > job.setInputFormatClass(TextInputFormat.class); > > job.setOutputFormatClass(EsOutputFormat.class); > > job.setMapOutputKeyClass(LongWritable.class); > > job.setMapOutputValueClass(Text.class); > > Path jarPath = new Path(args[0]); > > FileSystem fs = FileSystem.get(conf); > > Path dst = new Path(fs.getHomeDirectory(),jarPath.getName());
> > fs.copyFromLocalFile(false, true, jarPath, dst); > > FileInputFormat.setInputPaths(job, dst); > > > > job.waitForCompletion(true); > > Am I missing anything, pls help > > > > -- > > You received this message because you are subscribed to theGoogle Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emailsfrom it, send an email to
> >elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:> <javascript:>
<javascript:>>. > > To view this discussion on the web visit > ><> <<> > <<> <<> > > For more options, visithttps://groups.google.com/d/optout <http://groups.google.com/d/optout> <https://groups.google.com/d/optout
<https://groups.google.com/d/optout>>. > > -- > Costin > > -- > You received this message because you are subscribed to the GoogleGroups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it,send an email to
>elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:> <javascript:>>.
> To view this discussion on the web visit ><> <<> For more options, visithttps://groups.google.com/d/optout <https://groups.google.com/d/optout>.
-- Costin--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearc...@googlegroups.com <javascript:> <mailto:
elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
To view this discussion on the web visitFor more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/11f042f0-2c5b-46d0-973e-3720662ed7bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.