Azure cloud repository error in elasticsearch 2.4.1?

Yaswanth · March 28, 2017, 11:33am

HI,

I installed azure cloud plugin in all the nodes of ES cluster with the following configuration in elasticsearch.yml

cloud:
    azure:
        storage:
            my_account:
                account: your_azure_storage_account
                key: your_azure_storage_key

but when i am executing the following command
curl -XPUT 'http://localhost:9200/_snapshot/azure_backup' -d '{ "type":"azure"}'

{"error":{"root_cause":[{"type":"repository_exception","reason":"[azure_backup] failed to create repository"}],"type":"repository_exception","reason":"[azure_backup] failed to create repository","caused_by":{"type":"illegal_argument_exception","reason":"Unknown [repository] type [azure]"}},"status":500}

Thanks

dadoonet · March 28, 2017, 11:44am

May be you didn't restart?

Yaswanth · March 28, 2017, 11:49am

Thanks..

I restarted every node in my ES cluster.

Still the problem remains.. As soon as i am executing it is directly throwing error

Thanks

dadoonet · March 28, 2017, 12:00pm

Can you run GET _cat/plugins?v

Yaswanth · March 28, 2017, 12:04pm

data-4 cloud-azure  2.4.1   j        
data-4 graph        2.4.1   j        
data-4 license      2.4.1   j        
data-4 marvel-agent 2.4.1   j        
data-4 shield       2.4.1   j        
data-4 watcher      2.4.1   j        
data-2 cloud-azure  2.4.1   j        
data-2 graph        2.4.1   j        
data-2 license      2.4.1   j        
data-2 marvel-agent 2.4.1   j        
data-2 shield       2.4.1   j        
data-2 watcher      2.4.1   j        
data-0 cloud-azure  2.4.1   j        
data-0 graph        2.4.1   j        
data-0 license      2.4.1   j        
data-0 marvel-agent 2.4.1   j        
data-0 shield       2.4.1   j        
data-0 watcher      2.4.1   j        
data-3 cloud-azure  2.4.1   j        
data-3 graph        2.4.1   j        
data-3 license      2.4.1   j        
data-3 marvel-agent 2.4.1   j        
data-3 shield       2.4.1   j        
data-3 watcher      2.4.1   j        
data-1 cloud-azure  2.4.1   j        
data-1 graph        2.4.1   j        
data-1 license      2.4.1   j        
data-1 marvel-agent 2.4.1   j        
data-1 shield       2.4.1   j        
data-1 watcher      2.4.1   j

we are using ES 2.4.1

dadoonet · March 28, 2017, 12:19pm

Is there a stacktrace somewhere?

Yaswanth · March 28, 2017, 12:32pm

Do you want to see ES logs?

[2017-03-28 12:29:06,070][WARN ][rest.suppressed          ] path: /_snapshot/azure_backup, params: {repository=azure_backup}
RemoteTransportException[[data-1][10.0.0.9:9300][cluster:admin/repository/put]]; nested: RepositoryException[[azure_backup] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [azure]];
Caused by: RepositoryException[[azure_backup] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [azure]];
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
        at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:368)
        at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55)
        at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Unknown [repository] type [azure]
        at org.elasticsearch.common.util.ExtensionPoint$SelectedType.bindType(ExtensionPoint.java:146)
        at org.elasticsearch.repositories.RepositoryTypesRegistry.bindType(RepositoryTypesRegistry.java:49)
        at org.elasticsearch.repositories.RepositoryModule.configure(RepositoryModule.java:58)
        at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
        at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:233)
        at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:105)
        at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:143)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
        ... 11 more

dadoonet · March 28, 2017, 12:46pm

May I have the full logs (from restart) of machine 10.0.0.9?

dadoonet · March 28, 2017, 12:49pm

May be you need to upgrade?

Yaswanth · March 28, 2017, 1:11pm

From where i have to upgrade the fix logger ?

Do i need to change the source code by myself?

Sry,i am new to this environment. Pls correct me if i am wrong?

Thanks

dadoonet · March 28, 2017, 1:14pm

If you want to upgrade, you have to upgrade both elasticsearch and the plugin.

See rolling upgrade page in our docs.

But first can you share the logs I asked?

Yaswanth · March 28, 2017, 1:36pm

[2017-03-28 13:28:06,142][WARN ][repositories             ] [esndata-1] failed to create repository [azure][azure_backup]
java.lang.IllegalArgumentException: Unknown [repository] type [azure]
        at org.elasticsearch.common.util.ExtensionPoint$SelectedType.bindType(ExtensionPoint.java:146)
        at org.elasticsearch.repositories.RepositoryTypesRegistry.bindType(RepositoryTypesRegistry.java:49)
        at org.elasticsearch.repositories.RepositoryModule.configure(RepositoryModule.java:58)
        at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
        at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:233)
        at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:105)
        at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:143)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
        at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2017-03-28 13:28:06,146][WARN ][repositories             ] [esndata-1] failed to create repository [azure_backup]
RepositoryException[[azure_backup] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [azure]];
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
        at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Unknown [repository] type [azure]
        at org.elasticsearch.common.util.ExtensionPoint$SelectedType.bindType(ExtensionPoint.java:146)
        at org.elasticsearch.repositories.RepositoryTypesRegistry.bindType(RepositoryTypesRegistry.java:49)
        at org.elasticsearch.repositories.RepositoryModule.configure(RepositoryModule.java:58)
        at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
        at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:233)
        at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:105)
        at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:143)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
        ... 8 more
[2017-03-28 13:28:06,713][WARN ][repositories             ] [esndata-1] [azure_backup] failed to verify repository
RepositoryMissingException[[azure_backup] missing]
        at org.elasticsearch.repositories.RepositoriesService.indexShardRepository(RepositoriesService.java:347)
        at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:120)
        at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$200(VerifyNodeRepositoryAction.java:50)
        at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:156)
        at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:152)
        at

                continuation

Yaswanth · March 28, 2017, 1:36pm

org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
            at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
            at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
            at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
            at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
            at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
            at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
            at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
            at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
            at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
            at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
            at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
            at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
            at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
            at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
            at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
            at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
            at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)

sorry, i am cutting it and sending because the space limitations

Thanks

Yaswanth · March 28, 2017, 1:44pm

Now, the error is changed to something like this:

{
   "error": {
      "root_cause": [
         {
            "type": "repository_verification_exception",
            "reason": "[azure_backup] [6SkEVsP_R-a4B834Y0kkYQ, 'RemoteTransportException[[data-1][10.0.0.9:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [RS4adIFkRwWb0rufqqUeOQ, 'RemoteTransportException[[data-0][10.0.0.6:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [zp07QTqiQJGrjCW-3ES8yg, 'RemoteTransportException[[data-3][10.0.0.8:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [vzocANnEQY2V9q2H6LFCog, 'RemoteTransportException[[data-2][10.0.0.7:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];']]"
         }
      ],
      "type": "repository_verification_exception",
      "reason": "[azure_backup] [6SkEVsP_R-a4B834Y0kkYQ, 'RemoteTransportException[[data-1][10.0.0.9:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [RS4adIFkRwWb0rufqqUeOQ, 'RemoteTransportException[[data-0][10.0.0.6:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [zp07QTqiQJGrjCW-3ES8yg, 'RemoteTransportException[[data-3][10.0.0.8:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [vzocANnEQY2V9q2H6LFCog, 'RemoteTransportException[[data-2][10.0.0.7:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];']]"
   },
   "status": 500
}

dadoonet · March 28, 2017, 1:49pm

I asked for:

May I have the full logs (from restart) of machine 10.0.0.9?

Can you provide this? You can share that on gist.github.com.

Yaswanth · March 28, 2017, 2:01pm

yeah i had everything in full logs in gist .

if i upgraded the azure cloud plugin 2.4.1 did it work with ES 2.4.1?

Thanks

dadoonet · March 28, 2017, 2:17pm

Hmmm. Interesting.

Can you run another test for me please?

On another machine:

install elasticsearch 2.4.1
install cloud-azure plugin
start
create a repository

If it fails:

install elasticsearch 2.4.2 and do the same steps

If 2.4.1 did not fail it might indicate an "issue" with Shield. In that case, can you install license plugin and shield and restart and try again?

Thanks!

Yaswanth · March 28, 2017, 3:51pm

you mean in another server like 10.0.0.x or want to do entirely in separate machine ?

May i know why i am getting the error?Whether I have to install licensed azure plugin for it?

Actually my production environment is in live i cant able to upgrade it as such . So, i have to solve in ES 2.4.1 itself.

Is there any way to do it in ES 2.4.1 itself?

THANKS

dadoonet · March 28, 2017, 4:55pm

you mean in another server like 10.0.0.x or want to do entirely in separate machine ?

As you wish.

May i know why i am getting the error?

I don't know but I suspect that you are having another issue which is hidden by the bug I linked to. This bug has been fixed in 2.4.2. But that's only a guess for now.

Is there any way to do it in ES 2.4.1 itself?

If we find what is the actual root cause, you will be able hopefully to fix it in 2.4.1 without needing to upgrade although I recommend to upgrade and stick with latest patchs which is 2.4.4 ATM for the 2.x branch.

Yaswanth · March 28, 2017, 5:12pm

Thanks for your suggestion..

Ok. Then i will upgrade by ES from 2.4.1 to 2.4.2 and the same to azure cloud plugin . After I will update you on that..

Thanks