Azure cloud repository error in elasticsearch 2.4.1?

HI,

I installed azure cloud plugin in all the nodes of ES cluster with the following configuration in elasticsearch.yml

cloud:
    azure:
        storage:
            my_account:
                account: your_azure_storage_account
                key: your_azure_storage_key

but when i am executing the following command
curl -XPUT 'http://localhost:9200/_snapshot/azure_backup' -d '{ "type":"azure"}'

{"error":{"root_cause":[{"type":"repository_exception","reason":"[azure_backup] failed to create repository"}],"type":"repository_exception","reason":"[azure_backup] failed to create repository","caused_by":{"type":"illegal_argument_exception","reason":"Unknown [repository] type [azure]"}},"status":500}

Thanks

May be you didn't restart?

Thanks..

I restarted every node in my ES cluster.

Still the problem remains.. As soon as i am executing it is directly throwing error

Thanks

Can you run GET _cat/plugins?v

data-4 cloud-azure  2.4.1   j        
data-4 graph        2.4.1   j        
data-4 license      2.4.1   j        
data-4 marvel-agent 2.4.1   j        
data-4 shield       2.4.1   j        
data-4 watcher      2.4.1   j        
data-2 cloud-azure  2.4.1   j        
data-2 graph        2.4.1   j        
data-2 license      2.4.1   j        
data-2 marvel-agent 2.4.1   j        
data-2 shield       2.4.1   j        
data-2 watcher      2.4.1   j        
data-0 cloud-azure  2.4.1   j        
data-0 graph        2.4.1   j        
data-0 license      2.4.1   j        
data-0 marvel-agent 2.4.1   j        
data-0 shield       2.4.1   j        
data-0 watcher      2.4.1   j        
data-3 cloud-azure  2.4.1   j        
data-3 graph        2.4.1   j        
data-3 license      2.4.1   j        
data-3 marvel-agent 2.4.1   j        
data-3 shield       2.4.1   j        
data-3 watcher      2.4.1   j        
data-1 cloud-azure  2.4.1   j        
data-1 graph        2.4.1   j        
data-1 license      2.4.1   j        
data-1 marvel-agent 2.4.1   j        
data-1 shield       2.4.1   j        
data-1 watcher      2.4.1   j

we are using ES 2.4.1

Is there a stacktrace somewhere?

Do you want to see ES logs?

[2017-03-28 12:29:06,070][WARN ][rest.suppressed          ] path: /_snapshot/azure_backup, params: {repository=azure_backup}
RemoteTransportException[[data-1][10.0.0.9:9300][cluster:admin/repository/put]]; nested: RepositoryException[[azure_backup] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [azure]];
Caused by: RepositoryException[[azure_backup] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [azure]];
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
        at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:368)
        at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55)
        at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Unknown [repository] type [azure]
        at org.elasticsearch.common.util.ExtensionPoint$SelectedType.bindType(ExtensionPoint.java:146)
        at org.elasticsearch.repositories.RepositoryTypesRegistry.bindType(RepositoryTypesRegistry.java:49)
        at org.elasticsearch.repositories.RepositoryModule.configure(RepositoryModule.java:58)
        at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
        at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:233)
        at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:105)
        at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:143)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
        ... 11 more

May I have the full logs (from restart) of machine 10.0.0.9?

May be you need to upgrade?

From where i have to upgrade the fix logger ?

Do i need to change the source code by myself?

Sry,i am new to this environment. Pls correct me if i am wrong?

Thanks

If you want to upgrade, you have to upgrade both elasticsearch and the plugin.

See rolling upgrade page in our docs.

But first can you share the logs I asked?

[2017-03-28 13:28:06,142][WARN ][repositories             ] [esndata-1] failed to create repository [azure][azure_backup]
java.lang.IllegalArgumentException: Unknown [repository] type [azure]
        at org.elasticsearch.common.util.ExtensionPoint$SelectedType.bindType(ExtensionPoint.java:146)
        at org.elasticsearch.repositories.RepositoryTypesRegistry.bindType(RepositoryTypesRegistry.java:49)
        at org.elasticsearch.repositories.RepositoryModule.configure(RepositoryModule.java:58)
        at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
        at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:233)
        at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:105)
        at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:143)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
        at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2017-03-28 13:28:06,146][WARN ][repositories             ] [esndata-1] failed to create repository [azure_backup]
RepositoryException[[azure_backup] failed to create repository]; nested: IllegalArgumentException[Unknown [repository] type [azure]];
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
        at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Unknown [repository] type [azure]
        at org.elasticsearch.common.util.ExtensionPoint$SelectedType.bindType(ExtensionPoint.java:146)
        at org.elasticsearch.repositories.RepositoryTypesRegistry.bindType(RepositoryTypesRegistry.java:49)
        at org.elasticsearch.repositories.RepositoryModule.configure(RepositoryModule.java:58)
        at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
        at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:233)
        at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:105)
        at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:143)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
        at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
        ... 8 more
[2017-03-28 13:28:06,713][WARN ][repositories             ] [esndata-1] [azure_backup] failed to verify repository
RepositoryMissingException[[azure_backup] missing]
        at org.elasticsearch.repositories.RepositoriesService.indexShardRepository(RepositoriesService.java:347)
        at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:120)
        at org.elasticsearch.repositories.VerifyNodeRepositoryAction.access$200(VerifyNodeRepositoryAction.java:50)
        at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:156)
        at org.elasticsearch.repositories.VerifyNodeRepositoryAction$VerifyNodeRepositoryRequestHandler.messageReceived(VerifyNodeRepositoryAction.java:152)
        at

                continuation
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
            at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
            at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
            at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
            at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
            at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
            at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
            at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
            at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
            at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
            at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
            at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
            at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
            at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
            at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
            at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
            at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
            at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
            at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
            at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)

sorry, i am cutting it and sending because the space limitations

Thanks

Now, the error is changed to something like this:

{
   "error": {
      "root_cause": [
         {
            "type": "repository_verification_exception",
            "reason": "[azure_backup] [6SkEVsP_R-a4B834Y0kkYQ, 'RemoteTransportException[[data-1][10.0.0.9:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [RS4adIFkRwWb0rufqqUeOQ, 'RemoteTransportException[[data-0][10.0.0.6:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [zp07QTqiQJGrjCW-3ES8yg, 'RemoteTransportException[[data-3][10.0.0.8:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [vzocANnEQY2V9q2H6LFCog, 'RemoteTransportException[[data-2][10.0.0.7:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];']]"
         }
      ],
      "type": "repository_verification_exception",
      "reason": "[azure_backup] [6SkEVsP_R-a4B834Y0kkYQ, 'RemoteTransportException[[data-1][10.0.0.9:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [RS4adIFkRwWb0rufqqUeOQ, 'RemoteTransportException[[data-0][10.0.0.6:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [zp07QTqiQJGrjCW-3ES8yg, 'RemoteTransportException[[data-3][10.0.0.8:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];'], [vzocANnEQY2V9q2H6LFCog, 'RemoteTransportException[[data-2][10.0.0.7:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[azure_backup] missing];']]"
   },
   "status": 500
}

I asked for:

May I have the full logs (from restart) of machine 10.0.0.9?

Can you provide this? You can share that on gist.github.com.

yeah i had everything in full logs in gist .

if i upgraded the azure cloud plugin 2.4.1 did it work with ES 2.4.1?

Thanks

Hmmm. Interesting.

Can you run another test for me please?

On another machine:

  • install elasticsearch 2.4.1
  • install cloud-azure plugin
  • start
  • create a repository

If it fails:

  • install elasticsearch 2.4.2 and do the same steps

If 2.4.1 did not fail it might indicate an "issue" with Shield. In that case, can you install license plugin and shield and restart and try again?

Thanks!

you mean in another server like 10.0.0.x or want to do entirely in separate machine ?

May i know why i am getting the error?Whether I have to install licensed azure plugin for it?

Actually my production environment is in live i cant able to upgrade it as such . So, i have to solve in ES 2.4.1 itself.

Is there any way to do it in ES 2.4.1 itself?

THANKS

you mean in another server like 10.0.0.x or want to do entirely in separate machine ?

As you wish.

May i know why i am getting the error?

I don't know but I suspect that you are having another issue which is hidden by the bug I linked to. This bug has been fixed in 2.4.2. But that's only a guess for now.

Is there any way to do it in ES 2.4.1 itself?

If we find what is the actual root cause, you will be able hopefully to fix it in 2.4.1 without needing to upgrade although I recommend to upgrade and stick with latest patchs which is 2.4.4 ATM for the 2.x branch.

Thanks for your suggestion..

Ok. Then i will upgrade by ES from 2.4.1 to 2.4.2 and the same to azure cloud plugin . After I will update you on that..

Thanks