Failed to set monitoring pipeline [xpack_monitoring_2] And process cluster event (put-pipeline-xpack_monitoring_2)

Elastic Stack Elasticsearch
1

Hi All,

Below is the Error im getting in Elasticsearch container logs

'failed to set monitoring pipeline [xpack_monitoring_2] Org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-pipeline-xpack_monitoring_2) within 30s'

How to Solve please help me

NOTE :- I want to switch my ES version from 5.0.0 to 7.6.2 Latest version before that i deploy ES v6.8.13 then i will move to 7.6.2 version then i got error which u can see above.

MY ELASTICSEARCH DATA IS VERY IMP and I DONT WANT TO LOSE IT .

Please help me

2

So backups are really important here.

One thing you can do to avoid multiple steps is to reindex your data from the old cluster into the new one.

Start a new cluster with 7.11 and use the reindex from remote API to read your old data and reindex in the new cluster.

Otherwise, I would propose recommend to:

  • run a snapshot
  • upgrade to the latest 5.x version
  • use the upgrade assistant plugin in kibana (I think it's freely available in xpack)
  • if everything is ok, upgrade to 6.8
  • use the upgrade assistant
  • if everything is ok, upgrade to 7.11

Note that you will have to reindex the data locally but the upgrade assistant should guide you. If you started your project prior to 5.x, it needs to rewrite the old indices to the 5.x format before switching to 6.x.
Then in 6.x, it needs to rewrite to a 6.x format before switching to 7.x.

It's a very long journey so I'd recommend testing all that process on a test server.

4

Im deploying ES, Kibana and others through docker stack NOT using logstack

Same version im using for ES and Kibana is 6.8.13

can u share me a website or tutorial so that i can easily deploy and upgrade my ES

Im new to ES and dont know how to use snapchat, upgrade assistand plugin, xpack and how to reindex from remote API ..dont know

5

Didn't you say that you want to upgrade from elastic 5.0.0?

I'm lost.

There's a upgrade part in the documentation. I suggest that you carefully read it.

6

ok
I'm not able to deploy kibana container and

Here is my docker service ls

nx97wiuk0gxy   test_fx-kibana    replicated   0/1      kibana:6.8.13
gc08wzl3j6n3   test_fx-elasticsearch     replicated   1/1      elasticsearch:6.8.13

And when i check service logs its returning empty

Here is the kibana.yaml file

    version: '3.5'
    services:
      fx-kibana:
        image: "kibana:6.8.13"
        deploy:
          replicas: 1
          placement:
            constraints: [node.role == manager]
          update_config:
            parallelism: 2
            delay: 10s
            order: stop-first
        environment:
          - SERVER_NAME=kibana
          - ELASTICSEARCH_URL=http://fx-elasticsearch:9200
          - SERVER_PORT=192.168.0.13:5601
        volumes:
          - ./kibana/config/:/usr/share/kibana/config:ro   
          - ./kibana.yml:/usr/share/kibana/config/kibana.yml   
        ports:
          - 5601:5601
        depends_on:
          - elasticsearch

AND
Here is the Elastisearch.yaml file

  fx-elasticsearch:
#    image: "elasticsearch:7.6.2"
    image: "elasticsearch:6.8.13"
#    image: "elasticsearch:5.0.0"
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      placement:
        constraints: [node.role == manager]
    environment:
      - ES_JAVA_OPTS=${ELASTICSEARCH_MEM}
      - discovery.type=single-node
      - cluster.name=elasticsearch
      - node.max_local_storage_nodes=2
      - xpack.monitoring.enabled=false
    ports:
      - 9200:9200
      - 9300:9300
    volumes:
      - esdata:/usr/share/elasticsearch/data

I DONT know where im wrong

7

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

8

Thanks You

i have make changes in my question and please reply me

9

What are elasticsearch logs?

10

Here is the ES container logs

[2021-03-01T14:32:33,256][INFO ][o.e.e.NodeEnvironment    ] [opDJrKj] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda5)]], net usable_space [541.5gb], net total_space [915.3gb], types [ext4]
[2021-03-01T14:32:33,265][INFO ][o.e.e.NodeEnvironment    ] [opDJrKj] heap size [2gb], compressed ordinary object pointers [true]
[2021-03-01T14:32:33,441][INFO ][o.e.n.Node               ] [opDJrKj] node name derived from node ID [opDJrKjdSkWakc2f4pS6MA]; set [node.name] to override
[2021-03-01T14:32:33,442][INFO ][o.e.n.Node               ] [opDJrKj] version[6.8.13], pid[1], build[default/docker/be13c69/2020-10-16T09:09:46.555371Z], OS[Linux/5.8.0-44-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15/15+36]
[2021-03-01T14:32:33,442][INFO ][o.e.n.Node               ] [opDJrKj] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2408253882814728710, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xms2g, -Xmx2g, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
[2021-03-01T14:32:35,546][INFO ][o.e.p.PluginsService     ] [opDJrKj] loaded module [aggs-matrix-stats]
[2021-03-01T14:32:35,546][INFO ][o.e.p.PluginsService     ] [opDJrKj] loaded module [analysis-common]
[2021-03-01T14:32:35,546][INFO ][o.e.p.PluginsService     ] [opDJrKj] loaded module [ingest-common]
[2021-03-01T14:32:35,547][INFO ][o.e.p.PluginsService     ] [opDJrKj] loaded module [ingest-geoip]
[2021-03-01T14:32:35,547][INFO ][o.e.p.PluginsService     ] [opDJrKj] loaded module [ingest-user-agent]
[2021-03-01T14:32:35,547][INFO ][o.e.p.PluginsService     ] [opDJrKj] loaded module [lang-expression]

[2021-03-01T14:32:35,550][INFO ][o.e.p.PluginsService     ] [opDJrKj] loaded module [x-pack-upgrade]
[2021-03-01T14:32:35,551][INFO ][o.e.p.PluginsService     ] [opDJrKj] loaded module [x-pack-watcher]
[2021-03-01T14:32:35,551][INFO ][o.e.p.PluginsService     ] [opDJrKj] no plugins loaded
[2021-03-01T14:32:39,672][INFO ][o.e.x.s.a.s.FileRolesStore] [opDJrKj] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2021-03-01T14:32:40,240][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [opDJrKj] [controller/97] [Main.cc@114] controller (64 bit): Version 6.8.13 (Build c959f131d0de3b) Copyright (c) 2020 Elasticsearch BV
[2021-03-01T14:32:40,901][INFO ][o.e.d.DiscoveryModule    ] [opDJrKj] using discovery type [single-node] and host providers [settings]
[2021-03-01T14:32:41,891][INFO ][o.e.n.Node               ] [opDJrKj] initialized
[2021-03-01T14:32:41,891][INFO ][o.e.n.Node               ] [opDJrKj] starting ...
[2021-03-01T14:32:42,032][INFO ][o.e.t.TransportService   ] [opDJrKj] publish_address {10.0.0.127:9300}, bound_addresses {0.0.0.0:9300}
[2021-03-01T14:32:42,166][INFO ][o.e.h.n.Netty4HttpServerTransport] [opDJrKj] publish_address {10.0.0.127:9200}, bound_addresses {0.0.0.0:9200}
[2021-03-01T14:32:42,166][INFO ][o.e.n.Node               ] [opDJrKj] started
[2021-03-01T14:32:45,139][WARN ][r.suppressed             ] [opDJrKj] path: /_stats, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:191) ~[elasticsearch-6.8.13.jar:6.8.13]
	at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:72) ~[elasticsearch-6.8.13.jar:6.8.13]
	at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:49) ~[elasticsearch-6.8.13.jar:6.8.13]
	at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.<init>(TransportBroadcastByNodeAction.java:258) ~[elasticsearch-6.8.13.jar:6.8.13]
	 
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-03-01T14:32:50,021][WARN ][r.suppressed             ] [opDJrKj] path: /_stats, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:191) ~[elasticsearch-6.8.13.jar:6.8.13]
	at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:72) ~[elasticsearch-6.8.13.jar:6.8.13]
	at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:49) ~[elasticsearch-6.8.13.jar:6.8.13]
	inal.jar:4.1.32.Final]
 org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.<init>(TransportBroadcastByNodeAction.java:258) ~[elasticsearch-6.8.13.jar:6.8.13]

	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:121) [x-pack-security-6.8.13.jar:6.8.13]
	
[2021-03-01T14:33:11,421][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [opDJrKj] Failed to clear cache for realms [[]]
[2021-03-01T14:33:11,460][INFO ][o.e.l.LicenseService     ] [opDJrKj] license [9be1d0a1-e528-464d-b77a-ca4627d7a9f7] mode [basic] - valid
[2021-03-01T14:33:11,463][INFO ][o.e.g.GatewayService     ] [opDJrKj] recovered [27] indices into cluster_state
[2021-03-01T14:34:56,026][INFO ][o.e.c.r.a.AllocationService] [opDJrKj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[fx-notifications][1], [fx-notifications][4]] ...]).

[2021-03-01T16:54:01,857][WARN ][o.e.m.j.JvmGcMonitorService] [opDJrKj] [gc][8397] overhead, spent [872ms] collecting in the last [1.1s]
[2021-03-01T16:56:52,050][INFO ][o.e.m.j.JvmGcMonitorService] [opDJrKj] [gc][8567] overhead, spent [472ms] collecting in the last [1.1s]
[2021-03-01T17:03:56,654][WARN ][o.e.m.j.JvmGcMonitorService] [opDJrKj] [gc][8990] overhead, spent [697ms] collecting in the last [1s]
11

Hi dadoonet,

Please reply me and i already send you logs

12

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the Community Code of Conduct specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you.

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

13

What are the Kibana logs now?

14

Thanks for reply

I solve the issue by adding comment to volumes in kibana.yaml file

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.