Version mismatch message even though versions match

McJava1967 · July 6, 2023, 3:53pm

Hi all. I'm trying out ELK 8.8.2, and getting this message:

Job creation error
The client noticed that the server is not Elasticsearch and we do not support this unknown product.

All explanations in various posts say this is mismatch between Kibana and Elasticsearch versions. But ours do match. (See below). Any other ideas how to fix this?

Installed Packages
Name : elasticsearch
Arch : x86_64
Version : 8.8.2
Release : 1
Size : 1.2 G
Repo : installed
From repo : elasticsearch
Summary : Distributed RESTful search engine built for the cloud
URL : https://www.elastic.co/

Installed Packages
Name : kibana
Arch : x86_64
Version : 8.8.2
Release : 1
Size : 715 M
Repo : installed
From repo : elasticsearch
Summary : Explore and visualize your Elasticsearch data
URL : https://www.elastic.co

McJava1967 · July 6, 2023, 6:50pm

I should add I get this only on trying to save a job. I can get my data from the same ES in my dashboard, and in Discover.

McJava1967 · July 6, 2023, 6:57pm

And we are using the 30-day Platinum trial.

dadoonet · July 11, 2023, 3:27pm

What do you mean by "when saving a job". Where is that happening. Do you have more logs in Kibana? And/or a screen capture?

McJava1967 · July 11, 2023, 3:51pm

Thanks, David! Please see screen shot.

McJava1967 · July 11, 2023, 3:54pm

For logs, do you want Client, Server, or both? And do I need to set to DEBUG?

McJava1967 · July 11, 2023, 4:01pm

When I try to Save again, I see this at INFO level in one of the logs:

[2023-07-11T15:57:45,791][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [usngksl1055bmp.us.int.kn] Opening job [brian_test_2]
[2023-07-11T15:57:45,794][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [usngksl1055bmp.us.int.kn] [brian_test_2] Loading model snapshot [N/A], job latest_record_timestamp [N/A]
[2023-07-11T15:57:45,879][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CResourceMonitor.cc@84] Setting model memory limit to 11 MB
[2023-07-11T15:57:46,281][INFO ][o.e.x.m.d.DatafeedJob    ] [usngksl1055bmp.us.int.kn] [brian_test_2] Datafeed started (from: 2023-07-04T15:33:08.652Z to: 2023-07-11T15:48:08.653Z) with frequency [450000ms]
[2023-07-11T15:57:46,507][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 10000 records written to autodetect; missingFieldCount=9936, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:46,633][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 20000 records written to autodetect; missingFieldCount=19886, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:46,828][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 30000 records written to autodetect; missingFieldCount=29809, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:47,447][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 40000 records written to autodetect; missingFieldCount=39746, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:47,776][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CTimeSeriesTestForChange.cc@699] time shift p-value = 0.997725
[2023-07-11T15:57:47,780][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 50000 records written to autodetect; missingFieldCount=49652, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedJob    ] [usngksl1055bmp.us.int.kn] [brian_test_2] Lookback has finished
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedRunner ] [usngksl1055bmp.us.int.kn] [no_realtime] attempt to stop datafeed [datafeed-brian_test_2] for job [brian_test_2]
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedRunner ] [usngksl1055bmp.us.int.kn] [no_realtime] try lock [20s] to stop datafeed [datafeed-brian_test_2] for job [brian_test_2]...
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedRunner ] [usngksl1055bmp.us.int.kn] [no_realtime] stopping datafeed [datafeed-brian_test_2] for job [brian_test_2], acquired [true]...
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedRunner ] [usngksl1055bmp.us.int.kn] [no_realtime] datafeed [datafeed-brian_test_2] for job [brian_test_2] has been stopped
[2023-07-11T15:57:47,996][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [usngksl1055bmp.us.int.kn] Closing job [brian_test_2], because [close job (api)]
[2023-07-11T15:57:47,997][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CTimeSeriesTestForChange.cc@699] time shift p-value = 0.784759 | repeated [5]
[2023-07-11T15:57:47,997][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CCmdSkeleton.cc@66] Handled 52439 records
[2023-07-11T15:57:47,998][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CAnomalyJob.cc@1624] Pruning obsolete models
[2023-07-11T15:57:48,063][INFO ][o.e.x.m.p.AbstractNativeProcess] [usngksl1055bmp.us.int.kn] [brian_test_2] State output finished
[2023-07-11T15:57:48,073][INFO ][o.e.x.m.j.p.a.o.AutodetectResultProcessor] [usngksl1055bmp.us.int.kn] [brian_test_2] 672 buckets parsed from autodetect output
[2023-07-11T15:57:48,314][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [usngksl1055bmp.us.int.kn] [brian_test_2] autodetect connection for job closed

McJava1967 · July 11, 2023, 4:02pm

I see this in one of the logs, on INFO level:

[2023-07-11T15:57:45,791][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [usngksl1055bmp.us.int.kn] Opening job [brian_test_2]
[2023-07-11T15:57:45,794][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [usngksl1055bmp.us.int.kn] [brian_test_2] Loading model snapshot [N/A], job latest_record_timestamp [N/A]
[2023-07-11T15:57:45,879][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CResourceMonitor.cc@84] Setting model memory limit to 11 MB
[2023-07-11T15:57:46,281][INFO ][o.e.x.m.d.DatafeedJob    ] [usngksl1055bmp.us.int.kn] [brian_test_2] Datafeed started (from: 2023-07-04T15:33:08.652Z to: 2023-07-11T15:48:08.653Z) with frequency [450000ms]
[2023-07-11T15:57:46,507][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 10000 records written to autodetect; missingFieldCount=9936, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:46,633][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 20000 records written to autodetect; missingFieldCount=19886, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:46,828][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 30000 records written to autodetect; missingFieldCount=29809, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:47,447][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 40000 records written to autodetect; missingFieldCount=39746, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:47,776][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CTimeSeriesTestForChange.cc@699] time shift p-value = 0.997725
[2023-07-11T15:57:47,780][INFO ][o.e.x.m.j.p.DataCountsReporter] [usngksl1055bmp.us.int.kn] [brian_test_2] 50000 records written to autodetect; missingFieldCount=49652, invalidDateCount=0, outOfOrderCount=0
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedJob    ] [usngksl1055bmp.us.int.kn] [brian_test_2] Lookback has finished
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedRunner ] [usngksl1055bmp.us.int.kn] [no_realtime] attempt to stop datafeed [datafeed-brian_test_2] for job [brian_test_2]
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedRunner ] [usngksl1055bmp.us.int.kn] [no_realtime] try lock [20s] to stop datafeed [datafeed-brian_test_2] for job [brian_test_2]...
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedRunner ] [usngksl1055bmp.us.int.kn] [no_realtime] stopping datafeed [datafeed-brian_test_2] for job [brian_test_2], acquired [true]...
[2023-07-11T15:57:47,949][INFO ][o.e.x.m.d.DatafeedRunner ] [usngksl1055bmp.us.int.kn] [no_realtime] datafeed [datafeed-brian_test_2] for job [brian_test_2] has been stopped
[2023-07-11T15:57:47,996][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [usngksl1055bmp.us.int.kn] Closing job [brian_test_2], because [close job (api)]
[2023-07-11T15:57:47,997][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CTimeSeriesTestForChange.cc@699] time shift p-value = 0.784759 | repeated [5]
[2023-07-11T15:57:47,997][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CCmdSkeleton.cc@66] Handled 52439 records
[2023-07-11T15:57:47,998][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [usngksl1055bmp.us.int.kn] [brian_test_2] [autodetect/6774] [CAnomalyJob.cc@1624] Pruning obsolete models
[2023-07-11T15:57:48,063][INFO ][o.e.x.m.p.AbstractNativeProcess] [usngksl1055bmp.us.int.kn] [brian_test_2] State output finished
[2023-07-11T15:57:48,073][INFO ][o.e.x.m.j.p.a.o.AutodetectResultProcessor] [usngksl1055bmp.us.int.kn] [brian_test_2] 672 buckets parsed from autodetect output
[2023-07-11T15:57:48,314][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [usngksl1055bmp.us.int.kn] [brian_test_2] autodetect connection for job closed

stephenb · July 11, 2023, 4:14pm

I ran into this before trying to find the answer / solution....

It is a configuration issue

1st do you have elasticsearch setup with the machine learning role?

can you run

GET _cat/nodes?v

McJava1967 · July 11, 2023, 4:23pm

stephenb · July 11, 2023, 4:26pm

Do you have security enabled?

oh @McJava1967 Welcome to the community!!!

Also can you click on the show full error and share that.

In short I think you are going to need to enable security

McJava1967 · July 11, 2023, 4:43pm

We don't have security enabled. It's just a POC right now. I should have shown this earlier, from the previous screen:

McJava1967 · July 11, 2023, 4:44pm

And this is the full error message you asked for:

stephenb · July 11, 2023, 4:45pm

Yup unfortunately I do not know if it is a BUG or Feature but pretty sure you are going to need to enable the default security at least. The default install would have done that.

I have relayed this back internal, but I have seen this before, enabling security fixed it.

McJava1967 · July 11, 2023, 5:41pm

Word on the street is that's it's a Feature designed to block interaction with AWS (a.k.a "this unknown product") Looks highly effective. But might want to let Elastic Kibana on through.

OK. Working on setting up minimal security. Thank you for help!

McJava1967 · July 12, 2023, 3:49pm

Just to confirm, Stephen. THAT WORKED.

If it helps any, my internet research says your Kibana is now requiring any ELK server to return something like "server=elasticsearch" as a header. My guess is your ELK server is failing to include that under certain circumstances.

Again, THANK YOU!

system · August 9, 2023, 3:49pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch log show kibana_version_mismatch Elasticsearch	11	36	December 4, 2024
Wrong version of elasticsearch release Elasticsearch	10	4779	February 27, 2017
Suppress Kibana version mismatch warning from the logs Kibana	5	1288	May 22, 2019
This version of Kibana (v8.4.0) is incompatible Kibana	4	4257	July 19, 2022
org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [34] Elasticsearch	1	646	July 6, 2017

Version mismatch message even though versions match

Related topics