Not able to view Overview page ,Analytics page ,API logs page in AppSearch

Subhasis_Dash · October 6, 2020, 7:24am

Hi ,

While clicking on Overview Metrics or Analytics(similar error) page we get the following error logs:

[2020-10-06T07:01:44.151+00:00][25][2298][action_controller][INFO]: [b5d5890cc0a1bcabb2e2a2fec1927002] Processing by LocoMoco::EnginesController#overview_metrics as JSON
[2020-10-06T07:01:44.151+00:00][25][2298][action_controller][INFO]: [b5d5890cc0a1bcabb2e2a2fec1927002]   Parameters: {"host"=>"dev-dwt-elastic-app-search.cloud.corporate.ge.com", "protocol"=>"http", "slug"=>"search-engine}
[2020-10-06T07:01:44.245+00:00][25][2298][action_controller][INFO]: [b5d5890cc0a1bcabb2e2a2fec1927002] Completed 500 Internal Server Error in 93ms
[2020-10-06T07:01:44.249+00:00][25][2298][app-server][ERROR]: [b5d5890cc0a1bcabb2e2a2fec1927002] /tmp/jetty-0.0.0.0-80-app-search.war-_-any-1161062368294022305.dir/webapp/WEB-INF/lib/analytics_query_service.class:307:in `make_request': Logs unavailable. Caused by MultiJson::ParseError: unexpected token at '<html>
 (SwiftypeExceptions::LogsUnavailableError)
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>

While clicking on Api logs page we get the following error logs:

[2020-10-06T07:16:02.204+00:00][25][2296][action_controller][INFO]: [c278231f171d6c74a7f8d811f165829f] Processing by LocoMoco::ApiLogsController#details as JSON
[2020-10-06T07:16:02.204+00:00][25][2296][action_controller][INFO]: [c278231f171d6c74a7f8d811f165829f]   Parameters: {"host"=>"dev-dwt-elastic-app-search.cloud.corporate.ge.com", "protocol"=>"http", "engine_slug"=>"search-engine"}
[2020-10-06T07:16:02.308+00:00][25][2296][action_controller][INFO]: [c278231f171d6c74a7f8d811f165829f] Completed 200 OK in 103ms (Views: 0.4ms)
[2020-10-06T07:16:02.675+00:00][25][2294][app-server][INFO]: [5d0234ba59d1788ae43854e37b243bf3] Started GET "/as/engines/search-engine/api_logs/collection?filters%5Bdate%5D%5Bfrom%5D=2020-10-05T07%3A01%3A32.540Z&filters%5Bdate%5D%5Bto%5D=2020-10-06T07%3A01%3A32.540Z&sort_direction=desc&page%5Bcurrent%5D=1" for 172.16.8.0 at 2020-10-06 07:16:02 +0000
[2020-10-06T07:16:02.676+00:00][25][2314][app-server][INFO]: [c664bfc896bbb9761c4346340f3cc223] Started GET "/as/engines/search-engine/api_logs/collection?filters%5Bdate%5D%5Bfrom%5D=2020-10-05T07%3A16%3A02.484Z&filters%5Bdate%5D%5Bto%5D=2020-10-06T07%3A16%3A02.484Z&sort_direction=desc&page%5Bcurrent%5D=1" for 172.16.11.0 at 2020-10-06 07:16:02 +0000
[2020-10-06T07:16:02.682+00:00][25][2294][action_controller][INFO]: [5d0234ba59d1788ae43854e37b243bf3] Processing by LocoMoco::ApiLogsController#index as JSON
[2020-10-06T07:16:02.682+00:00][25][2294][action_controller][INFO]: [5d0234ba59d1788ae43854e37b243bf3]   Parameters: {"filters"=>{"date"=>{"from"=>"2020-10-05T07:01:32.540Z", "to"=>"2020-10-06T07:01:32.540Z"}}, "sort_direction"=>"desc", "page"=>{"current"=>"1"}, "host"=>"dev-dwt-elastic-app-search.cloud.corporate.ge.com", "protocol"=>"http", "engine_slug"=>"search-engine"}
[2020-10-06T07:16:02.682+00:00][25][2314][action_controller][INFO]: [c664bfc896bbb9761c4346340f3cc223] Processing by LocoMoco::ApiLogsController#index as JSON
[2020-10-06T07:16:02.683+00:00][25][2314][action_controller][INFO]: [c664bfc896bbb9761c4346340f3cc223]   Parameters: {"filters"=>{"date"=>{"from"=>"2020-10-05T07:16:02.484Z", "to"=>"2020-10-06T07:16:02.484Z"}}, "sort_direction"=>"desc", "page"=>{"current"=>"1"}, "host"=>"dev-dwt-elastic-app-search.cloud.corporate.ge.com", "protocol"=>"http", "engine_slug"=>"search-engine"}
**[2020-10-06T07:16:02.783+00:00][25][2314][app-server][ERROR]: [c664bfc896bbb9761c4346340f3cc223] ["Logs are currently unavailable; please try again in a few minutes"]**
**[2020-10-06T07:16:02.784+00:00][25][2294][app-server][ERROR]: [5d0234ba59d1788ae43854e37b243bf3] ["Logs are currently unavailable; please try again in a few minutes"]**
[2020-10-06T07:16:02.795+00:00][25][2314][action_controller][INFO]: [c664bfc896bbb9761c4346340f3cc223] Completed 503 Service Unavailable in 112ms (Views: 0.6ms)
[2020-10-06T07:16:02.797+00:00][25][2294][action_controller][INFO]: [5d0234ba59d1788ae43854e37b243bf3] Completed 503 Service Unavailable in 114ms (Views: 0.3ms)

Please let me know if you know what is the issue or solution to this problem

JasonStoltz · October 20, 2020, 3:50pm

Could you please give us more information?

What version of App Search are you using? Where do you have it installed? Anything unique about your setup that is relevant?

gourab.das · November 12, 2020, 12:07pm

Hi Jason,

I am also part of the same team as Subhasish, working on same project.The app search version is 7.9.3 and it's deployed on kubernetes cluster runs on AWS cloud env. Also I am sending the error details below. Please advise us as it's impacting our current work. I would be looking forward to your reply

@JasonStoltz @Subhasis_Dash

ross.bell · November 13, 2020, 3:59pm

Given the breadth of errors, I'm wondering if Enterprise Search is having trouble connecting to Elasticsearch, although I doubt the app would load as far as it did if it wasn't. I'll ask around to see if this analytics data is stored in a different way/place that perhaps your instance is finding uniquely unaccessible.

And I would definitely try reinstalling or upgrading to 7.10, which was recently released. That will help remove any doubt about versions. Let us know if that helps.

Subhasis_Dash · November 18, 2020, 1:50pm

@JasonStoltz @ross.bell some other issues:

Not able to see any data API logs tab in app search UI
Schema is not auto created upon indexing to a new engine:
Expanding more on this if we create a new engine and try to index a document( through api or UI) we don't see that document in engine till we go and create schema manually.

So basically functionality that are not working as expected are:
Analytics, API logs and auto schema creation

Also previously we were using 7.6.2 version of the App Search were initially all functionalities were working then after few months we started facing the above issue so we upgraded to 7.9.3 but the issues still persist.

ross.bell · November 18, 2020, 4:40pm

In general I'd recommend folks upgrade to the latest version (7.10). Don't forget you'll want to upgrade Elasticsearch as well if that's an option. A related option would be to use a fresh deploy and re-index your data. I understand that's a pain if you're already heavily using the existing deploy.

The problems you're describing are unexpected and I suspect there's an issue between App Search and Elasticsearch.

Subhasis_Dash · December 2, 2020, 6:44am

@JasonStoltz @ross.bell Hi as per your suggestion we have now upgraded to 7.10 but the issues still persist.
we still get 401 Unauthorized upon clicking Api logs and the other issues mentioned above

[2020-12-02T06:23:24.324+00:00][28][2520][app-server][INFO]: [1fae6d8f1e55d8b48770df17703bd4d9] Started GET "/as/log_settings" for 3.251.21.186 at 2020-12-02 06:23:24 +0000
[2020-12-02T06:23:24.324+00:00][28][7976][app-server][INFO]: [91c841f972fb089ebe13d8fff0e40172] Started GET "/as/engines/tests/api_logs/collection?filters%5Bdate%5D%5Bfrom%5D=2020-12-01T06%3A23%3A24.149Z&filters%5Bdate%5D%5Bto%5D=2020-12-02T06%3A23%3A24.149Z&sort_direction=desc&page%5Bcurrent%5D=1" for 3.251.21.186 at 2020-12-02 06:23:24 +0000
[2020-12-02T06:23:24.324+00:00][28][7972][action_controller][INFO]: [4cb01bd4c29a623aebd3f9997c398e8d]   Parameters: {"filters"=>{"date"=>{"from"=>"2020-12-01T05:37:29.952Z", "to"=>"2020-12-02T05:37:29.952Z"}}, "sort_direction"=>"desc", "page"=>{"current"=>"1"}, "host"=>"host.com", "protocol"=>"https", "engine_slug"=>"tests"}
[2020-12-02T06:23:24.327+00:00][28][2520][action_controller][INFO]: [1fae6d8f1e55d8b48770df17703bd4d9] Processing by LocoMoco::Dashboard::LogSettingsController#show as JSON
[2020-12-02T06:23:24.327+00:00][28][7976][action_controller][INFO]: [91c841f972fb089ebe13d8fff0e40172] Processing by LocoMoco::Dashboard::ApiLogsController#index as JSON
[2020-12-02T06:23:24.327+00:00][28][2520][action_controller][INFO]: [1fae6d8f1e55d8b48770df17703bd4d9]   Parameters: {"host"=>"host.com", "protocol"=>"https"}
[2020-12-02T06:23:24.327+00:00][28][7976][action_controller][INFO]: [91c841f972fb089ebe13d8fff0e40172]   Parameters: {"filters"=>{"date"=>{"from"=>"2020-12-01T06:23:24.149Z", "to"=>"2020-12-02T06:23:24.149Z"}}, "sort_direction"=>"desc", "page"=>{"current"=>"1"}, "host"=>"host.com", "protocol"=>"https", "engine_slug"=>"tests"}
[2020-12-02T06:23:24.500+00:00][28][2520][action_controller][INFO]: [1fae6d8f1e55d8b48770df17703bd4d9] Filter chain halted as :verify_can_update_settings rendered or redirected
[2020-12-02T06:23:24.504+00:00][28][2520][action_controller][INFO]: [1fae6d8f1e55d8b48770df17703bd4d9] Completed 401 Unauthorized in 177ms
[2020-12-02T06:23:24.621+00:00][28][7976][action_controller][INFO]: [91c841f972fb089ebe13d8fff0e40172] Completed 200 OK in 293ms (Views: 0.4ms)
[2020-12-02T06:23:24.625+00:00][28][7972][action_controller][INFO]: [4cb01bd4c29a623aebd3f9997c398e8d] Completed 200 OK in 301ms (Views: 0.3ms)
[2020-12-02T06:23:30.802+00:00][28][2498][app-server][INFO]: [2ea0855ae22fb447e375d329d6444736] Started GET "/as/engines/tests/api_logs/collection?filters%5Bdate%5D%5Bfrom%5D=2020-12-01T06%3A23%3A30.626Z&filters%5Bdate%5D%5Bto%5D=2020-12-02T06%3A23%3A30.626Z&sort_direction=desc&page%5Bcurrent%5D=1" for 3.251.21.186 at 2020-12-02 06:23:30 +0000
[2020-12-02T06:23:30.805+00:00][28][2498][action_controller][INFO]: [2ea0855ae22fb447e375d329d6444736] Processing by LocoMoco::Dashboard::ApiLogsController#index as JSON
[2020-12-02T06:23:30.806+00:00][28][2498][action_controller][INFO]: [2ea0855ae22fb447e375d329d6444736]   Parameters: {"filters"=>{"date"=>{"from"=>"2020-12-01T06:23:30.626Z", "to"=>"2020-12-02T06:23:30.626Z"}}, "sort_direction"=>"desc", "page"=>{"current"=>"1"}, "host"=>"host.com", "protocol"=>"https", "engine_slug"=>"tests"}
[2020-12-02T06:23:30.962+00:00][28][2498][action_controller][INFO]: [2ea0855ae22fb447e375d329d6444736] Completed 200 OK in 156ms (Views: 0.4ms)

And

we use the following configs for the elastic search cluster:

    cluster\.name: "appsearch"
    network\.host: 0.0.0.0
    action\.auto_create_index: ".ent-search-*-logs-*,-.ent-search-*,-test-.ent-search-*,+*"
    xpack\.security\.enabled: true
    xpack\.security\.authc\.api_key\.enabled: true
    xpack\.security\.transport\.ssl\.enabled: true
    xpack\.security\.transport\.ssl\.verification_mode: certificate
    xpack\.security\.transport\.ssl\.key: /usr/share/elasticsearch/config/certs/tls.key
    xpack\.security\.transport\.ssl\.certificate: /usr/share/elasticsearch/config/certs/tls.crt
    xpack\.security\.http\.ssl\.enabled: true
    xpack\.security\.http\.ssl\.verification_mode: certificate
    xpack\.security\.http\.ssl\.key: /usr/share/elasticsearch/config/certs/tls.key
    xpack\.security\.http\.ssl\.certificate: /usr/share/elasticsearch/config/certs/tls.crt

Please let us know if you could narrow down the issue with these logs

ross.bell · December 2, 2020, 4:53pm

@Subhasis_Dash can you confirm whether the schema auto-create issue originally reported still exists? I'm noticing the errors in the logs are a little different. Is the user you're testing with an admin?

Subhasis_Dash · December 3, 2020, 6:06am

We found what was causing the issue :
Filebeat appends the port to the elastic url which by default is 9200 but we are proxying 443 to 9200 so it was failing and we were having these issues (Api logs, Analytics and schema auto create).
We have fixed it now and we are using 7.10 version

ross.bell · December 3, 2020, 1:50pm

Awesome, glad to hear you found the issue!

system · December 31, 2020, 1:50pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.