Kibana stop working

elasticforme · December 7, 2021, 1:22am

working kibana stop working

here is message that comes up when I try to access kibana:5601

{"type":"log","@timestamp":"2021-12-06T19:18:16-06:00","tags":["error","plugins","taskManager"],"pid":3497,"message":"Failed to poll for work: Error: work has timed out"}

{"type":"log","@timestamp":"2021-12-06T19:19:40-06:00","tags":["error","plugins","taskManager"],"pid":3497,"message":"[Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity"}

what went wrong?

I can call api query

warkolm · December 7, 2021, 2:27am

Please share more of the Kibana log.

elasticforme · December 7, 2021, 1:44pm

This are last few lines

{"type":"response","@timestamp":"2021-12-07T07:40:36-06:00","tags":[],"pid":6206,"method":"get","statusCode":200,"req":{"url":"/app/home","method":"get","headers":{"host":"houelk11:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.10.64.95","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36"},"res":{"statusCode":200,"responseTime":29821,"contentLength":127543},"message":"GET /app/home 200 29821ms - 124.6KB"}
{"type":"log","@timestamp":"2021-12-07T07:41:07-06:00","tags":["error","plugins","taskManager"],"pid":6206,"message":"[Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity"}


{"type":"log","@timestamp":"2021-12-07T07:42:28-06:00","tags":["error","plugins","taskManager"],"pid":6206,"message":"Failed to poll for work: Error: work has timed out"}
{"type":"log","@timestamp":"2021-12-07T07:43:00-06:00","tags":["warning","plugins","taskManager"],"pid":6206,"message":"Cancelling task security:endpoint-diagnostics \"security:endpoint-diagnostics:1.0.0\" as it expired at 2021-12-07T13:42:59.298Z (with timeout set at 1m)."}

[root@houelkm13 ~]# curl  -u elastic:elastic houelk11:5601/status
Kibana server is not ready

elasticforme · December 7, 2021, 2:04pm

Finally kibana gui return. but no command gets executed from it

for example when I go to dev tool and type "GET /_cluster/health"

nothing returns just goes in circle. and log message pops up in logfile

{"type":"log","@timestamp":"2021-12-07T07:59:33-06:00","tags":["error","plugins","taskManager"],"pid":6244,"message":"[Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity"}

and kibana restarts by self as well here it restarted just second ago

{"type":"response","@timestamp":"2021-12-07T07:59:33-06:00","tags":[],"pid":6244,"method":"post","statusCode":200,"req":{"url":"/internal/bsearch","method":"post","headers":{"host":"houelk11:5601","connection":"keep-alive","content-length":"23589","kbn-version":"7.12.0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://houelk11:5601","referer":"http://houelk11:5601/s/validation/app/dashboards","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.10.80.24","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","referer":"http://houelk11:5601/s/validation/app/dashboards"},"res":{"statusCode":200,"responseTime":2288},"message":"POST /internal/bsearch 200 2288ms"}
{"type":"log","@timestamp":"2021-12-07T08:01:24-06:00","tags":["info","plugins-service"],"pid":6303,"message":"Plugin \"fleet\" is disabled."}
{"type":"log","@timestamp":"2021-12-07T08:01:24-06:00","tags":["warning","config","deprecation"],"pid":6303,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-12-07T08:01:24-06:00","tags":["info","plugins-service"],"pid":6303,"message":"Plugin \"osquery\" is disabled."}
{"type":"log","@timestamp":"2021-12-07T08:01:25-06:00","tags":["info","plugins-system"],"pid":6303,"message":"Setting up [99] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsLegacy,kibanaLegacy,translations,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeTimelion,features,licenseManagement,watcher,canvas,visTypeVislib,visTypeVega,visTypeTagcloud,visTypeTable,visTypeMarkdown,visTypeMetric,tileMap,regionMap,visTypeXy,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,indexManagement,rollup,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,upgradeAssistant,snapshotRestore,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,ml,securitySolution,case,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-12-07T08:01:25-06:00","tags":["info","plugins","taskManager"],"pid":6303,"message":"TaskManager is identified by the Kibana UUID: 698c47eb-d3fe-4338-a148-71e4140d86c4"}
{"type":"log","@timestamp":"2021-12-07T08:01:25-06:00","tags":["warning","plugins","security","config"],"pid":6303,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2021-12-07T08:01:25-06:00","tags":["warning","plugins","reporting","config"],"pid":6303,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 8.4 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-12-07T08:01:25-06:00","tags":["info","plugins","monitoring","monitoring"],"pid":6303,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-12-07T08:01:25-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-12-07T08:01:26-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-12-07T08:01:26-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH"}
{"type":"log","@timestamp":"2021-12-07T08:01:26-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH"}
{"type":"log","@timestamp":"2021-12-07T08:01:26-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS"}
{"type":"log","@timestamp":"2021-12-07T08:01:26-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS"}
{"type":"log","@timestamp":"2021-12-07T08:01:26-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK"}
{"type":"log","@timestamp":"2021-12-07T08:01:26-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK"}
{"type":"log","@timestamp":"2021-12-07T08:01:26-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE"}
{"type":"log","@timestamp":"2021-12-07T08:01:26-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana_task_manager] Migration completed after 295ms"}
{"type":"log","@timestamp":"2021-12-07T08:02:03-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE"}

{"type":"log","@timestamp":"2021-12-07T08:02:03-06:00","tags":["info","savedobjects-service"],"pid":6303,"message":"[.kibana] Migration completed after 37368ms"}
{"type":"log","@timestamp":"2021-12-07T08:02:03-06:00","tags":["info","plugins-system"],"pid":6303,"message":"Starting [99] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsLegacy,kibanaLegacy,translations,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeTimelion,features,licenseManagement,watcher,canvas,visTypeVislib,visTypeVega,visTypeTagcloud,visTypeTable,visTypeMarkdown,visTypeMetric,tileMap,regionMap,visTypeXy,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,indexManagement,rollup,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,upgradeAssistant,snapshotRestore,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,ml,securitySolution,case,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-12-07T08:02:06-06:00","tags":["listening","info"],"pid":6303,"message":"Server running at http://houelk11:5601"}
{"type":"log","@timestamp":"2021-12-07T08:02:06-06:00","tags":["info","plugins","watcher"],"pid":6303,"message":"Your basic license does not support watcher. Please upgrade your license."}
{"type":"log","@timestamp":"2021-12-07T08:02:06-06:00","tags":["info","plugins","crossClusterReplication"],"pid":6303,"message":"Your basic license does not support crossClusterReplication. Please upgrade your license."}
{"type":"log","@timestamp":"2021-12-07T08:02:06-06:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":6303,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2021-12-07T08:02:07-06:00","tags":["warning","plugins","reporting"],"pid":6303,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}

{"type":"log","@timestamp":"2021-12-07T08:02:11-06:00","tags":["info","http","server","Kibana"],"pid":6303,"message":"http server running at http://houelk11:5601"}
{"type":"response","@timestamp":"2021-12-07T08:02:11-06:00","tags":[],"pid":6303,"method":"post","statusCode":200,"req":{"url":"/api/monitoring/v1/alert/paC-XS3YTGeGoG2xIrGjUA/status","method":"post","headers":{"host":"houelk11:5601","connection":"keep-alive","content-length":"117","kbn-version":"7.12.0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://houelk11:5601","referer":"http://houelk11:5601/app/monitoring","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.10.80.24","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","referer":"http://houelk11:5601/app/monitoring"},"res":{"statusCode":200,"responseTime":189,"contentLength":2},"message":"POST /api/monitoring/v1/alert/paC-XS3YTGeGoG2xIrGjUA/status 200 189ms - 2.0B"}
{"type":"response","@timestamp":"2021-12-07T08:02:11-06:00","tags":[],"pid":6303,"method":"post","statusCode":200,"req":{"url":"/api/monitoring/v1/clusters/paC-XS3YTGeGoG2xIrGjUA/logstash/nodes","method":"post","headers":{"host":"houelk11:5601","connection":"keep-alive","content-length":"81","accept":"application/json, text/plain, */*","kbn-version":"7.12.0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","content-type":"application/json;charset=UTF-8","origin":"http://houelk11:5601","referer":"http://houelk11:5601/app/monitoring","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.10.80.24","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","referer":"http://houelk11:5601/app/monitoring"},"res":{"statusCode":200,"responseTime":1139,"contentLength":1921},"message":"POST /api/monitoring/v1/clusters/paC-XS3YTGeGoG2xIrGjUA/logstash/nodes 200 1139ms - 1.9KB"}

elasticforme · December 7, 2021, 2:09pm

and bunch of these messages as well

{"type":"log","@timestamp":"2021-12-07T08:04:31-06:00","tags":["warning","plugins","taskManager"],"pid":6303,"message":"Cancelling task security:endpoint-diagnostics \"security:endpoint-diagnostics:1.0.0\" as it expired at 2021-12-07T14:04:30.899Z (with timeout set at 1m)."}
{"type":"log","@timestamp":"2021-12-07T08:05:00-06:00","tags":["error","plugins","taskManager"],"pid":6303,"message":"Failed to poll for work: Error: work has timed out"}


{"type":"log","@timestamp":"2021-12-07T08:05:33-06:00","tags":["warning","plugins","taskManager"],"pid":6303,"message":"Cancelling task security:endpoint-diagnostics \"security:endpoint-diagnostics:1.0.0\" as it expired at 2021-12-07T14:05:31.869Z (with timeout set at 1m)."}
{"type":"log","@timestamp":"2021-12-07T08:06:03-06:00","tags":["error","plugins","taskManager"],"pid":6303,"message":"Failed to poll for work: Error: work has timed out"}
{"type":"log","@timestamp":"2021-12-07T08:06:35-06:00","tags":["warning","plugins","taskManager"],"pid":6303,"message":"Cancelling task security:endpoint-diagnostics \"security:endpoint-diagnostics:1.0.0\" as it expired at 2021-12-07T14:06:34.425Z (with timeout set at 1m)."}
{"type":"log","@timestamp":"2021-12-07T08:07:05-06:00","tags":["error","plugins","taskManager"],"pid":6303,"message":"Failed to poll for work: Error: work has timed out"}
{"type":"log","@timestamp":"2021-12-07T08:07:09-06:00","tags":["warning","plugins","taskManager"],"pid":6303,"message":"Cancelling task search_sessions_monitor \"data_enhanced_search_sessions_monitor\" as it expired at 2021-12-07T14:07:09.627Z after running for 05m 00s (with timeout set at 5m)."}
{"type":"log","@timestamp":"2021-12-07T08:07:37-06:00","tags":["warning","plugins","taskManager"],"pid":6303,"message":"Cancelling task security:endpoint-diagnostics \"security:endpoint-diagnostics:1.0.0\" as it expired at 2021-12-07T14:07:36.414Z (with timeout set at 1m)."}
{"type":"log","@timestamp":"2021-12-07T08:07:39-06:00","tags":["error","plugins","taskManager"],"pid":6303,"message":"Failed to poll for work: Error: work has timed out"}
{"type":"log","@timestamp":"2021-12-07T08:08:09-06:00","tags":["error","plugins","taskManager"],"pid":6303,"message":"Failed to poll for work: Error: work has timed out"}

{"type":"log","@timestamp":"2021-12-07T08:08:41-06:00","tags":["warning","plugins","taskManager"],"pid":6303,"message":"Cancelling task security:endpoint-diagnostics \"security:endpoint-diagnostics:1.0.0\" as it expired at 2021-12-07T14:08:40.754Z (with timeout set at 1m)."}

elasticforme · December 7, 2021, 3:17pm

found this which exactly shows what problem I have. but can't figure out how to fix it

github.com/elastic/kibana

[Task Manager] When a Task's timeout is an invalid interval it can crash Task Manager

opened 05:11PM - 11 Jan 21 UTC

closed 10:46AM - 11 Aug 21 UTC

gmmorris

bug Feature:Task Manager Team:Alerting Services resilience estimate:small

This error happens at times when a task has an invalid timeout (such as `1.5s`).… ``` [error][plugins][taskManager] [Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity ``` I'm not sure what edge case causes it as trying to naively reproducing locally didn't result in that error, though it doesn happen consistently on https://github.com/elastic/kibana/pull/64853

elasticforme · December 7, 2021, 4:02pm

few more update. I remove .kibana_task_manager index and .kibana-event-log index

make sure all I have in config is this

[root@houelk11 ~]# cat /etc/kibana/kibana.yml |grep -v ^# |sed '/^$/d'
server.port: 5601
server.host: "houelk11"
server.name: "houelk11"
elasticsearch.hosts: ["http://houelk11:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "kibana"
xpack.security.enabled: true
xpack.fleet.enabled: false
logging.dest: /s1/log/kibana/kibana.log
xpack.security.session.idleTimeout: 24h

and restarted kibana. got this meesage

{"type":"log","@timestamp":"2021-12-07T09:57:33-06:00","tags":["listening","info"],"pid":2525,"message":"Server running at http://houelk11:5601"}
{"type":"log","@timestamp":"2021-12-07T09:57:33-06:00","tags":["info","plugins","watcher"],"pid":2525,"message":"Your basic license does not support watcher. Please upgrade your license."}
{"type":"log","@timestamp":"2021-12-07T09:57:33-06:00","tags":["info","plugins","crossClusterReplication"],"pid":2525,"message":"Your basic license does not support crossClusterReplication. Please upgrade your license."}
{"type":"log","@timestamp":"2021-12-07T09:57:33-06:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":2525,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2021-12-07T09:57:33-06:00","tags":["info","http","server","Kibana"],"pid":2525,"message":"http server running at http://houelk11:5601"}
{"type":"response","@timestamp":"2021-12-07T09:57:34-06:00","tags":[],"pid":2525,"method":"post","statusCode":200,"req":{"url":"/api/monitoring/v1/alert/paC-XS3YTGeGoG2xIrGjUA/status","method":"post","headers":{"host":"houelk11:5601","connection":"keep-alive","content-length":"117","kbn-version":"7.12.0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://houelk11:5601","referer":"http://houelk11:5601/app/monitoring","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.10.80.24","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","referer":"http://houelk11:5601/app/monitoring"},"res":{"statusCode":200,"responseTime":333,"contentLength":2},"message":"POST /api/monitoring/v1/alert/paC-XS3YTGeGoG2xIrGjUA/status 200 333ms - 2.0B"}
{"type":"response","@timestamp":"2021-12-07T09:57:34-06:00","tags":[],"pid":2525,"method":"post","statusCode":200,"req":{"url":"/api/monitoring/v1/clusters/paC-XS3YTGeGoG2xIrGjUA/logstash/nodes","method":"post","headers":{"host":"houelk11:5601","connection":"keep-alive","content-length":"81","accept":"application/json, text/plain, */*","kbn-version":"7.12.0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","content-type":"application/json;charset=UTF-8","origin":"http://houelk11:5601","referer":"http://houelk11:5601/app/monitoring","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.10.80.24","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","referer":"http://houelk11:5601/app/monitoring"},"res":{"statusCode":200,"responseTime":1176,"contentLength":1923},"message":"POST /api/monitoring/v1/clusters/paC-XS3YTGeGoG2xIrGjUA/logstash/nodes 200 1176ms - 1.9KB"}
{"type":"log","@timestamp":"2021-12-07T09:57:35-06:00","tags":["warning","plugins","reporting"],"pid":2525,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2021-12-07T09:58:12-06:00","tags":["error","plugins","taskManager"],"pid":2525,"message":"Task apm-telemetry-task \"apm-telemetry-task\" failed: TimeoutError: Request timed out"}
{"type":"log","@timestamp":"2021-12-07T09:58:12-06:00","tags":["error","plugins","taskManager"],"pid":2525,"message":"[Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity"}

elasticforme · December 7, 2021, 7:03pm

Only one way I knew the answer is remove .kibana index.

finally after lot of searching around I did that. unlucky me I have not yet setup back up on this hence I have to resetup spaces and dashboards.

system · January 4, 2022, 7:03pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.