I have a self-hosted elastic stack running 8.6.1. I'm trying to get fleet running but I'm hitting a wall on one part. The fleet server and any agents show as "healthy" but no data every shows up.
Digging into things I've discovered on the fleet server the filebeat instance is trying to connect via http instead of https. The agent itself is communicating over https to my elasticsearch server. So I continually see the following errors in the elasticsearch.log file:
[2023-02-15T14:52:55,738][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [hostname] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.1.12:9200, remoteAddress=/192.168.1.13:46696}
I've confirmed this by sniffing the traffic (using tcpdump) and it is definitely filebeat.
Running the elastic-agent diagnostics and digging into the components.yaml file I see this:
hosts:
kind:
listvalue:
values:
- kind:
stringvalue: http://192.168.1.12:9200
type:
kind:
stringvalue: elasticsearch
In fact, just using grep I find http://192.168.1.12:9200 appearing at least 5 times.
I've gone through the fleet server policy in Kibana and everything there shows correct (https) as far as I can tell. (Even viewing the generated policy after manually downloading it.)
I've even gone so far as deleting and letting the system recreate the fleet server policy to ensure it was correct.
I'm installing the fleet agent using this command:./elastic-agent install -f --fleet-server-es=https://192.168.1.12:9200 --fleet-server-service-token=<add_token> --fleet-server-policy=fleet-server-policy --fleet-server-es-ca=<path_filename_of_ca>
What am I missing that is causing the use of http instead of https?
Hi,
I'm missing the certificate-authorities argument, though that might not be the root cause. Configure SSL/TLS for self-managed Fleet Servers | Fleet and Elastic Agent Guide [master] | Elastic
Could you post your full agent policies (without sensitive info) to see if something stands out?
Thank you for responding.
The install command for the agent on a MacOS system I use is:https://192.168.1.13:8220 --enrollment-token=<token_goes_here> --insecure
The agent on the MacOS system installs and immediately shows up in the Fleet dashboard. I'm not sure how long it takes, but it shows up as "Updating" and seems to stay that way, but if I stop/restart it then it immediately flips to "Healthy".
In the Agent logs on the MacOS system it's showing a "failed to connect" but it's showing the correct URL for my elasticsearch server:
{"log.level":"error","@timestamp ":"2023-02-16T12:12:26.256Z","message":"Failed to connect to backoff(elasticsearch(https://192.168.1.12:9200 )): Get "https://192.168.1.12:9200 ": x509: “” certificate is not standards compliant","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":150,"file.name":"pipeline/client_worker.go"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","ecs.version":"1.6.0"}
I will post the full fleet agent and agent policies in upcoming (separate) posts below for you.
Fleet Server Policy:
id: fleet-server-policy
revision: 1
outputs:
default:
type: elasticsearch
hosts:
- 'https://192.168.1.12:9200'
output_permissions:
default:
_elastic_agent_monitoring:
indices:
- names:
- logs-elastic_agent.apm_server-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.apm_server-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.auditbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.auditbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.cloudbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.cloudbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.elastic_agent-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.endpoint_security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.endpoint_security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.filebeat_input-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.filebeat_input-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.filebeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.filebeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.fleet_server-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.fleet_server-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.heartbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.heartbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.metricbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.metricbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.osquerybeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.osquerybeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.packetbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.packetbeat-default
privileges:
- auto_configure
- create_doc
_elastic_agent_checks:
cluster:
- monitor
775f61a8-7125-4ef4-9123-a9400a515f93:
indices:
- names:
- logs-system.auth-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.syslog-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.application-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.system-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.cpu-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.diskio-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.filesystem-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.fsstat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.load-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.memory-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.network-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.process-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.process.summary-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.socket_summary-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.uptime-default
privileges:
- auto_configure
- create_doc
agent:
download:
sourceURI: 'https://artifacts.elastic.co/downloads/'
monitoring:
enabled: true
use_output: default
namespace: default
logs: true
metrics: true
inputs:
- id: logfile-system-775f61a8-7125-4ef4-9123-a9400a515f93
name: system-1
revision: 1
type: logfile
use_output: default
meta:
package:
name: system
version: 1.24.1
data_stream:
namespace: default
package_policy_id: 775f61a8-7125-4ef4-9123-a9400a515f93
streams:
- id: logfile-system.auth-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.auth
type: logs
ignore_older: 72h
paths:
- /var/log/auth.log*
- /var/log/secure*
exclude_files:
- .gz$
multiline:
pattern: ^\s
match: after
tags:
- system-auth
processors:
- add_locale: null
- id: logfile-system.syslog-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.syslog
type: logs
paths:
- /var/log/messages*
- /var/log/syslog*
exclude_files:
- .gz$
multiline:
pattern: ^\s
match: after
processors:
- add_locale: null
ignore_older: 72h
- id: winlog-system-775f61a8-7125-4ef4-9123-a9400a515f93
name: system-1
revision: 1
type: winlog
use_output: default
meta:
package:
name: system
version: 1.24.1
data_stream:
namespace: default
package_policy_id: 775f61a8-7125-4ef4-9123-a9400a515f93
streams:
- id: winlog-system.application-775f61a8-7125-4ef4-9123-a9400a515f93
name: Application
data_stream:
dataset: system.application
type: logs
condition: '${host.platform} == ''windows'''
ignore_older: 72h
- id: winlog-system.security-775f61a8-7125-4ef4-9123-a9400a515f93
name: Security
data_stream:
dataset: system.security
type: logs
condition: '${host.platform} == ''windows'''
ignore_older: 72h
- id: winlog-system.system-775f61a8-7125-4ef4-9123-a9400a515f93
name: System
data_stream:
dataset: system.system
type: logs
condition: '${host.platform} == ''windows'''
ignore_older: 72h
- id: system/metrics-system-775f61a8-7125-4ef4-9123-a9400a515f93
name: system-1
revision: 1
type: system/metrics
use_output: default
meta:
package:
name: system
version: 1.24.1
data_stream:
namespace: default
package_policy_id: 775f61a8-7125-4ef4-9123-a9400a515f93
streams:
- id: system/metrics-system.cpu-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.cpu
type: metrics
metricsets:
- cpu
cpu.metrics:
- percentages
- normalized_percentages
period: 10s
- id: system/metrics-system.diskio-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.diskio
type: metrics
metricsets:
- diskio
diskio.include_devices: null
period: 10s
- id: system/metrics-system.filesystem-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.filesystem
type: metrics
metricsets:
- filesystem
period: 1m
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- id: system/metrics-system.fsstat-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.fsstat
type: metrics
metricsets:
- fsstat
period: 1m
processors:
- drop_event.when.regexp:
system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- id: system/metrics-system.load-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.load
type: metrics
metricsets:
- load
condition: '${host.platform} != ''windows'''
period: 10s
- id: system/metrics-system.memory-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.memory
type: metrics
metricsets:
- memory
period: 10s
- id: system/metrics-system.network-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.network
type: metrics
metricsets:
- network
period: 10s
network.interfaces: null
- id: system/metrics-system.process-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.process
type: metrics
metricsets:
- process
period: 10s
process.include_top_n.by_cpu: 5
process.include_top_n.by_memory: 5
process.cmdline.cache.enabled: true
process.cgroups.enabled: false
process.include_cpu_ticks: false
processes:
- .*
- id: >-
system/metrics-system.process.summary-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.process.summary
type: metrics
metricsets:
- process_summary
period: 10s
- id: >-
system/metrics-system.socket_summary-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.socket_summary
type: metrics
metricsets:
- socket_summary
period: 10s
- id: system/metrics-system.uptime-775f61a8-7125-4ef4-9123-a9400a515f93
data_stream:
dataset: system.uptime
type: metrics
metricsets:
- uptime
period: 10s
- id: fleet-server-fleet_server-6f2215b0-f3c2-4239-9562-e37eebaf7b21
name: fleet_server-1
revision: 1
type: fleet-server
use_output: default
meta:
package:
name: fleet_server
version: 1.2.0
data_stream:
namespace: default
package_policy_id: 6f2215b0-f3c2-4239-9562-e37eebaf7b21
server:
port: 8220
host: 0.0.0.0
fleet:
hosts:
- 'https://192.168.1.13:8220'
Agent Policy:
id: 95299440-ad5c-11ed-82bd-8d6f2173b61d
revision: 2
outputs:
default:
type: elasticsearch
hosts:
- 'https://192.168.1.12:9200'
output_permissions:
default:
_elastic_agent_monitoring:
indices:
- names:
- logs-elastic_agent.apm_server-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.apm_server-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.auditbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.auditbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.cloudbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.cloudbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.elastic_agent-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.endpoint_security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.endpoint_security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.filebeat_input-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.filebeat_input-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.filebeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.filebeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.fleet_server-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.fleet_server-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.heartbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.heartbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.metricbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.metricbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.osquerybeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.osquerybeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.packetbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.packetbeat-default
privileges:
- auto_configure
- create_doc
_elastic_agent_checks:
cluster:
- monitor
eb1af3df-2113-4478-83c2-75620da7677e:
indices:
- names:
- logs-system.auth-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.syslog-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.application-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.system-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.cpu-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.diskio-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.filesystem-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.fsstat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.load-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.memory-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.network-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.process-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.process.summary-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.socket_summary-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.uptime-default
privileges:
- auto_configure
- create_doc
agent:
download:
sourceURI: 'https://artifacts.elastic.co/downloads/'
monitoring:
enabled: true
use_output: default
namespace: default
logs: true
metrics: true
inputs:
- id: logfile-system-eb1af3df-2113-4478-83c2-75620da7677e
name: system-2
revision: 2
type: logfile
use_output: default
meta:
package:
name: system
version: 1.24.1
data_stream:
namespace: default
package_policy_id: eb1af3df-2113-4478-83c2-75620da7677e
streams:
- id: logfile-system.auth-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.auth
type: logs
exclude_files:
- .gz$
ignore_older: 72h
paths:
- /var/log/auth.log*
- /var/log/secure*
multiline:
pattern: ^\s
match: after
processors:
- add_locale: null
tags:
- system-auth
- id: logfile-system.syslog-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.syslog
type: logs
exclude_files:
- .gz$
ignore_older: 72h
paths:
- /var/log/messages*
- /var/log/syslog*
- /var/log/system*
multiline:
pattern: ^\s
match: after
processors:
- add_locale: null
- id: winlog-system-eb1af3df-2113-4478-83c2-75620da7677e
name: system-2
revision: 2
type: winlog
use_output: default
meta:
package:
name: system
version: 1.24.1
data_stream:
namespace: default
package_policy_id: eb1af3df-2113-4478-83c2-75620da7677e
streams:
- id: winlog-system.application-eb1af3df-2113-4478-83c2-75620da7677e
name: Application
data_stream:
dataset: system.application
type: logs
condition: '${host.platform} == ''windows'''
ignore_older: 72h
- id: winlog-system.security-eb1af3df-2113-4478-83c2-75620da7677e
name: Security
data_stream:
dataset: system.security
type: logs
condition: '${host.platform} == ''windows'''
ignore_older: 72h
- id: winlog-system.system-eb1af3df-2113-4478-83c2-75620da7677e
name: System
data_stream:
dataset: system.system
type: logs
condition: '${host.platform} == ''windows'''
ignore_older: 72h
- id: system/metrics-system-eb1af3df-2113-4478-83c2-75620da7677e
name: system-2
revision: 2
type: system/metrics
use_output: default
meta:
package:
name: system
version: 1.24.1
data_stream:
namespace: default
package_policy_id: eb1af3df-2113-4478-83c2-75620da7677e
streams:
- id: system/metrics-system.cpu-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.cpu
type: metrics
period: 10s
cpu.metrics:
- percentages
- normalized_percentages
metricsets:
- cpu
- id: system/metrics-system.diskio-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.diskio
type: metrics
period: 10s
diskio.include_devices: null
metricsets:
- diskio
- id: system/metrics-system.filesystem-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.filesystem
type: metrics
period: 1m
metricsets:
- filesystem
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- id: system/metrics-system.fsstat-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.fsstat
type: metrics
period: 1m
metricsets:
- fsstat
processors:
- drop_event.when.regexp:
system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- id: system/metrics-system.load-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.load
type: metrics
condition: '${host.platform} != ''windows'''
period: 10s
metricsets:
- load
- id: system/metrics-system.memory-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.memory
type: metrics
period: 10s
metricsets:
- memory
- id: system/metrics-system.network-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.network
type: metrics
period: 10s
network.interfaces: null
metricsets:
- network
- id: system/metrics-system.process-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.process
type: metrics
process.include_top_n.by_memory: 5
period: 10s
processes:
- .*
process.include_top_n.by_cpu: 5
process.cgroups.enabled: false
process.cmdline.cache.enabled: true
metricsets:
- process
process.include_cpu_ticks: false
- id: >-
system/metrics-system.process.summary-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.process.summary
type: metrics
period: 10s
metricsets:
- process_summary
- id: >-
system/metrics-system.socket_summary-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.socket_summary
type: metrics
period: 10s
metricsets:
- socket_summary
- id: system/metrics-system.uptime-eb1af3df-2113-4478-83c2-75620da7677e
data_stream:
dataset: system.uptime
type: metrics
period: 10s
metricsets:
- uptime
fleet:
hosts:
- 'https://192.168.1.13:8220'
Does it help if you add --certificate-authorities argument to the agent enroll?
I found this issue mentioned in one of elastic repos, seems to be Mac specific: Reduce expiration time of certificates by jsoriano · Pull Request #1146 · elastic/elastic-package · GitHub
I don't see anything wrong with the agent policies, asked the Agent team to see if they can suggest a solution.
I was just digging around on the MacOS "certificate is not standards compliant" item. I came to the same conclusion:
When I originally installed elastic and generated the certs, it was prior to Apple changing a policy to limit self-signed to 825 days. For my Mac I'll have to regenerate certificates.
For comparison, I installed the agent on a Linux server and this is what I see in the log:
{"log.level":"error","@timestamp":"2023-02-16T14:30:09.673Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":38,"file.name":"transport/logging.go"},"service.name":"filebeat","log.logger":"esclientleg","network":"tcp","address":"192.168.1.12:9200","ecs.version":"1.6.0","ecs.version":"1.6.0"}
I did use the --insecure option installing it.
So there appear to be multiple potential issues going on:
Agent on MacOS: Certificate's expiration is too far out, I need to regenerate certs and that should resolve the issue on MacOS (at least as far as the current cert issue). But, the system is trying to connect to the correct elastic server host using https. Appear to connect OK to the fleet server agent.
Agent on Linux: Doesn't like the self-signed certificate. But, the system is trying to connect to the correct elastic server host using https. Appear to connect OK to the fleet server agent.
Fleet Agent on Linux: This is the original issue I posted about. The fleet agent can communicate OK, but the filebeat/metricbeat/etc. is attempting to connect to the correct elastic server but using HTTP instead of HTTPS.
I think #3 above is the most serious one to resolve (and my original issue). The others can probably be sorted out by fixing cert related issues. Of those two, the MacOS one (#1 ) should be easy, for the other one (#2 ) I'll do some digging.
I believe this can be closed out at this point:
I stood up a few virtual systems where I installed elasticsearch, kibana, fleet server and an agent without any issues.
This indicates to me there is "something" in my physical install that's out of whack. Given the original install was done a few years ago (using 7.x) and upgraded as new releases came out, it's possible there's a remnant file/setting that is causing my issue (or an issue with the existing certs).
At some point I'll backup my current configurations/dashboards and perform a new, clean install from scratch with 8.x. which should put everything back to a known, clean starting point.
Last follow-up to confirm:
Performed complete remove/reinstall from scratch and fleet server installed on a system without any issues and is properly feeding data into Elastic.
This confirms there was "something" someplace on my previous install that was causing the weird issue.
system
March 21, 2023, 1:39pm
10
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.