kelvinferraz
(Kelvin Ferraz da Silva)
May 16, 2022, 2:43am
1
Hello everyone!
Recently I decided to test the version of 8.2 for my stack Elastic, including APM. I was using the 7.5 version.
But I found the serious problems to understand and configure the self-hosted version.
In this scenario, I do the configuration of the complete stack, including the APM again, but when I tried to test, my scenario don't work very well, like my APM received some erros like:
[elastic_agent.apm_server][error] failed to index event (security_exception): action [indices:admin/auto_create] is unauthorized for API key id [xB6myoAB0eENg0fBank6] of user [elastic/fleet-server] on indices [metrics-apm.internal-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]
I don't know if I made the correct configuration, but for some reason, my script test can't show me something in the monitoring dashboard of Kibana (just that I already configured the fleet in the environment).
APM Status:
Fleet status:
Kibana version : 8.2
Elasticsearch version :8.2
APM Server version : (I don't know exactly what this use, because after configured my fleet, the server stop to start, I believe is 8.2 too)
I will be grateful for any help.
Regards.
kelvinferraz
(Kelvin Ferraz da Silva)
May 30, 2022, 5:16pm
2
kelvinferraz:
of
Hello someone can help me?
simitt
(Silvia Mitter)
May 30, 2022, 6:06pm
3
Hi @kelvinferraz could you share the Elastic Agent policy that you are using (please redact any sensitive information before sharing).
You will find the policy when navigating to Fleet/Agent policies/ clicking on the Actions button and select View policies , similar to:
simitt
(Silvia Mitter)
May 31, 2022, 5:52am
5
What you shared is just the overview of the installed integrations, please click on Actions/View Policy and share the policy.
kelvinferraz
(Kelvin Ferraz da Silva)
May 31, 2022, 11:26am
6
So sorry,
My Policy:
id: 7e703890-d1f1-11ec-b247-a942f89349a3
revision: 10
outputs:
default:
type: elasticsearch
hosts:
- 'https://MYIPADDRESS:9200'
ssl.ca_trusted_fingerprint: 89ac986af82ab7b31da528d5bb655621e53fbad63de339636f568905372f4308
output_permissions:
default:
_elastic_agent_monitoring:
indices:
- names:
- metrics-elastic_agent.osquerybeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.packetbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.filebeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.metricbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.fleet_server-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.heartbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.apm_server-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.apm_server-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.fleet_server-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.osquerybeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.metricbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.packetbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.endpoint_security-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.elastic_agent-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.heartbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.auditbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.endpoint_security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.filebeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.auditbeat-default
privileges:
- auto_configure
- create_doc
_elastic_agent_checks:
cluster:
- monitor
system-3:
indices:
- names:
- logs-system.auth-production
privileges:
- auto_configure
- create_doc
- names:
- logs-system.syslog-production
privileges:
- auto_configure
- create_doc
- names:
- logs-system.application-production
privileges:
- auto_configure
- create_doc
- names:
- logs-system.security-production
privileges:
- auto_configure
- create_doc
- names:
- logs-system.system-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.cpu-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.diskio-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.filesystem-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.fsstat-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.load-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.memory-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.network-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.process-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.process.summary-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.socket_summary-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.uptime-production
privileges:
- auto_configure
- create_doc
apm-server:
indices:
- names:
- logs-apm.app-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-apm.app.*-production
privileges:
- auto_configure
- create_doc
- names:
- logs-apm.error-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-apm.internal-production
privileges:
- auto_configure
- create_doc
- names:
- metrics-apm.profiling-production
privileges:
- auto_configure
- create_doc
- names:
- traces-apm.rum-production
privileges:
- auto_configure
- create_doc
- names:
- traces-apm.sampled-production
privileges:
- auto_configure
- create_doc
- maintenance
- monitor
- read
- names:
- traces-apm-production
privileges:
- auto_configure
- create_doc
cluster:
- 'cluster:monitor/main'
agent:
monitoring:
enabled: true
use_output: default
namespace: default
logs: true
metrics: true
inputs:
- id: logfile-system-bd988dae-3733-4715-81a2-2a6bb0a39505
name: system-3
revision: 2
type: logfile
use_output: default
meta:
package:
name: system
version: 1.11.0
data_stream:
namespace: production
streams:
- id: logfile-system.auth-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.auth
type: logs
exclude_files:
- .gz$
paths:
- /var/log/auth.log*
- /var/log/secure*
multiline:
pattern: ^\s
match: after
processors:
- add_locale: null
- id: logfile-system.syslog-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.syslog
type: logs
exclude_files:
- .gz$
paths:
- /var/log/messages*
- /var/log/syslog*
multiline:
pattern: ^\s
match: after
processors:
- add_locale: null
- id: winlog-system-bd988dae-3733-4715-81a2-2a6bb0a39505
name: system-3
revision: 2
type: winlog
use_output: default
meta:
package:
name: system
version: 1.11.0
data_stream:
namespace: production
streams:
- id: winlog-system.application-bd988dae-3733-4715-81a2-2a6bb0a39505
name: Application
data_stream:
dataset: system.application
type: logs
condition: '${host.platform} == ''windows'''
ignore_older: 72h
- id: winlog-system.security-bd988dae-3733-4715-81a2-2a6bb0a39505
name: Security
data_stream:
dataset: system.security
type: logs
condition: '${host.platform} == ''windows'''
ignore_older: 72h
- id: winlog-system.system-bd988dae-3733-4715-81a2-2a6bb0a39505
name: System
data_stream:
dataset: system.system
type: logs
condition: '${host.platform} == ''windows'''
ignore_older: 72h
- id: system/metrics-system-bd988dae-3733-4715-81a2-2a6bb0a39505
name: system-3
revision: 2
type: system/metrics
use_output: default
meta:
package:
name: system
version: 1.11.0
data_stream:
namespace: production
streams:
- id: system/metrics-system.cpu-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.cpu
type: metrics
period: 10s
cpu.metrics:
- percentages
- normalized_percentages
metricsets:
- cpu
- id: system/metrics-system.diskio-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.diskio
type: metrics
period: 10s
diskio.include_devices: null
metricsets:
- diskio
- id: system/metrics-system.filesystem-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.filesystem
type: metrics
period: 1m
metricsets:
- filesystem
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- id: system/metrics-system.fsstat-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.fsstat
type: metrics
period: 1m
metricsets:
- fsstat
processors:
- drop_event.when.regexp:
system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- id: system/metrics-system.load-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.load
type: metrics
condition: '${host.platform} != ''windows'''
period: 10s
metricsets:
- load
- id: system/metrics-system.memory-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.memory
type: metrics
period: 10s
metricsets:
- memory
- id: system/metrics-system.network-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.network
type: metrics
period: 10s
network.interfaces: null
metricsets:
- network
- id: system/metrics-system.process-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.process
type: metrics
process.include_top_n.by_memory: 5
period: 10s
processes:
- .*
process.include_top_n.by_cpu: 5
process.cgroups.enabled: false
process.cmdline.cache.enabled: true
metricsets:
- process
process.include_cpu_ticks: false
- id: >-
system/metrics-system.process.summary-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.process.summary
type: metrics
period: 10s
metricsets:
- process_summary
- id: >-
system/metrics-system.socket_summary-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.socket_summary
type: metrics
period: 10s
metricsets:
- socket_summary
- id: system/metrics-system.uptime-bd988dae-3733-4715-81a2-2a6bb0a39505
data_stream:
dataset: system.uptime
type: metrics
period: 10s
metricsets:
- uptime
- id: 318b41c2-4297-4f4e-973d-742376b03207
name: apm-server
revision: 4
type: apm
use_output: default
meta:
package:
name: apm
version: 8.2.0
data_stream:
namespace: production
apm-server:
capture_personal_data: true
max_connections: 0
max_event_size: 307200
auth:
api_key:
enabled: true
limit: 100
anonymous:
enabled: false
allow_agent:
- rum-js
- js-base
- iOS/swift
allow_service: null
rate_limit:
ip_limit: 1000
event_limit: 300
secret_token: >-
MzYyY2IyNzhkMjQwMGFmMjE0Y2NiZGI2MTMzZTc0ZDdkMzQyNWI0ZjZiZDNlZmUwZGFlMzVlOGQ4
M2E3MTQyZiAgLQo=
default_service_environment: null
shutdown_timeout: 30s
sampling:
tail:
enabled: false
policies:
- sample_rate: 0.1
interval: 1m
rum:
enabled: true
exclude_from_grouping: ^/webpack
allow_headers: null
response_headers: null
library_pattern: node_modules|bower_components|~
allow_origins:
- '*'
source_mapping:
metadata: []
ssl:
enabled: false
key_passphrase: null
certificate: null
supported_protocols:
- TLSv1.0
- TLSv1.1
- TLSv1.2
curve_types: null
key: null
cipher_suites: null
response_headers: null
write_timeout: 30s
host: 'MYIPADDRESS:8200'
max_header_size: 1048576
idle_timeout: 45s
expvar.enabled: false
read_timeout: 3600s
java_attacher:
enabled: false
discovery-rules: null
download-agent-version: null
agent_config:
- service:
name: ENVIRONMENT_ALL
etag: 84e01191de16e92ec80eb39d5dc3524fc396a46a
config:
transaction_max_spans: '500'
transaction_sample_rate: '1.0'
fleet:
hosts:
- 'https://MYIPADDRESS:8220'
- 'https://MYIPADDRESS:8220'
- 'https://MYIPADDRESS:8220'
- 'https://MYIPADDRESS:8220'
- 'https://MYIPADDRESS:8220'
- 'https://MYIPADDRESS:8220'
- 'https://localhost:8220'
- 'https://localhost:8220'
simitt
(Silvia Mitter)
May 31, 2022, 2:39pm
7
The policy shows that you have configured a custom namespace
for apm:
data_stream:
namespace: production
Unfortunately there was a bug in 8.2
, breaking ingestion for custom namespaces (apm-server#8087 ). The bug is fixed and the fix will be released with the next version. Until then, please switch back the namespace
to default
and data ingestion should work as expected again.
1 Like
kelvinferraz
(Kelvin Ferraz da Silva)
May 31, 2022, 3:20pm
8
This information is great to know.
How I can change this configuration to default?
Thanks agan.
simitt
(Silvia Mitter)
May 31, 2022, 5:38pm
9
When changing via UI you navigate to the elastic agent policy, click on the apm integration which will open it for editing. One of the first attributes is the namespace
. Remove the production
and replace by default
. Then save the changes.
By default the namespace is set to default
, so someone from your organization must have changed this. If they applied the change via automation or tooling, I suggest you directly update it there.
system
(system)
Closed
June 21, 2022, 1:39pm
10
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.