Need to set yyyy-dd-mm in apm-server prcessor event index

Kibana version: 7.17.23

Elasticsearch version:7.17.0

APM Server version: 7.17.20

Original install method (e.g. download page, yum, deb, from source, etc.) and version**: deb

Fresh install or upgraded from other version? Fresh

We have clone different ilm policies from apm-server-30-days and applied on transection, error, metric and span. rollover is working fine.

But we need to change index name pattern means need to add date in index name yyyy-mm-dd format.

for that we have configured in output elasticsarch block as below

index: "apm-%{[observer.version]}-{{type}}-%{+yyyy.MM.dd}"
indices:
# - index: "apm-%{[observer.version]}-sourcemap"
#   when.contains:
#     processor.event: "sourcemap"
#
 - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
   when.contains:
     processor.event: "error"
     setup.template.name: "apm-7.17.20-error"
     setup.template.pattern: "apm-7.17.20-error*"
#
 - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
   when.contains:
     processor.event: "transaction"
     setup.template.name: "apm-7.17.20-transaction"
     setup.template.pattern: "apm-7.17.20-transaction*"
#
 - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
   when.contains:
     processor.event: "span"
     setup.template.name: "apm-7.17.20-span"
     setup.template.pattern: "apm-7.17.20-span*"
#
 - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
   when.contains:
     processor.event: "metric"
     setup.template.name: "apm-7.17.20-metric"
     setup.template.pattern: "apm-7.17.20-metric*"

but we didn't get any result index still creating

apm-7.17.20-metric-000297
apm-7.17.20-error-000299
apm-7.17.20-span-000264
apm-7.17.20-transaction-000085

can any one help me how can I set date on index name?

Thanks in advanse

Hello Ekta, the configuration block does not look correct to me.

As per the documentation configuring the Elasticsearch output (docs) I think only one of output.elasticsearch.index or output.elasticsearch.indices must be set, but not both.

Looking at the docs for output.elasticsearch.index (docs) I think you just need to specify

output.elasticsearch:
  ### any other relevant option
  index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}" 

to achieve what you want, which is having the date in format YYYY-MM-DD in your indices names.

Please note that this configuration will disable Customize index lifecycle management. You will then be responsible to set it up on the new indices (looks like you already did from what you mention).

You will also need to set setup.template.name and setup.template.pattern to appropriate values (defaults are apm-server and apm-* which should work but you will need to specify them explicitly) (docs).

Hi @Edoardo_Tenani

we have configured as per your suggertions but still date is not appending

#================================= Template =================================
setup.template.enabled: true
setup.template.name: "apm-7.17.20"
setup.template.pattern: "apm-7.17.20-*"

output.elasticsearch:
hosts: ["TEST02:9200"]
enabled: true
protocol: "https"
username: "elastic"
password: "password"
index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"

can you please here?

Thanks in advance

Fresh install or upgraded from other version? Fresh

If this is a fresh install then any reason for not using the latest 8.x versions?

can you please here?

Can you check if you are getting any errors in the Stack Management -> Index Management page (should be accessible by URI app/management/data/index_management/indices on Kibana)? If there are any errors you should see a warning label with something like xx indices have lifecycle errors.

Also, could you try updating your config with:

setup.template.enabled: true
setup.template.name: "apm-%{[observer.version]}"
setup.template.pattern: "apm-%{[observer.version]}-*"
...
output.elasticsearch:
  ...
  index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"

@lahsivjar I have tried with same settings, and ten restarted apm-server service also but did not get result

Hello @Ekta, I tried a similar setup with custom "index" override in APM Server version 7.17.20 and I can confirm it works as expected.

To further debug your issue please provide a copy of your "apm-server.yaml" configuration file with all sensitive information stripped (to ensure that there is no subtle misconfiguration). Also please ensure that there is no related errors inside the APM Server logs, and when the APM Server is restarted the next INFO log lines are emitted.

...
{"log.level":"info","@timestamp":"2024-09-25T13:22:35.360-0700","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/manager.go","file.line":229},"message":"Finished template setup for apm-7.17.20-metric.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-25T13:22:35.513-0700","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/manager.go","file.line":229},"message":"Finished template setup for apm-7.17.20-span.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-25T13:22:35.513-0700","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/manager.go","file.line":130},"message":"Finished index management setup.","service.name":"apm-server","ecs.version":"1.6.0"}

@Kostya_Masliuk please find the apm-server yml as below

######################### APM Server Configuration #########################

################################ APM Server ################################

apm-server:
  # Defines the host and port the server is listening on. Use "unix:/path/to.sock" to listen on a unix domain socket.
  host: "TEST04:8200"

  # Agent authorization configuration. If no methods are defined, all requests will be allowed.
  #auth:
    # Agent authorization using Elasticsearch API Keys.
    #api_key:
      #enabled: false
      #
      # Restrict how many unique API keys are allowed per minute. Should be set to at least the amount of different
      # API keys configured in your monitored services. Every unique API key triggers one request to Elasticsearch.
      #limit: 100

    # Define a shared secret token for authorizing agents using the "Bearer" authorization method.
    #secret_token:

    # Allow anonymous access only for specified agents and/or services. This is primarily intended to allow
    # limited access for untrusted agents, such as Real User Monitoring.
    #anonymous:
      # By default anonymous auth is automatically enabled when either auth.api_key or
      # auth.secret_token is enabled, and RUM is enabled. Otherwise, anonymous auth is
      # disabled by default.
      #
      # When anonymous auth is enabled, only agents matching allow_agent and services
      # matching allow_service are allowed. See below for details on default values for
      # allow_agent.
      #enabled:

      # Allow anonymous access only for specified agents.
      #allow_agent: [rum-js, js-base]

      # Allow anonymous access only for specified service names. By default, all service names are allowed.
      #allow_service: []

      # Rate-limit anonymous access by IP and number of events.
      #rate_limit:
        # Rate limiting is defined per unique client IP address, for a limited number of IP addresses.
        # Sites with many concurrent clients should consider increasing this limit. Defaults to 1000.
        #ip_limit: 1000

        # Defines the maximum amount of events allowed per IP per second. Defaults to 300. The overall
        # maximum event throughput for anonymous access is (event_limit * ip_limit).
        #event_limit: 300

  # Maximum permitted size in bytes of a request's header accepted by the server to be processed.
  #max_header_size: 1048576

  # Maximum amount of time to wait for the next incoming request before underlying connection is closed.
  #idle_timeout: 45s

  # Maximum permitted duration for reading an entire request.
  #read_timeout: 30s

  # Maximum permitted duration for writing a response.
  #write_timeout: 30s

  # Maximum duration before releasing resources when shutting down the server.
  #shutdown_timeout: 5s

  # Maximum permitted size in bytes of an event accepted by the server to be processed.
  #max_event_size: 307200

  # Maximum number of new connections to accept simultaneously (0 means unlimited).
  #max_connections: 0

  # Custom HTTP headers to add to all HTTP responses, e.g. for security policy compliance.
  #response_headers:
  #  X-My-Header: Contents of the header

  # If true (default), APM Server captures the IP of the instrumented service
  # or the IP and User Agent of the real user (RUM requests).
  #capture_personal_data: true

  # If specified, APM Server will record this value in events which have no service environment
  # defined, and add it to agent configuration queries to Kibana when none is specified in the
  # request from the agent.
  #default_service_environment:

  # Enable APM Server Golang expvar support (https://golang.org/pkg/expvar/).
  #expvar:
    #enabled: false

    # Url to expose expvar.
    #url: "/debug/vars"

  # A pipeline is a definition of processors applied to documents when ingesting them to Elasticsearch.
  # Using pipelines involves two steps:
  # (1) registering a pipeline
  # (2) applying a pipeline during data ingestion (see `output.elasticsearch.pipeline`)
  #
  # You can manually register a pipeline, or use this configuration option to ensure
  # the pipeline is loaded and registered at the configured Elasticsearch instances.
  # Find the default pipeline configuration at `ingest/pipeline/definition.json`.
  # Automatic pipeline registration requires the `output.elasticsearch` to be enabled and configured.
  #register.ingest.pipeline:
    # Registers APM pipeline definition in Elasticsearch on APM Server startup. Defaults to true.
    #enabled: true
    # Overwrites existing APM pipeline definition in Elasticsearch. Defaults to false.
    #overwrite: false


  #---------------------------- APM Server - Secure Communication with Agents ----------------------------

  # Enable secure communication between APM agents and the server. By default ssl is disabled.
  ssl:
    enabled: true

    # Path to file containing the certificate for server authentication.
    # Needs to be configured when ssl is enabled.
    certificate: '/etc/apm-server/cert/certificate.crt'

    # Path to file containing server certificate key.
    # Needs to be configured when ssl is enabled.
    key: '/etc/apm-server/cert/certificate.key'

    # Optional configuration options for ssl communication.

    # Passphrase for decrypting the Certificate Key.
    # It is recommended to use the provided keystore instead of entering the passphrase in plain text.
    #key_passphrase: ''

    # List of supported/valid protocol versions. By default TLS versions 1.1 up to 1.3 are enabled.
    supported_protocols: [TLSv1.2]
    ##verification_mode: none
    ##certificate_authorities: ["/etc/apm-server/cert/certificate_ca.crt"]
    # Configure cipher suites to be used for SSL connections.
    # Note that cipher suites are not configurable for TLS 1.3.
    #cipher_suites: []

    # Configure curve types for ECDHE based cipher suites.
    #curve_types: []

  # The APM Server endpoints can be secured by configuring a secret token or enabling the usage of API keys. Both
  # options can be enabled in parallel, allowing Elastic APM agents to chose whichever mechanism they support.
  # As soon as one of the options is enabled, requests without a valid token are denied by the server. An exception
  # to this are requests to any enabled RUM endpoint. RUM endpoints are generally not secured by any token.
  #
  # Configure authorization via a common `secret_token`. By default it is disabled.
  # Agents include the token in the following format: Authorization: Bearer <secret-token>.
  # It is recommended to use an authorization token in combination with SSL enabled,
  # and save the token in the apm-server keystore.
  #
  # WARNING: This configuration is deprecated and replaced with `apm-server.auth.secret_token`, and will be removed
  # in the 8.0 release. If that config is defined, this one will be ignored.
  #secret_token:

  # Enable API key authorization by setting enabled to true. By default API key support is disabled.
  # Agents include a valid API key in the following format: Authorization: ApiKey <token>.
  # The key must be the base64 encoded representation of the API key's "id:key".
  #
  # WARNING: This configuration is deprecated and replaced with `apm-server.auth.api_key`, and will be removed
  # in the 8.0 release. If that config is defined, this one will be ignored.
  #api_key:
    #enabled: false

    # Restrict how many unique API keys are allowed per minute. Should be set to at least the amount of different
    # API keys configured in your monitored services. Every unique API key triggers one request to Elasticsearch.
    #limit: 100

    # API keys need to be fetched from Elasticsearch. If nothing is configured, configuration settings from the
    # output section will be reused.
    # Note that configuration needs to point to a secured Elasticsearch cluster that is able to serve API key requests.
    #elasticsearch:
      #hosts: ["localhost:9200"]

      #protocol: "http"

      # Username and password are only needed for the apm-server apikey sub-command, and they are ignored otherwise
      # See `apm-server apikey --help` for details.
      #username: "elastic"
      #password: "changeme"

      # Optional HTTP Path.
      #path: ""

      # Proxy server url.
      #proxy_url: ""
      #proxy_disable: false

      # Configure http request timeout before failing an request to Elasticsearch.
      #timeout: 5s

      # Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
      #ssl.enabled: true

      # Optional SSL configuration options. SSL is off by default, change the `protocol` option if you want to enable `https`.
      #
      # Control the verification of Elasticsearch certificates. Valid values are:
      # * full, which verifies that the provided certificate is signed by a trusted
      # authority (CA) and also verifies that the server's hostname (or IP address)
      # matches the names identified within the certificate.
      # * strict, which verifies that the provided certificate is signed by a trusted
      # authority (CA) and also verifies that the server's hostname (or IP address)
      # matches the names identified within the certificate. If the Subject Alternative
      # Name is empty, it returns an error.
      # * certificate, which verifies that the provided certificate is signed by a
      # trusted authority (CA), but does not perform any hostname verification.
      #  * none, which performs no verification of the server's certificate. This
      # mode disables many of the security benefits of SSL/TLS and should only be used
      # after very careful consideration. It is primarily intended as a temporary
      # diagnostic mechanism when attempting to resolve TLS errors; its use in
      # production environments is strongly discouraged.
      #ssl.verification_mode: full

      # List of supported/valid TLS versions. By default all TLS versions 1.0 up to
      # 1.2 are enabled.
      #ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

      # List of root certificates for HTTPS server verifications.
      #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

      # Certificate for SSL client authentication.
      #ssl.certificate: "/etc/pki/client/cert.pem"

      # Client Certificate Key
      #ssl.key: "/etc/pki/client/cert.key"

      # Optional passphrase for decrypting the Certificate Key.
      # It is recommended to use the provided keystore instead of entering the passphrase in plain text.
      #ssl.key_passphrase: ''

      # Configure cipher suites to be used for SSL connections.
      #ssl.cipher_suites: []

      # Configure curve types for ECDHE based cipher suites.
      #ssl.curve_types: []

      # Configure what types of renegotiation are supported. Valid options are
      # never, once, and freely. Default is never.
      #ssl.renegotiation: never


  #---------------------------- APM Server - RUM Real User Monitoring ----------------------------

  # Enable Real User Monitoring (RUM) Support. By default RUM is disabled.
  # RUM does not support token based authorization. Enabled RUM endpoints will not require any authorization
  # token configured for other endpoints.
  rum:
    enabled: true

    # Rate-limit RUM agents.
    #
    # WARNING: This configuration is deprecated and replaced with `apm-server.auth.anonymous.rate_limit`,
    # and will be removed in the 8.0 release. If that config is defined, this one will be ignored.
    #event_rate:

      # Defines the maximum amount of events allowed to be sent to the APM Server RUM
      # endpoint per IP per second. Defaults to 300.
      #limit: 300

      # An LRU cache is used to keep a rate limit per IP for the most recently seen IPs.
      # This setting defines the number of unique IPs that can be tracked in the cache.
      # Sites with many concurrent clients should consider increasing this limit. Defaults to 1000.
      #lru_size: 1000

    #-- General RUM settings

    # A list of service names to allow, to limit service-specific indices and data streams
    # created for unauthenticated RUM events.
    # If the list is empty, any service name is allowed.
    #
    # WARNING: This configuration is deprecated and replaced with `apm-server.auth.anonymous.allow_service`,
    # and will be removed in the 8.0 release. If that config is defined, this one will be ignored.
    #allow_service_names: []

    # A list of permitted origins for real user monitoring.
    # User-agents will send an origin header that will be validated against this list.
    # An origin is made of a protocol scheme, host and port, without the url path.
    # Allowed origins in this setting can have * to match anything (eg.: http://*.example.com)
    # If an item in the list is a single '*', everything will be allowed.
    #allow_origins: ['*']

    # A list of Access-Control-Allow-Headers to allow RUM requests, in addition to "Content-Type",
    # "Content-Encoding", and "Accept"
    #allow_headers: []

    # Custom HTTP headers to add to RUM responses, e.g. for security policy compliance.
    #response_headers:
    #  X-My-Header: Contents of the header

    # Regexp to be matched against a stacktrace frame's `file_name` and `abs_path` attributes.
    # If the regexp matches, the stacktrace frame is considered to be a library frame.
    #library_pattern: "node_modules|bower_components|~"

    # Regexp to be matched against a stacktrace frame's `file_name`.
    # If the regexp matches, the stacktrace frame is not used for calculating error groups.
    # The default pattern excludes stacktrace frames that have a filename starting with '/webpack'
    #exclude_from_grouping: "^/webpack"

    # If a source map has previously been uploaded, source mapping is automatically applied.
    # to all error and transaction documents sent to the RUM endpoint.
    #source_mapping:

      # Sourcemapping is enabled by default.
      #enabled: true

      # Timeout for fetching source maps.
      #timeout: 5s

      # Source maps may be fetched from Elasticsearch by using the
      # output.elasticsearch configuration.
      # A different instance must be configured when using any other output.
      # This setting only affects sourcemap reads - the output determines where sourcemaps are written.
      # Note: Configuring elasticsearch is not supported if apm-server is being
      # managed by Fleet.
      #elasticsearch:
        # Array of hosts to connect to.
        # Scheme and port can be left out and will be set to the default (`http` and `9200`).
        # In case you specify and additional path, the scheme is required: `http://localhost:9200/path`.
        # IPv6 addresses should always be defined as: `https://[2001:db8::1]:9200`.
        # hosts: ["localhost:9200"]

        # Protocol - either `http` (default) or `https`.
        #protocol: "https"

        # Authentication credentials - either API key or username/password.
        #api_key: "id:api_key"
        #username: "elastic"
        #password: "changeme"

      # The `cache.expiration` determines how long a source map should be cached before fetching it again from Elasticsearch.
      # Note that values configured without a time unit will be interpreted as seconds.
      #cache:
        #expiration: 5m

      # Source maps are stored in a separate index.
      # If the default index pattern for source maps at 'outputs.elasticsearch.indices'
      # is changed, a matching index pattern needs to be specified here.
      #index_pattern: "apm-*-sourcemap*"

  #---------------------------- APM Server - Agent Configuration ----------------------------

  # When using APM agent configuration, information fetched from Kibana will be cached in memory for some time.
  # Specify cache key expiration via this setting. Default is 30 seconds.
  #agent.config.cache.expiration: 30s

  #kibana:
    # For APM Agent configuration in Kibana, enabled must be true.
    #enabled: false

    # Scheme and port can be left out and will be set to the default (`http` and `5601`).
    # In case you specify an additional path, the scheme is required: `http://localhost:5601/path`.
    # IPv6 addresses should always be defined as: `https://[2001:db8::1]:5601`.
    #host: "localhost:5601"

    # Optional protocol and basic auth credentials.
    #protocol: "https"
    #username: "elastic"
    #password: "changeme"

    # Optional HTTP path.
    #path: ""

    # Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
    #ssl.enabled: true

    # Optional SSL configuration options. SSL is off by default, change the `protocol` option if you want to enable `https`.
    #
    # Control the verification of Kibana certificates. Valid values are:
    # * full, which verifies that the provided certificate is signed by a trusted
    # authority (CA) and also verifies that the server's hostname (or IP address)
    # matches the names identified within the certificate.
    # * strict, which verifies that the provided certificate is signed by a trusted
    # authority (CA) and also verifies that the server's hostname (or IP address)
    # matches the names identified within the certificate. If the Subject Alternative
    # Name is empty, it returns an error.
    # * certificate, which verifies that the provided certificate is signed by a
    # trusted authority (CA), but does not perform any hostname verification.
    #  * none, which performs no verification of the server's certificate. This
    # mode disables many of the security benefits of SSL/TLS and should only be used
    # after very careful consideration. It is primarily intended as a temporary
    # diagnostic mechanism when attempting to resolve TLS errors; its use in
    # production environments is strongly discouraged.
    #ssl.verification_mode: full

    # List of supported/valid TLS versions. By default all TLS versions 1.0 up to
    # 1.2 are enabled.
    #ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

    # List of root certificates for HTTPS server verifications.
    #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

    # Certificate for SSL client authentication.
    #ssl.certificate: "/etc/pki/client/cert.pem"

    # Client Certificate Key
    #ssl.key: "/etc/pki/client/cert.key"

    # Optional passphrase for decrypting the Certificate Key.
    # It is recommended to use the provided keystore instead of entering the passphrase in plain text.
    #ssl.key_passphrase: ''

    # Configure cipher suites to be used for SSL connections.
    #ssl.cipher_suites: []

    # Configure curve types for ECDHE based cipher suites.
    #ssl.curve_types: []

  #---------------------------- APM Server - ILM Index Lifecycle Management ----------------------------

  #ilm:
    # Supported values are `auto`, `true` and `false`.
    # `true`: Make use of Elasticsearch's Index Lifecycle Management (ILM) for APM indices. If no Elasticsearch output is
    # configured or the configured instance does not support ILM, APM Server cannot apply ILM and must create
    # unmanaged indices instead.
    # `false`: APM Server does not make use of ILM in Elasticsearch.
    # `auto`: If an Elasticsearch output is configured with default index and indices settings, and the configured
    # Elasticsearch instance supports ILM, `auto` will resolve to `true`. Otherwise `auto` will resolve to `false`.
    # Default value is `auto`.
    #enabled: "auto"

    #setup:
      # Only disable setup if you want to set up everything related to ILM on your own.
      # When setup is enabled, the APM Server creates:
      # - aliases and ILM policies if `apm-server.ilm.enabled` resolves to `true`.
      # - An ILM specific template per event type. This is required to map ILM aliases and policies to indices. In case
      # ILM is disabled, the templates will be created without any ILM settings.
      # Be aware that if you turn off setup, you need to manually manage event type specific templates on your own.
      # If you simply want to disable ILM, use the above setting, `apm-server.ilm.enabled`, instead.
      # Defaults to true.
      #enabled: true

      # Configure whether or not existing policies and ILM related templates should be updated. This needs to be
      # set to true when customizing your policies.
      # Defaults to false.
      #overwrite: false

      # Set `require_policy` to `false` when policies are set up outside of APM Server but referenced here.
      # Default value is `true`.
      #require_policy: true

      # Customized mappings will be merged with the default setup, so you only need to configure mappings for the
      # event types, policies, and index suffixes that you want to customize.
      # Indices are named in this way: `apm-%{[observer.version]}-%{[event.type]}-{index_suffix}`,
      # e.g., apm-7.9.0-span-custom*. The `index_suffix` is optional.
      # NOTE: When configuring an `index_suffix`, ensure that no previously set up templates conflict with the
      #       newly configured ones. If an index matches multiple templates with the same order, the settings of
      #       the templates will override each other. Any conflicts need to be cleaned up manually.
      # NOTE: When customizing `setup.template.name` and `setup.template.pattern`, ensure they still match the indices.
      #mapping:
        #- event_type: "error"
        #  policy_name: "apm-rollover-30-days"
        #  index_suffix: ""
        #- event_type: "span"
        #  policy_name: "apm-rollover-30-days"
        #  index_suffix: ""
        #- event_type: "transaction"
        #  policy_name: "apm-rollover-30-days"
        #  index_suffix: ""
        #- event_type: "metric"
        #  policy_name: "apm-rollover-30-days"
        #  index_suffix: ""

      # Configured policies are added to pre-defined default policies.
      # If a policy with the same name as a default policy is configured, the configured policy overwrites the default policy.
      #policies:
        #- name: "apm-rollover-30-days"
          #policy:
            #phases:
              #hot:
                #actions:
                  #rollover:
                    #max_size: "50gb"
                    #max_age: "30d"
                  #set_priority:
                    #priority: 100



  #---------------------------- APM Server - Experimental Jaeger integration ----------------------------

  # When enabling Jaeger integration, APM Server acts as Jaeger collector. It supports jaeger.thrift over HTTP
  # and gRPC. This is an experimental feature, use with care.
  #
  # WARNING: This configuration is deprecated, and will be removed in the 8.0 release.
  #
  # Jaeger gRPC is now served on the same port as Elastic APM agents, defined by the
  # "apm-server.host" configuration; it is implicitly enabled, and an agent tag called
  # "elastic-apm-auth" is required when auth is enabled.
  #jaeger:
    #grpc:
      # Set to true to enable the Jaeger gRPC collector service.
      #enabled: false

      # Defines the gRPC host and port the server is listening on.
      # Defaults to the standard Jaeger gRPC collector port 14250.
      #host: "localhost:14250"

      # Set to the name of a process tag to use for authorizing
      # Jaeger agents.
      #
      # The tag value should have the same format as an HTTP
      # Authorization header, i.e. "Bearer <secret_token>" or
      # "ApiKey <base64(id:key)>".
      #
      # By default (if the auth_tag value is empty), authorization
      # does not apply to Jaeger agents.
      #auth_tag: ""

    #http:
      # Set to true to enable the Jaeger HTTP collector endpoint.
      #enabled: false

      # Defines the HTTP host and port the server is listening on.
      # Defaults to the standard Jaeger HTTP collector port 14268.
      #host: "localhost:14268"

#================================= General =================================

# Data is buffered in a memory queue before it is published to the configured output.
# The memory queue will present all available events (up to the outputs
# bulk_max_size) to the output, the moment the output is ready to serve
# another batch of events.
#queue:
  # Queue type by name (default 'mem').
  #mem:
    # Max number of events the queue can buffer.
    #events: 4096

    # Hints the minimum number of events stored in the queue,
    # before providing a batch of events to the outputs.
    # The default value is set to 2048.
    # A value of 0 ensures events are immediately available
    # to be sent to the outputs.
    #flush.min_events: 2048

    # Maximum duration after which events are available to the outputs,
    # if the number of events stored in the queue is < `flush.min_events`.
    #flush.timeout: 1s

# Sets the maximum number of CPUs that can be executing simultaneously. The
# default is the number of logical CPUs available in the system.
#max_procs:

#================================= Template =================================

# A template is used to set the mapping in Elasticsearch.
# By default template loading is enabled and the template is loaded.
# These settings can be adjusted to load your own template or overwrite existing ones.

# Set to false to disable template loading.
setup.template.enabled: true

# Template name. By default the template name is "apm-%{[observer.version]}"
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.name: "apm-%{[observer.version]}"

# Template pattern. By default the template pattern is "apm-%{[observer.version]}-*" to apply to the default index settings.
# The first part is the version of apm-server and then -* is used to match all daily indices.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.pattern: "apm-%{[observer.version]}-*"

# Path to fields.yml file to generate the template.
#setup.template.fields: "${path.config}/fields.yml"

# Overwrite existing template.
#setup.template.overwrite: false

# Elasticsearch template settings.
#setup.template.settings:

  # A dictionary of settings to place into the settings.index dictionary
  # of the Elasticsearch template. For more details, please check
  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
  #index:
    #number_of_shards: 1
    #codec: best_compression
    #number_of_routing_shards: 30
    #mapping.total_fields.limit: 2000

#============================= Elastic Cloud =============================

# These settings simplify using APM Server with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` option.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =================================

# Configure the output to use when sending the data collected by apm-server.

#-------------------------- Elasticsearch output --------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (`http` and `9200`).
  # In case you specify and additional path, the scheme is required: `http://localhost:9200/path`.
  # IPv6 addresses should always be defined as: `https://[2001:db8::1]:9200`.
  hosts: ["TEST04:9200"]

  # Boolean flag to enable or disable the output module.
  #enabled: true

  # Set gzip compression level.
  #compression_level: 0

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "changeme"

  # Dictionary of HTTP parameters to pass within the url with index operations.
  #parameters:
    #param1: value1
    #param2: value2

  # Number of workers per Elasticsearch host.
  #worker: 1

  # By using the configuration below, APM documents are stored to separate indices,
  # depending on their `processor.event`:
  # - error
  # - transaction
  # - span
  # - sourcemap
  #
  # The indices are all prefixed with `apm-%{[observer.version]}`.
  # To allow managing indices based on their age, all indices (except for sourcemaps)
  # end with the information of the day they got indexed.
  # e.g. "apm-7.3.0-transaction-2019.07.20"
  #
  # Be aware that you can only specify one Elasticsearch template.
  # If you modify the index patterns you must also update these configurations accordingly,
  # as they need to be aligned:
  # * `setup.template.name`
  # * `setup.template.pattern`
  #index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
  index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"
  #indices:
  #  - index: "apm-%{[observer.version]}-sourcemap"
  #    when.contains:
  #      processor.event: "sourcemap"
  #
  #  - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "error"
  #
  #  - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "transaction"
  #
  #  - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "span"
  #
  #  - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "metric"
  #
  #  - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "onboarding"

  # A pipeline is a definition of processors applied to documents when ingesting them to Elasticsearch.
  # APM Server comes with a default pipeline definition, located at `ingest/pipeline/definition.json`, which is
  # loaded to Elasticsearch by default (see `apm-server.register.ingest.pipeline`).
  # APM pipeline is enabled by default. To disable it, set `pipeline: _none`.
  #pipeline: "apm"

  # Optional HTTP Path.
  #path: "/elasticsearch"

  # Custom HTTP headers to add to each request.
  #headers:
  #  X-My-Header: Contents of the header

  # Proxy server url.
  #proxy_url: http://proxy:3128

  # The number of times a particular Elasticsearch index operation is attempted. If
  # the indexing operation doesn't succeed after this many retries, the events are
  # dropped. The default is 3.
  #max_retries: 3

  # The maximum number of events to bulk in a single Elasticsearch bulk API index request.
  # The default is 50.
  #bulk_max_size: 50

  # The number of seconds to wait before trying to reconnect to Elasticsearch
  # after a network error. After waiting backoff.init seconds, apm-server
  # tries to reconnect. If the attempt fails, the backoff timer is increased
  # exponentially up to backoff.max. After a successful connection, the backoff
  # timer is reset. The default is 1s.
  #backoff.init: 1s

  # The maximum number of seconds to wait before attempting to connect to
  # Elasticsearch after a network error. The default is 60s.
  #backoff.max: 60s

  # Configure http request timeout before failing an request to Elasticsearch.
  #timeout: 90

  # Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
  ssl.enabled: true

  # Optional SSL configuration options. SSL is off by default, change the `protocol` option if you want to enable `https`.
  #
  # Control the verification of Elasticsearch certificates. Valid values are:
  # * full, which verifies that the provided certificate is signed by a trusted
  # authority (CA) and also verifies that the server's hostname (or IP address)
  # matches the names identified within the certificate.
  # * strict, which verifies that the provided certificate is signed by a trusted
  # authority (CA) and also verifies that the server's hostname (or IP address)
  # matches the names identified within the certificate. If the Subject Alternative
  # Name is empty, it returns an error.
  # * certificate, which verifies that the provided certificate is signed by a
  # trusted authority (CA), but does not perform any hostname verification.
  #  * none, which performs no verification of the server's certificate. This
  # mode disables many of the security benefits of SSL/TLS and should only be used
  # after very careful consideration. It is primarily intended as a temporary
  # diagnostic mechanism when attempting to resolve TLS errors; its use in
  # production environments is strongly discouraged.
  ssl.verification_mode: none

  # List of supported/valid TLS versions. By default all TLS versions 1.0 up to
  # 1.2 are enabled.
  ssl.supported_protocols: [TLSv1.2]

  # List of root certificates for HTTPS server verifications.
  ssl.certificate_authorities: ["/etc/apm-server/cert/certificate_ca.crt"]

  # Certificate for SSL client authentication.
  ssl.certificate: "/etc/apm-server/cert/certificate.crt"

  # Client Certificate Key
  ssl.key: "/etc/apm-server/cert/certificate.key"

  # Optional passphrase for decrypting the Certificate Key.
  # It is recommended to use the provided keystore instead of entering the passphrase in plain text.
  #ssl.key_passphrase: ''

  # Configure cipher suites to be used for SSL connections.
  #ssl.cipher_suites: []

  # Configure curve types for ECDHE based cipher suites.
  #ssl.curve_types: []

  # Configure what types of renegotiation are supported. Valid options are
  # never, once, and freely. Default is never.
  #ssl.renegotiation: never

  # Enable Kerberos support. Kerberos is automatically enabled if any Kerberos setting is set.
  #kerberos.enabled: true

  # Authentication type to use with Kerberos. Available options: keytab, password.
  #kerberos.auth_type: password

  # Path to the keytab file. It is used when auth_type is set to keytab.
  #kerberos.keytab: /etc/elastic.keytab

  # Path to the Kerberos configuration.
  #kerberos.config_path: /etc/krb5.conf

  # Name of the Kerberos user.
  #kerberos.username: elastic

  # Password of the Kerberos user. It is used when auth_type is set to password.
  #kerberos.password: changeme

  # Kerberos realm.
  #kerberos.realm: ELASTIC

@Ekta thank you for sharing your full apm-server.yml configuration file, because of the text formatting it's really hard to validate it accurately. Could you please reformat as "preformatted text" with ``` ?

Additionally please provide the output of the next queries to validate the state of your indices and index templates.


GET /_template/apm-7*

GET /_cat/indices/apm-7*

Hi @Kostya_Masliuk

please find indices detail in below

GET /_cat/indices/apm-7*

green open apm-7.17.20-error-000001 s2T1xAWwSema2iiHwcLabg 1 0 0 0 226b 226b
green open apm-7.17.20-span-000001 ChkAh8GdSWaDflaQWxIP7g 1 0 0 0 226b 226b
green open apm-7.17.20-transaction-000001 NVh5-7t_RpCf0FnBYDkh-g 1 0 0 0 226b 226b
green open apm-7.17.20-onboarding-2024.09.24 6P1VQfpQQuu7YUaHJDKFMw 1 0 5 0 42.1kb 42.1kb
green open apm-7.17.20-profile-000001 cQh-9MRpRNmXOAz6dEEpTQ 1 0 0 0 226b 226b
green open apm-7.17.20-metric-000001 wn4PuWDnS9m4CnzK8OYr6A 1 0 0 0 226b 226b

Please find template detail in below
apm-7.17.20
apm-7.17.20-error
apm-7.17.20-metric
apm-7.17.20-profile
apm-7.17.20-span
apm-7.17.20-transaction

Looking at the list of provided indices there is an index with the correct naming:

green open apm-7.17.20-onboarding-2024.09.24 6P1VQfpQQuu7YUaHJDKFMw 1 0 5 0 42.1kb 42.1kb

We have also been able to have a working setup based on the configuration you shared.

May you verify that you are successfully ingesting data and indexing does not show any errors? It may also help if you could perform a test from scratch (with no previously created indices).

######################### APM Server Configuration #########################

################################ APM Server ################################

apm-server:
"# Defines the host and port the server is listening on. Use "unix:/path/to.sock" to listen on a unix domain socket.
host: "TEST04:8200"

"# Agent authorization configuration. If no methods are defined, all requests will be allowed.
#auth:
# Agent authorization using Elasticsearch API Keys.
#api_key:
#enabled: false
#
# Restrict how many unique API keys are allowed per minute. Should be set to at least the amount of different
# API keys configured in your monitored services. Every unique API key triggers one request to Elasticsearch.
#limit: 100
"# Define a shared secret token for authorizing agents using the "Bearer" authorization method.
#secret_token:

# Allow anonymous access only for specified agents and/or services. This is primarily intended to allow
# limited access for untrusted agents, such as Real User Monitoring.
#anonymous:
  # By default anonymous auth is automatically enabled when either auth.api_key or
  # auth.secret_token is enabled, and RUM is enabled. Otherwise, anonymous auth is
  # disabled by default.
  #
  # When anonymous auth is enabled, only agents matching allow_agent and services
  # matching allow_service are allowed. See below for details on default values for
  # allow_agent.
  #enabled:

  # Allow anonymous access only for specified agents.
  #allow_agent: [rum-js, js-base]

  # Allow anonymous access only for specified service names. By default, all service names are allowed.
  #allow_service: []

  # Rate-limit anonymous access by IP and number of events.
  #rate_limit:
    # Rate limiting is defined per unique client IP address, for a limited number of IP addresses.
    # Sites with many concurrent clients should consider increasing this limit. Defaults to 1000.
    #ip_limit: 1000

    # Defines the maximum amount of events allowed per IP per second. Defaults to 300. The overall
   "# maximum event throughput for anonymous access is (event_limit * ip_limit).
    #event_limit: 300

"# Maximum permitted size in bytes of a request's header accepted by the server to be processed.
#max_header_size: 1048576

"# Maximum amount of time to wait for the next incoming request before underlying connection is closed.
#idle_timeout: 45s

"# Maximum permitted duration for reading an entire request.
#read_timeout: 30s

"# Maximum permitted duration for writing a response.
#write_timeout: 30s

"# Maximum duration before releasing resources when shutting down the server.
#shutdown_timeout: 5s

"# Maximum permitted size in bytes of an event accepted by the server to be processed.
#max_event_size: 307200

"# Maximum number of new connections to accept simultaneously (0 means unlimited).
#max_connections: 0

'# Custom HTTP headers to add to all HTTP responses, e.g. for security policy compliance.
'#response_headers:
'# X-My-Header: Contents of the header

'# If true (default), APM Server captures the IP of the instrumented service
'# or the IP and User Agent of the real user (RUM requests).
#capture_personal_data: true

'# If specified, APM Server will record this value in events which have no service environment
' # defined, and add it to agent configuration queries to Kibana when none is specified in the
'# request from the agent.
#default_service_environment:

'# Enable APM Server Golang expvar support (expvar package - expvar - Go Packages).
#expvar:
#enabled: false

# Url to expose expvar.
#url: "/debug/vars"

'# A pipeline is a definition of processors applied to documents when ingesting them to Elasticsearch.
' # Using pipelines involves two steps:
'# (1) registering a pipeline
'# (2) applying a pipeline during data ingestion (see output.elasticsearch.pipeline)

'# You can manually register a pipeline, or use this configuration option to ensure
'# the pipeline is loaded and registered at the configured Elasticsearch instances.
'# Find the default pipeline configuration at ingest/pipeline/definition.json.
'# Automatic pipeline registration requires the output.elasticsearch to be enabled and configured.
#register.ingest.pipeline:
# Registers APM pipeline definition in Elasticsearch on APM Server startup. Defaults to true.
#enabled: true
# Overwrites existing APM pipeline definition in Elasticsearch. Defaults to false.
#overwrite: false

#---------------------------- APM Server - Secure Communication with Agents ----------------------------

'# Enable secure communication between APM agents and the server. By default ssl is disabled.
ssl:
enabled: true

# Path to file containing the certificate for server authentication.
# Needs to be configured when ssl is enabled.
certificate: '/etc/apm-server/cert/certificate.crt'

# Path to file containing server certificate key.
# Needs to be configured when ssl is enabled.
key: '/etc/apm-server/cert/certificate.key'

# Optional configuration options for ssl communication.

# Passphrase for decrypting the Certificate Key.
# It is recommended to use the provided keystore instead of entering the passphrase in plain text.
#key_passphrase: ''

# List of supported/valid protocol versions. By default TLS versions 1.1 up to 1.3 are enabled.
supported_protocols: [TLSv1.2]
##verification_mode: none
##certificate_authorities: ["/etc/apm-server/cert/certificate_ca.crt"]
# Configure cipher suites to be used for SSL connections.
# Note that cipher suites are not configurable for TLS 1.3.
#cipher_suites: []

# Configure curve types for ECDHE based cipher suites.
#curve_types: []

'# The APM Server endpoints can be secured by configuring a secret token or enabling the usage of API keys. Both
'# options can be enabled in parallel, allowing Elastic APM agents to chose whichever mechanism they support.
'# As soon as one of the options is enabled, requests without a valid token are denied by the server. An exception
'# to this are requests to any enabled RUM endpoint. RUM endpoints are generally not secured by any token.

'# Configure authorization via a common secret_token. By default it is disabled.
'# Agents include the token in the following format: Authorization: Bearer .
' # It is recommended to use an authorization token in combination with SSL enabled,
'# and save the token in the apm-server keystore.

'# WARNING: This configuration is deprecated and replaced with apm-server.auth.secret_token, and will be removed
'# in the 8.0 release. If that config is defined, this one will be ignored.
#secret_token:

' # Enable API key authorization by setting enabled to true. By default API key support is disabled.
' # Agents include a valid API key in the following format: Authorization: ApiKey .
'# The key must be the base64 encoded representation of the API key's "id:key".

'# WARNING: This configuration is deprecated and replaced with apm-server.auth.api_key, and will be removed
'# in the 8.0 release. If that config is defined, this one will be ignored.
#api_key:
#enabled: false

# Restrict how many unique API keys are allowed per minute. Should be set to at least the amount of different
# API keys configured in your monitored services. Every unique API key triggers one request to Elasticsearch.
#limit: 100

# API keys need to be fetched from Elasticsearch. If nothing is configured, configuration settings from the
# output section will be reused.
# Note that configuration needs to point to a secured Elasticsearch cluster that is able to serve API key requests.
#elasticsearch:
  #hosts: ["localhost:9200"]

  #protocol: "http"

  # Username and password are only needed for the apm-server apikey sub-command, and they are ignored otherwise
  # See `apm-server apikey --help` for details.
  #username: "elastic"
  #password: "changeme"

  # Optional HTTP Path.
  #path: ""

  # Proxy server url.
  #proxy_url: ""
  #proxy_disable: false

  # Configure http request timeout before failing an request to Elasticsearch.
  #timeout: 5s

  # Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
  #ssl.enabled: true

  '# Optional SSL configuration options. SSL is off by default, change the `protocol` option if you want to enable `https`.
  #
  # Control the verification of Elasticsearch certificates. Valid values are:
  # * full, which verifies that the provided certificate is signed by a trusted
  # authority (CA) and also verifies that the server's hostname (or IP address)
  # matches the names identified within the certificate.
  # * strict, which verifies that the provided certificate is signed by a trusted
  # authority (CA) and also verifies that the server's hostname (or IP address)
  # matches the names identified within the certificate. If the Subject Alternative
  # Name is empty, it returns an error.
  # * certificate, which verifies that the provided certificate is signed by a
  # trusted authority (CA), but does not perform any hostname verification.
  #  * none, which performs no verification of the server's certificate. This
  # mode disables many of the security benefits of SSL/TLS and should only be used
  # after very careful consideration. It is primarily intended as a temporary
  # diagnostic mechanism when attempting to resolve TLS errors; its use in
  # production environments is strongly discouraged.
  #ssl.verification_mode: full

  # List of supported/valid TLS versions. By default all TLS versions 1.0 up to
  # 1.2 are enabled.
  #ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

  # List of root certificates for HTTPS server verifications.
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication.
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

  # Optional passphrase for decrypting the Certificate Key.
  # It is recommended to use the provided keystore instead of entering the passphrase in plain text.
  #ssl.key_passphrase: ''

  # Configure cipher suites to be used for SSL connections.
  #ssl.cipher_suites: []

  # Configure curve types for ECDHE based cipher suites.
  #ssl.curve_types: []

  # Configure what types of renegotiation are supported. Valid options are
  # never, once, and freely. Default is never.
  #ssl.renegotiation: never

#---------------------------- APM Server - RUM Real User Monitoring ----------------------------

' # Enable Real User Monitoring (RUM) Support. By default RUM is disabled.
'# RUM does not support token based authorization. Enabled RUM endpoints will not require any authorization
'# token configured for other endpoints.
rum:
enabled: true

# Rate-limit RUM agents.
#
# WARNING: This configuration is deprecated and replaced with `apm-server.auth.anonymous.rate_limit`,
# and will be removed in the 8.0 release. If that config is defined, this one will be ignored.
#event_rate:

  # Defines the maximum amount of events allowed to be sent to the APM Server RUM
  # endpoint per IP per second. Defaults to 300.
  #limit: 300

  # An LRU cache is used to keep a rate limit per IP for the most recently seen IPs.
  # This setting defines the number of unique IPs that can be tracked in the cache.
  # Sites with many concurrent clients should consider increasing this limit. Defaults to 1000.
  #lru_size: 1000

#-- General RUM settings

# A list of service names to allow, to limit service-specific indices and data streams
# created for unauthenticated RUM events.
# If the list is empty, any service name is allowed.
#
# WARNING: This configuration is deprecated and replaced with `apm-server.auth.anonymous.allow_service`,
# and will be removed in the 8.0 release. If that config is defined, this one will be ignored.
#allow_service_names: []

'# A list of permitted origins for real user monitoring.
'# User-agents will send an origin header that will be validated against this list.
'# An origin is made of a protocol scheme, host and port, without the url path.
'# Allowed origins in this setting can have * to match anything (eg.: http://*.example.com)

' # If an item in the list is a single '', everything will be allowed.
#allow_origins: ['
']

'# A list of Access-Control-Allow-Headers to allow RUM requests, in addition to "Content-Type",
# "Content-Encoding", and "Accept"
#allow_headers: []

# Custom HTTP headers to add to RUM responses, e.g. for security policy compliance.
#response_headers:
#  X-My-Header: Contents of the header

# Regexp to be matched against a stacktrace frame's `file_name` and `abs_path` attributes.
# If the regexp matches, the stacktrace frame is considered to be a library frame.
#library_pattern: "node_modules|bower_components|~"

# Regexp to be matched against a stacktrace frame's `file_name`.
# If the regexp matches, the stacktrace frame is not used for calculating error groups.
# The default pattern excludes stacktrace frames that have a filename starting with '/webpack'
#exclude_from_grouping: "^/webpack"

# If a source map has previously been uploaded, source mapping is automatically applied.
# to all error and transaction documents sent to the RUM endpoint.
#source_mapping:

 ' # Sourcemapping is enabled by default.
  #enabled: true

 ' # Timeout for fetching source maps.
  #timeout: 5s

  '# Source maps may be fetched from Elasticsearch by using the
  '# output.elasticsearch configuration.
  '# A different instance must be configured when using any other output.
  '# This setting only affects sourcemap reads - the output determines where sourcemaps are written.
 ' # Note: Configuring elasticsearch is not supported if apm-server is being
  '# managed by Fleet.
  #elasticsearch:
    # Array of hosts to connect to.
    '# Scheme and port can be left out and will be set to the default (`http` and `9200`).
    '# In case you specify and additional path, the scheme is required: `http://localhost:9200/path`.
   ' # IPv6 addresses should always be defined as: `https://[2001:db8::1]:9200`.
    # hosts: ["localhost:9200"]

    # Protocol - either `http` (default) or `https`.
    #protocol: "https"

    '# Authentication credentials - either API key or username/password.
    #api_key: "id:api_key"
    #username: "elastic"
    #password: "changeme"

  '# The `cache.expiration` determines how long a source map should be cached before fetching it again from Elasticsearch.
  '# Note that values configured without a time unit will be interpreted as seconds.
  #cache:
    #expiration: 5m

'  # Source maps are stored in a separate index.
 ' # If the default index pattern for source maps at 'outputs.elasticsearch.indices'
 ' # is changed, a matching index pattern needs to be specified here.
  #index_pattern: "apm-*-sourcemap*"

#---------------------------- APM Server - Agent Configuration ----------------------------

'# When using APM agent configuration, information fetched from Kibana will be cached in memory for some time.
' # Specify cache key expiration via this setting. Default is 30 seconds.
#agent.config.cache.expiration: 30s

#kibana:
'# For APM Agent configuration in Kibana, enabled must be true.
#enabled: false

'# Scheme and port can be left out and will be set to the default (`http` and `5601`).
'# In case you specify an additional path, the scheme is required: `http://localhost:5601/path`.
'# IPv6 addresses should always be defined as: `https://[2001:db8::1]:5601`.
#host: "localhost:5601"

# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"

# Optional HTTP path.
#path: ""

# Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
#ssl.enabled: true

# Optional SSL configuration options. SSL is off by default, change the `protocol` option if you want to enable `https`.
#
# Control the verification of Kibana certificates. Valid values are:
# * full, which verifies that the provided certificate is signed by a trusted
# authority (CA) and also verifies that the server's hostname (or IP address)
# matches the names identified within the certificate.
# * strict, which verifies that the provided certificate is signed by a trusted
# authority (CA) and also verifies that the server's hostname (or IP address)
# matches the names identified within the certificate. If the Subject Alternative
# Name is empty, it returns an error.
# * certificate, which verifies that the provided certificate is signed by a
# trusted authority (CA), but does not perform any hostname verification.
#  * none, which performs no verification of the server's certificate. This
# mode disables many of the security benefits of SSL/TLS and should only be used
# after very careful consideration. It is primarily intended as a temporary
# diagnostic mechanism when attempting to resolve TLS errors; its use in
# production environments is strongly discouraged.
#ssl.verification_mode: full

' # List of supported/valid TLS versions. By default all TLS versions 1.0 up to
' # 1.2 are enabled.
#ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

'# List of root certificates for HTTPS server verifications.
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

'# Certificate for SSL client authentication.
#ssl.certificate: "/etc/pki/client/cert.pem"

'# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

'# Optional passphrase for decrypting the Certificate Key.
'# It is recommended to use the provided keystore instead of entering the passphrase in plain text.
#ssl.key_passphrase: ''

' # Configure cipher suites to be used for SSL connections.
#ssl.cipher_suites:

'# Configure curve types for ECDHE based cipher suites.
#ssl.curve_types: []

#---------------------------- APM Server - ILM Index Lifecycle Management ----------------------------

#ilm:
'# Supported values are auto, true and false.
' # true: Make use of Elasticsearch's Index Lifecycle Management (ILM) for APM indices. If no Elasticsearch output is
' # configured or the configured instance does not support ILM, APM Server cannot apply ILM and must create
# unmanaged indices instead.
'# false: APM Server does not make use of ILM in Elasticsearch.
'# auto: If an Elasticsearch output is configured with default index and indices settings, and the configured
'# Elasticsearch instance supports ILM, auto will resolve to true. Otherwise auto will resolve to false.
# Default value is auto.
#enabled: "auto"

#setup:
 ' # Only disable setup if you want to set up everything related to ILM on your own.
 ' # When setup is enabled, the APM Server creates:
  '# - aliases and ILM policies if `apm-server.ilm.enabled` resolves to `true`.
'  # - An ILM specific template per event type. This is required to map ILM aliases and policies to indices. In case
  '# ILM is disabled, the templates will be created without any ILM settings.
  '# Be aware that if you turn off setup, you need to manually manage event type specific templates on your own.
  '# If you simply want to disable ILM, use the above setting, `apm-server.ilm.enabled`, instead.
  '# Defaults to true.
  #enabled: true

  # Configure whether or not existing policies and ILM related templates should be updated. This needs to be
  # set to true when customizing your policies.
  # Defaults to false.
  #overwrite: false

  # Set `require_policy` to `false` when policies are set up outside of APM Server but referenced here.
  # Default value is `true`.
  #require_policy: true

  # Customized mappings will be merged with the default setup, so you only need to configure mappings for the
  # event types, policies, and index suffixes that you want to customize.
  # Indices are named in this way: `apm-%{[observer.version]}-%{[event.type]}-{index_suffix}`,
  # e.g., apm-7.9.0-span-custom*. The `index_suffix` is optional.
  # NOTE: When configuring an `index_suffix`, ensure that no previously set up templates conflict with the
'  #       newly configured ones. If an index matches multiple templates with the same order, the settings of
 ' #       the templates will override each other. Any conflicts need to be cleaned up manually.
  '# NOTE: When customizing `setup.template.name` and `setup.template.pattern`, ensure they still match the indices.
  #mapping:
    #- event_type: "error"
    #  policy_name: "apm-rollover-30-days"
    #  index_suffix: ""
    #- event_type: "span"
    #  policy_name: "apm-rollover-30-days"
    #  index_suffix: ""
    #- event_type: "transaction"
    #  policy_name: "apm-rollover-30-days"
    #  index_suffix: ""
    #- event_type: "metric"
    #  policy_name: "apm-rollover-30-days"
    #  index_suffix: ""

 '# Configured policies are added to pre-defined default policies.
  '# If a policy with the same name as a default policy is configured, the configured policy overwrites the default policy.
  #policies:
    #- name: "apm-rollover-30-days"
      #policy:
        #phases:
          #hot:
            #actions:
              #rollover:
                #max_size: "50gb"
                #max_age: "30d"
              #set_priority:
                #priority: 100

#---------------------------- APM Server - Experimental Jaeger integration ----------------------------

'# When enabling Jaeger integration, APM Server acts as Jaeger collector. It supports jaeger.thrift over HTTP
'# and gRPC. This is an experimental feature, use with care.

' # WARNING: This configuration is deprecated, and will be removed in the 8.0 release.

'# Jaeger gRPC is now served on the same port as Elastic APM agents, defined by the
'# "apm-server.host" configuration; it is implicitly enabled, and an agent tag called
'# "elastic-apm-auth" is required when auth is enabled.
#jaeger:
#grpc:
# Set to true to enable the Jaeger gRPC collector service.
#enabled: false

  # Defines the gRPC host and port the server is listening on.
  # Defaults to the standard Jaeger gRPC collector port 14250.
  #host: "localhost:14250"

  # Set to the name of a process tag to use for authorizing
  # Jaeger agents.
  #
  # The tag value should have the same format as an HTTP
  # Authorization header, i.e. "Bearer <secret_token>" or
  # "ApiKey <base64(id:key)>".
  #
  # By default (if the auth_tag value is empty), authorization
  # does not apply to Jaeger agents.
  #auth_tag: ""

#http:
 ' # Set to true to enable the Jaeger HTTP collector endpoint.
  #enabled: false

  # Defines the HTTP host and port the server is listening on.
  # Defaults to the standard Jaeger HTTP collector port 14268.
  #host: "localhost:14268"

#================================= General =================================

'# Data is buffered in a memory queue before it is published to the configured output.
'# The memory queue will present all available events (up to the outputs
'# bulk_max_size) to the output, the moment the output is ready to serve
'# another batch of events.
#queue:
'# Queue type by name (default 'mem').
'#mem:
'# Max number of events the queue can buffer.
#events: 4096

'# Hints the minimum number of events stored in the queue,

' # before providing a batch of events to the outputs.
'# The default value is set to 2048.
'# A value of 0 ensures events are immediately available
'# to be sent to the outputs.
#flush.min_events: 2048

'# Maximum duration after which events are available to the outputs,
'# if the number of events stored in the queue is < `flush.min_events`.
#flush.timeout: 1s

'# Sets the maximum number of CPUs that can be executing simultaneously. The
'# default is the number of logical CPUs available in the system.
#max_procs:

#================================= Template =================================

'# A template is used to set the mapping in Elasticsearch.
'# By default template loading is enabled and the template is loaded.
'# These settings can be adjusted to load your own template or overwrite existing ones.

'# Set to false to disable template loading.
setup.template.enabled: true

'# Template name. By default the template name is "apm-%{[observer.version]}"
'# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.name: "apm-%{[observer.version]}"

'# Template pattern. By default the template pattern is "apm-%{[observer.version]}-" to apply to the default index settings.
'# The first part is the version of apm-server and then -
is used to match all daily indices.
'# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.pattern: "apm-%{[observer.version]}-*"

'# Path to fields.yml file to generate the template.
'#setup.template.fields: "${path.config}/fields.yml"

'# Overwrite existing template.
#setup.template.overwrite: false

'# Elasticsearch template settings.
#setup.template.settings:

'# A dictionary of settings to place into the settings.index dictionary
'# of the Elasticsearch template. For more details, please check
'# Mapping | Elasticsearch Guide [8.15] | Elastic
#index:
#number_of_shards: 1
#codec: best_compression
#number_of_routing_shards: 30
#mapping.total_fields.limit: 2000

#============================= Elastic Cloud =============================

'# These settings simplify using APM Server with the Elastic Cloud (https://cloud.elastic.co/).

'# The cloud.id setting overwrites the output.elasticsearch.hosts option.
'# You can find the cloud.id in the Elastic Cloud web UI.
#cloud.id:

'# The cloud.auth setting overwrites the output.elasticsearch.username and
'# output.elasticsearch.password settings. The format is <user>:<pass>.
#cloud.auth:

#================================ Outputs =================================

'# Configure the output to use when sending the data collected by apm-server.

#-------------------------- Elasticsearch output --------------------------
output.elasticsearch:
'# Array of hosts to connect to.
'# Scheme and port can be left out and will be set to the default (http and 9200).
' # In case you specify and additional path, the scheme is required: http://localhost:9200/path.
'# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200.
hosts: ["TEST04:9200"]

'# Boolean flag to enable or disable the output module.
#enabled: true

'# Set gzip compression level.
#compression_level: 0

'# Protocol - either http (default) or https.
protocol: "https"

'# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
username: "elastic"
password: "changeme"

'# Dictionary of HTTP parameters to pass within the url with index operations.
#parameters:
#param1: value1
#param2: value2

'# Number of workers per Elasticsearch host.
#worker: 1

'# By using the configuration below, APM documents are stored to separate indices,
'# depending on their processor.event:
'# - error
'# - transaction
'# - span
'# - sourcemap

'# The indices are all prefixed with apm-%{[observer.version]}.
' # To allow managing indices based on their age, all indices (except for sourcemaps)
' # end with the information of the day they got indexed.
'# e.g. "apm-7.3.0-transaction-2019.07.20"

'# Be aware that you can only specify one Elasticsearch template.
'# If you modify the index patterns you must also update these configurations accordingly,
'# as they need to be aligned:
' # * setup.template.name
'# * setup.template.pattern
#index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"
#indices:
' # - index: "apm-%{[observer.version]}-sourcemap"
'# when.contains:
'# processor.event: "sourcemap"

'# - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
'# when.contains:
'# processor.event: "error"

'# - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
'# when.contains:
' # processor.event: "transaction"

' # - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
'# when.contains:
'# processor.event: "span"

'# - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
'# when.contains:
'# processor.event: "metric"

'# - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
'# when.contains:
' # processor.event: "onboarding"

'# A pipeline is a definition of processors applied to documents when ingesting them to Elasticsearch.
' # APM Server comes with a default pipeline definition, located at ingest/pipeline/definition.json, which is
' # loaded to Elasticsearch by default (see apm-server.register.ingest.pipeline).
'# APM pipeline is enabled by default. To disable it, set pipeline: _none.
#pipeline: "apm"

'# Optional HTTP Path.
#path: "/elasticsearch"

'# Custom HTTP headers to add to each request.
#headers:
' # X-My-Header: Contents of the header

' # Proxy server url.
#proxy_url: http://proxy:3128

' # The number of times a particular Elasticsearch index operation is attempted. If
'# the indexing operation doesn't succeed after this many retries, the events are
'# dropped. The default is 3.
#max_retries: 3

' # The maximum number of events to bulk in a single Elasticsearch bulk API index request.

The default is 50.

#bulk_max_size: 50

'# The number of seconds to wait before trying to reconnect to Elasticsearch
'# after a network error. After waiting backoff.init seconds, apm-server
'# tries to reconnect. If the attempt fails, the backoff timer is increased
'# exponentially up to backoff.max. After a successful connection, the backoff
'# timer is reset. The default is 1s.
#backoff.init: 1s

'# The maximum number of seconds to wait before attempting to connect to
' # Elasticsearch after a network error. The default is 60s.
#backoff.max: 60s

'# Configure http request timeout before failing an request to Elasticsearch.
#timeout: 90

'# Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
ssl.enabled: true

'# Optional SSL configuration options. SSL is off by default, change the protocol option if you want to enable https.

'# Control the verification of Elasticsearch certificates. Valid values are:
' # * full, which verifies that the provided certificate is signed by a trusted
' # authority (CA) and also verifies that the server's hostname (or IP address)
' # matches the names identified within the certificate.
'# * strict, which verifies that the provided certificate is signed by a trusted
'# authority (CA) and also verifies that the server's hostname (or IP address)
'# matches the names identified within the certificate. If the Subject Alternative
'# Name is empty, it returns an error.
'# * certificate, which verifies that the provided certificate is signed by a
'# trusted authority (CA), but does not perform any hostname verification.
'# * none, which performs no verification of the server's certificate. This
' # mode disables many of the security benefits of SSL/TLS and should only be used
'# after very careful consideration. It is primarily intended as a temporary
' # diagnostic mechanism when attempting to resolve TLS errors; its use in
'# production environments is strongly discouraged.
ssl.verification_mode: none

'# List of supported/valid TLS versions. By default all TLS versions 1.0 up to
'# 1.2 are enabled.
ssl.supported_protocols: [TLSv1.2]

' # List of root certificates for HTTPS server verifications.
ssl.certificate_authorities: ["/etc/apm-server/cert/certificate_ca.crt"]

'# Certificate for SSL client authentication.
ssl.certificate: "/etc/apm-server/cert/certificate.crt"

'# Client Certificate Key
ssl.key: "/etc/apm-server/cert/certificate.key"

' # Optional passphrase for decrypting the Certificate Key.
'# It is recommended to use the provided keystore instead of entering the passphrase in plain text.
#ssl.key_passphrase: ''

'# Configure cipher suites to be used for SSL connections.
#ssl.cipher_suites:

'# Configure curve types for ECDHE based cipher suites.
#ssl.curve_types:

'# Configure what types of renegotiation are supported. Valid options are
' # never, once, and freely. Default is never.
#ssl.renegotiation: never

'# Enable Kerberos support. Kerberos is automatically enabled if any Kerberos setting is set.
#kerberos.enabled: true

'# Authentication type to use with Kerberos. Available options: keytab, password.
#kerberos.auth_type: password

'# Path to the keytab file. It is used when auth_type is set to keytab.
#kerberos.keytab: /etc/elastic.keytab

'# Path to the Kerberos configuration.
#kerberos.config_path: /etc/krb5.conf

' # Name of the Kerberos user.
#kerberos.username: elastic

'# Password of the Kerberos user. It is used when auth_type is set to password.
#kerberos.password: changeme

'# Kerberos realm.
#kerberos.realm: ELASTIC

'#----------------------------- Console output -----------------------------
#output.console:
'# Boolean flag to enable or disable the output module.
#enabled: false

'# Configure JSON encoding.
#codec.json:
# Pretty-print JSON event.
#pretty: false

# Configure escaping HTML symbols in strings.
#escape_html: false"

@Edoardo_Tenani onboarding index is only come with date by default when we install apm-server rest of indices are not coming wih date
i have sent apm-server.yml configuration in proper format. can you please look into it and guid me what setting still remaining ?

thanks in advance

Hi, anyone can help here?

Hello @Ekta, the apm-server.yml configuration that you shared is valid, and we were able to create the indices with the expected index name pattern "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}" using this configuration.

One important thing to note that the new indices will be only created "lazily" when the data is sent to the APM Server, they are not automatically created when the APM Server is restarted. Therefore, could please share the way that you use to send data to the APM Server?