Searching for a custom made header

Elastic Observability APM
2

Hi,

Since headers are not indexed as keyword by default, they are not searchable.

The safest way would be to get your Dev team to use the agent public APIs and set the values of these headers as labels in the transaction that corresponds the request handling (meaning - programmatically read the custom header and add a label to the traced transaction). Labels will be searchable.

Alternatively, you can try and change the mappings of the Transaction indices, and make your custom header searchable. I tried the following and it worked:

  1. In your APM server installation, create a copy of the fields.yml file
  2. Assuming that the header's name is Custom_header, edit the fields.yml and replace the headers section under http -> request from:
          - name: headers
            type: object
            enabled: false
            description: >
              The canonical headers of the monitored HTTP request.

to:

          - name: headers
            type: group
            description: >
              The canonical headers of the monitored HTTP request.
            fields:

             - name: Custom_header
               type: keyword
               description: >
                 My custom header
               overwrite: true
  1. Edit the apm-server.yml file so that setup.template.overwrite is set to true
  2. Restart the APM server
  3. Look for indexes of the form apm-<version>-transaction-<index-ID> in your index management. Once a new transaction index gets created, the mapping should be proper, and your custom header should become searchable.

However, notice that changing the template manually may cause some unexpected behaviour, for example when upgrading, so decide whether you want to take this risk.

I hope this helps,
Eyal.