Apache2 User Agent not avalaible in the Visualize menu

[dkdlv38]

Hello all!

I'm trying to create a visualisation to track the user agent of the Apache clients connecting to my website, I have this information in the "Discover" menu, but it doesn't appear in the "Visualize" menu...

What am I missing?

Thanks for your help =)

[afharo (Alejandro Fernandez Haro)]

Hi @dkdlv38,
I think that the cause could be the mappings: the terms aggregations can only be performed in keyword fields. My first guess is that the field apache2_client_user_agent is type: "text".

If that's the case, you might be able to solve it by adding a multifield property keyword, so the mapping of your field looks like:

"apache2_client_user_agent": {
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above" : 256
  }
}

Once you've updated your mapping, you can refresh Kibana's Index Pattern to identify the new field, and you should see the field apache2_client_user_agent.keyword in the Field box now.

NOTE: For these mapping changes to take effect in previously indexed docs, you might need to run POST apache2-*/_update_by_query so Elasticsearch reprocesses all the documents again.

[dkdlv38]

Thanks fr your answer.

Candid question: why it is not possible to simply put:

        "apache2_client_user_agent" : {
  "type" : "keyword"
        },

And yes, your first guess was correct ^^

Your modification has to be done in the /etc/elasticsearch/templates/apache2.template.json
is that correct?

Thanks for the help!

[dkdlv38]

Ok, self answer to my question: Changing an existing field could invalidate data that’s already indexed.

(Update mapping API | Elasticsearch Guide [7.12] | Elastic)

So I now have an issue of syntax, my file is written this way,
How should I insert your modifcation?:

  "index_patterns" : [
    "apache2-*"
    ],
  "settings" : {
    "index" : {
      "number_of_shards" : "1",
      "number_of_replicas" : "0"
        }
  },
  "mappings" : {
    "doc" : {
      "properties" : {
        "apache2_client_ip" : {
          "type" : "ip"
                },
        "apache2_client_identd" : {
          "type" : "keyword"
                },
                "apache2_client_protocole" : {
          "type" : "keyword"
                },
                "apache2_client_protocole_version" : {
          "type" : "keyword"
                },
                "apache2_client_user_agent" : {
          "type" : "text"
                },

[afharo (Alejandro Fernandez Haro)]

Hi, sorry for the late response.

Your changes in the template will change it for any newly created indices (Elasticsearch uses the templates when creating new indices only).

For existing indices, you might need to call the Mapping updates API (PUT apache2-*/_mapping), as you previously stated in your comment.

The reason for not replacing text by keyword straight away, is because old index won't be able to convert/cast it. You'll need to reindex (aka create a new index with the new mappings and copy the info from the old indices).

My suggestion is additive, so there's no need for a full reindex

@dkdlv38

How should I insert your modifcation?:

With the Update Mappings API, you can call the request below to add the field to the existing indices:

PUT apache2-*/_mapping/_doc
{
  "properties": {
    "apache2_client_user_agent": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above" : 256
        }
      }
    }
  }
}

Then, you'll need to run POST apache2-*/_update_by_query for the changes to get applied to the existing documents.

Then, to keep those changes to new indices when they rotate, you'll need to change the template as well. I believe that you can either do it via editing the .json file you mentioned or by using the templates API: Index Templates | Elasticsearch Guide [6.8] | Elastic

[dkdlv38]

I'm facing this issue, and I don't really understand what is the meaning of the error:

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Rejecting mapping update to [apache2-2021.04.09] as the final mapping would have more than 1 type: [_doc, doc]"
}
],
"type": "illegal_argument_exception",
"reason": "Rejecting mapping update to [apache2-2021.04.09] as the final mapping would have more than 1 type: [_doc, doc]"
},
"status": 400
}

[afharo (Alejandro Fernandez Haro)]

Right! Sorry, I made a typo: you are using the type doc instead of the _doc.

The actual request is:

PUT apache2-*/_mapping/doc
{
  "properties": {
    "apache2_client_user_agent": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above" : 256
        }
      }
    }
  }
}