How to resolve ILM errors about missing "scaling_factor"?

Elastic Stack Elasticsearch
1

My ILM policy is configured to downsample metrics. The policy does work, and I have metrics being downsampled, but a few indices are stuck with lifecycle errors.

Specifically several .ds-metrics-docker.memory-default... indices.

Index lifecycle error
mapper_parsing_exception: Failed to parse mapping: Field [scaling_factor] is required

The Elasticsearch logs are not much help, I get an error message and a stack trace, but neither tell me anything about which field is missing the scaling_factor.

Here are some fields from the logs I found in Kibana:

elasticsearch.server.error.message:

Failed to parse mapping: Field [scaling_factor] is required

elasticsearch.server.error.stack_trace:

org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping: Field [scaling_factor] is required
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:400)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:376)
	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.createDownsampleIndexMapping(TransportDownsampleAction.java:534)
	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.lambda$masterOperation$6(TransportDownsampleAction.java:315)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:161)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:199)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:298)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:182)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.admin.indices.mapping.get.TransportGetMappingsAction.doMasterOperation(TransportGetMappingsAction.java:73)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.admin.indices.mapping.get.TransportGetMappingsAction.doMasterOperation(TransportGetMappingsAction.java:30)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:64)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:24)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:124)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:235)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: Field [scaling_factor] is required
	at org.elasticsearch.mapper.extras@8.10.3/org.elasticsearch.index.mapper.extras.ScaledFloatFieldMapper$Builder.lambda$new$5(ScaledFloatFieldMapper.java:99)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.FieldMapper$Parameter.validate(FieldMapper.java:775)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.FieldMapper$Builder.validate(FieldMapper.java:1203)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.FieldMapper$Builder.parse(FieldMapper.java:1358)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.FieldMapper$TypeParser.parse(FieldMapper.java:1432)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.FieldMapper$TypeParser.parse(FieldMapper.java:1389)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:325)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:235)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.RootObjectMapper.parse(RootObjectMapper.java:411)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:103)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:97)
	at org.elasticsearch.server@8.10.3/org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:398)
	... 24 more

elasticsearch.server.error.type:

org.elasticsearch.index.mapper.MapperParsingException

message:

policy [metrics] for index [.ds-metrics-docker.memory-default-2023.10.19-000004] failed on step [{"phase":"warm","action":"downsample","name":"rollup"}]. Moving to ERROR step

I do have a script that runs occasionally to copy all the index templates, make a couple settings changes, and then save them with a higher priority than what Elastic Agent sets. But I do not touch any of mappings, just a couple settings. So I'm not 100% sure that my settings aren't the cause, but I do think it unlikely.

What would help is if there was a way to figure out which field is causing the ILM error. Anyone know a way?

Anyone have any other suggestions on how to resolve this?

2

Do you have scaled_float in your mappings, perhaps look for those...

What changes...

3

I checked the mappings in the default and custom templates, as well as the mappings on the errored index. All scaled_float mappings have a scaling factor set.

I add this component template to the cluster:

{"template": {"settings": { "index": { "number_of_replicas": "0" }}}}

Then I load the existing metrics-* templates from _index_template/metrics*, modify their names with a prefix, up their priority, and add that component template to the list of component templates for each full template.

So I should have exactly the same mappings as what Elastic Agent sets.

4

What Version?
Are you specifically trying to do downsampling in your ILM?

5

Ah, sorry, 8.10.3 for everything.

Yes.

My metrics policy:

PUT _ilm/policy/metrics
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "2gb"
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "downsample": {
            "fixed_interval": "1m",
            "wait_timeout": "1d"
          },
          "forcemerge": {
            "max_num_segments": 1,
            "index_codec": "best_compression"
          },
          "set_priority": {
            "priority": 50
          },
          "shrink": {
            "number_of_shards": 1
          },
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "cold": {
        "min_age": "7d",
        "actions": {
          "downsample": {
            "fixed_interval": "5m",
            "wait_timeout": "1d"
          },
          "set_priority": {
            "priority": 0
          },
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
6

I am not sure what it is but seems related to Downsampling... and the Mapping not being correct ... perhaps related to your renaming etc...

Do you have a subscription if so, I would open a Support Ticket, I will also see if I can poke around a bit.

If you run _ilm/explain on one of those indices does it show anything

GET /.ds-metrics-docker.memory-default.../_ilm/explain

7

FYI, this is a new test cluster that started at 8.10.3. I'm running it single-node inside docker.

No subscription. I've tried to get one, and it isn't going to happen unless we get a sudden influx of money, or the cost goes way down. Perils of working at a non-profit community college. This is all part of my attempt at forcing the Elastic Stack to work in a way we can actually afford.

Here's the the ILM/explain from the oldest index that has the error:

{
  "indices": {
    ".ds-metrics-docker.memory-default-2023.10.16-000001": {
      "index": ".ds-metrics-docker.memory-default-2023.10.16-000001",
      "managed": true,
      "policy": "metrics",
      "index_creation_date_millis": 1697482569595,
      "time_since_index_creation": "11.15d",
      "lifecycle_date_millis": 1697569443671,
      "age": "10.14d",
      "phase": "warm",
      "phase_time_millis": 1698445873581,
      "action": "downsample",
      "action_time_millis": 1697656074915,
      "step": "ERROR",
      "step_time_millis": 1698445873987,
      "failed_step": "rollup",
      "is_auto_retryable_error": true,
      "failed_step_retry_count": 1308,
      "step_info": {
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping: Field [scaling_factor] is required",
        "caused_by": {
          "type": "illegal_argument_exception",
          "reason": "Field [scaling_factor] is required"
        }
      },
      "phase_execution": {
        "policy": "metrics",
        "phase_definition": {
          "min_age": "1d",
          "actions": {
            "downsample": {
              "fixed_interval": "1m",
              "wait_timeout": "1d"
            },
            "forcemerge": {
              "max_num_segments": 1,
              "index_codec": "best_compression"
            },
            "allocate": {
              "number_of_replicas": 0,
              "include": {},
              "exclude": {},
              "require": {}
            },
            "set_priority": {
              "priority": 50
            },
            "shrink": {
              "number_of_shards": 1
            }
          }
        },
        "version": 4,
        "modified_date_in_millis": 1698359524613
      }
    }
  }
}

From the newest index with the error:

{
  "indices": {
    ".ds-metrics-docker.memory-default-2023.10.25-000010": {
      "index": ".ds-metrics-docker.memory-default-2023.10.25-000010",
      "managed": true,
      "policy": "metrics",
      "index_creation_date_millis": 1698264817527,
      "time_since_index_creation": "2.1d",
      "lifecycle_date_millis": 1698351358552,
      "age": "1.09d",
      "phase": "warm",
      "phase_time_millis": 1698445873581,
      "action": "downsample",
      "action_time_millis": 1698437763256,
      "step": "ERROR",
      "step_time_millis": 1698445873987,
      "failed_step": "rollup",
      "is_auto_retryable_error": true,
      "failed_step_retry_count": 11,
      "step_info": {
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping: Field [scaling_factor] is required",
        "caused_by": {
          "type": "illegal_argument_exception",
          "reason": "Field [scaling_factor] is required"
        }
      },
      "phase_execution": {
        "policy": "metrics",
        "phase_definition": {
          "min_age": "1d",
          "actions": {
            "downsample": {
              "fixed_interval": "1m",
              "wait_timeout": "1d"
            },
            "forcemerge": {
              "max_num_segments": 1,
              "index_codec": "best_compression"
            },
            "allocate": {
              "number_of_replicas": 0,
              "include": {},
              "exclude": {},
              "require": {}
            },
            "set_priority": {
              "priority": 50
            },
            "shrink": {
              "number_of_shards": 1
            }
          }
        },
        "version": 4,
        "modified_date_in_millis": 1698359524613
      }
    }
  }
}
8

No worries on the subscription. I was just asking. We love our basic users !!

I don't have an answer for you but what I would try is to run one of these without doing your renaming thing... It's failing on the roll-up / down sample. And it's looking like on the down sampled result that somehow the mapping is not correct / incorrect.

Per the docs, did you try running down sampling manually

9

Do downsampled indices have the same template as the source index?

I'll figure out the manual run when I get back to the office next week.

Thanks for the help so far. :slight_smile:

10 
PUT /.ds-metrics-docker.memory-default-2023.10.16-000001/_block/write
POST /.ds-metrics-docker.memory-default-2023.10.16-000001/_downsample/downsample-manual-.ds-metrics-docker.memory-default-2023.10.16-000001
{
  "fixed_interval": "1m"
}

Results in:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping: Field [scaling_factor] is required"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping: Field [scaling_factor] is required",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Field [scaling_factor] is required"
    }
  },
  "status": 400
}

Is there a way to force it to log which field is supposedly missing the scaling_factor?

11

I pinged internally... we shall see...

12

This looks like a bug. Are you able to share the mapping for .ds-metrics-docker.memory-default-2023.10.16-000001 index?

13

Here are the mappings from .ds-metrics-docker.memory-default-2023.10.16-000001.

{
  "mappings": {
    "_meta": {
      "managed_by": "fleet",
      "managed": true,
      "package": {
        "name": "docker"
      }
    },
    "_data_stream_timestamp": {
      "enabled": true
    },
    "_source": {
      "mode": "synthetic"
    },
    "dynamic_templates": [
      {
        "strings_as_keyword": {
          "match_mapping_type": "string",
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      }
    ],
    "date_detection": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "agent": {
        "properties": {
          "ephemeral_id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "cloud": {
        "properties": {
          "account": {
            "properties": {
              "id": {
                "type": "keyword",
                "time_series_dimension": true
              }
            }
          },
          "availability_zone": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "instance": {
            "properties": {
              "id": {
                "type": "keyword",
                "time_series_dimension": true
              }
            }
          },
          "provider": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "region": {
            "type": "keyword",
            "time_series_dimension": true
          }
        }
      },
      "container": {
        "properties": {
          "id": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "image": {
            "properties": {
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "memory": {
            "properties": {
              "usage": {
                "type": "scaled_float",
                "meta": {
                  "unit": "percent"
                },
                "scaling_factor": 1000,
                "time_series_metric": "gauge"
              }
            }
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "runtime": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "data_stream": {
        "properties": {
          "dataset": {
            "type": "constant_keyword",
            "value": "docker.memory"
          },
          "namespace": {
            "type": "constant_keyword",
            "value": "default"
          },
          "type": {
            "type": "constant_keyword",
            "value": "metrics"
          }
        }
      },
      "docker": {
        "properties": {
          "container": {
            "properties": {
              "labels": {
                "properties": {
                  "*": {
                    "type": "object"
                  },
                  "alertdrop_condition_green_count": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "alertdrop_condition_red_count": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "alertdrop_condition_yellow_count": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "alertdrop_environment": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "alertdrop_identifier": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "alertdrop_service_type": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/audit_enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/deprecation_enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/gc_enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/log_enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/module": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/multiline_match": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/multiline_negate": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/multiline_pattern": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/multiline_type": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/parsers_1_multiline_match": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/parsers_1_multiline_negate": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/parsers_1_multiline_pattern": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/parsers_1_multiline_type": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/processors_1_add_tags_tags": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/processors_2_add_tags_tags": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/processors_3_add_tags_tags": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/server_enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/slowlog_enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_metrics/enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_metrics/hosts": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_metrics/metricsets": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_metrics/module": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_metrics/period": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_desktop_extension_api_version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_desktop_extension_icon": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_extension_additional-urls": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_extension_detailed-description": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_extension_publisher-url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_extension_screenshots": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_stack_namespace": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_node_id": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_service_id": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_service_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_task": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_task_id": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_task_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_vmware_cp_artifact_flavor": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "description": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "elk_log_type": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "elk_log_typee": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "io_k8s_description": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "io_k8s_display-name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "io_portainer_agent": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "io_portainer_server": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "lanecc_project_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "license": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "maintainer": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_build-date": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_license": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_schema-version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_usage": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_vcs-ref": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_vcs-url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_vendor": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_base_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_created": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_description": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_documentation": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_licenses": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_ref_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_revision": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_source": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_title": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_vendor": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "release": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "summary": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "traefik_enable": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "vendor": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              }
            }
          },
          "memory": {
            "properties": {
              "commit": {
                "properties": {
                  "peak": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  },
                  "total": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "counter"
                  }
                }
              },
              "fail": {
                "properties": {
                  "count": {
                    "type": "scaled_float",
                    "scaling_factor": 1000,
                    "time_series_metric": "counter"
                  }
                }
              },
              "limit": {
                "type": "long",
                "meta": {
                  "unit": "byte"
                },
                "time_series_metric": "gauge"
              },
              "private_working_set": {
                "properties": {
                  "total": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  }
                }
              },
              "rss": {
                "properties": {
                  "pct": {
                    "type": "scaled_float",
                    "meta": {
                      "unit": "percent"
                    },
                    "scaling_factor": 1000,
                    "time_series_metric": "gauge"
                  },
                  "total": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  }
                }
              },
              "stats": {
                "properties": {
                  "*": {
                    "type": "object"
                  },
                  "active_anon": {
                    "type": "long"
                  },
                  "active_file": {
                    "type": "long"
                  },
                  "anon": {
                    "type": "float"
                  },
                  "anon_thp": {
                    "type": "long"
                  },
                  "file": {
                    "type": "long"
                  },
                  "file_dirty": {
                    "type": "long"
                  },
                  "file_mapped": {
                    "type": "long"
                  },
                  "file_writeback": {
                    "type": "long"
                  },
                  "inactive_anon": {
                    "type": "float"
                  },
                  "inactive_file": {
                    "type": "long"
                  },
                  "kernel_stack": {
                    "type": "long"
                  },
                  "pgactivate": {
                    "type": "long"
                  },
                  "pgdeactivate": {
                    "type": "long"
                  },
                  "pgfault": {
                    "type": "long"
                  },
                  "pglazyfree": {
                    "type": "long"
                  },
                  "pglazyfreed": {
                    "type": "long"
                  },
                  "pgmajfault": {
                    "type": "long"
                  },
                  "pgrefill": {
                    "type": "long"
                  },
                  "pgscan": {
                    "type": "long"
                  },
                  "pgsteal": {
                    "type": "long"
                  },
                  "shmem": {
                    "type": "long"
                  },
                  "slab": {
                    "type": "long"
                  },
                  "slab_reclaimable": {
                    "type": "long"
                  },
                  "slab_unreclaimable": {
                    "type": "long"
                  },
                  "sock": {
                    "type": "long"
                  },
                  "thp_collapse_alloc": {
                    "type": "long"
                  },
                  "thp_fault_alloc": {
                    "type": "long"
                  },
                  "unevictable": {
                    "type": "long"
                  },
                  "workingset_activate": {
                    "type": "long"
                  },
                  "workingset_nodereclaim": {
                    "type": "long"
                  },
                  "workingset_refault": {
                    "type": "long"
                  }
                }
              },
              "usage": {
                "properties": {
                  "max": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  },
                  "pct": {
                    "type": "scaled_float",
                    "meta": {
                      "unit": "percent"
                    },
                    "scaling_factor": 1000,
                    "time_series_metric": "gauge"
                  },
                  "total": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  }
                }
              }
            }
          }
        }
      },
      "ecs": {
        "properties": {
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "elastic_agent": {
        "properties": {
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "snapshot": {
            "type": "boolean"
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "event": {
        "properties": {
          "agent_id_status": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "dataset": {
            "type": "constant_keyword",
            "value": "docker.memory"
          },
          "duration": {
            "type": "long"
          },
          "ingested": {
            "type": "date",
            "format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
          },
          "module": {
            "type": "constant_keyword",
            "value": "docker"
          }
        }
      },
      "host": {
        "properties": {
          "architecture": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "containerized": {
            "type": "boolean"
          },
          "hostname": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "ip": {
            "type": "ip"
          },
          "mac": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "os": {
            "properties": {
              "codename": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "family": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "full": {
                "type": "keyword",
                "ignore_above": 1024,
                "fields": {
                  "text": {
                    "type": "match_only_text"
                  }
                }
              },
              "kernel": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024,
                "fields": {
                  "text": {
                    "type": "match_only_text"
                  }
                }
              },
              "platform": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "type": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "version": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "metricset": {
        "properties": {
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "period": {
            "type": "long"
          }
        }
      },
      "service": {
        "properties": {
          "address": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
    }
  }
}
14

@mvg Just in case my reply didn't trigger a notification, is my last post the mapping you needed?

15

Any ideas on this?

16

@stephenb @mvg So, I ended up doing a completely clean install on 8.11.1. A few days later and the scaling_factor error is back.

image
image1548×252 23.6 KB

Here are the mappings for: .ds-metrics-docker.memory-testswarmstack-2023.11.27-000001

{
  "mappings": {
    "_meta": {
      "managed_by": "fleet",
      "managed": true,
      "package": {
        "name": "docker"
      }
    },
    "_data_stream_timestamp": {
      "enabled": true
    },
    "_source": {
      "mode": "synthetic"
    },
    "dynamic_templates": [
      {
        "strings_as_keyword": {
          "match_mapping_type": "string",
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      }
    ],
    "date_detection": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "agent": {
        "properties": {
          "ephemeral_id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "cloud": {
        "properties": {
          "account": {
            "properties": {
              "id": {
                "type": "keyword",
                "time_series_dimension": true
              }
            }
          },
          "availability_zone": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "instance": {
            "properties": {
              "id": {
                "type": "keyword",
                "time_series_dimension": true
              }
            }
          },
          "provider": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "region": {
            "type": "keyword",
            "time_series_dimension": true
          }
        }
      },
      "container": {
        "properties": {
          "id": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "image": {
            "properties": {
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "memory": {
            "properties": {
              "usage": {
                "type": "scaled_float",
                "meta": {
                  "unit": "percent"
                },
                "scaling_factor": 1000,
                "time_series_metric": "gauge"
              }
            }
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "runtime": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "data_stream": {
        "properties": {
          "dataset": {
            "type": "constant_keyword",
            "value": "docker.memory"
          },
          "namespace": {
            "type": "constant_keyword",
            "value": "testswarmstack"
          },
          "type": {
            "type": "constant_keyword",
            "value": "metrics"
          }
        }
      },
      "docker": {
        "properties": {
          "container": {
            "properties": {
              "labels": {
                "properties": {
                  "*": {
                    "type": "object"
                  },
                  "alertdrop_environment": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "alertdrop_identifier": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "alertdrop_service_type": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/fileset_stderr": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/fileset_stdout": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/log_enabled": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/module": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/multiline_match": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/multiline_negate": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/multiline_pattern": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/multiline_type": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/processors_1_add_tags_tags": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/processors_2_add_tags_tags": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "co_elastic_logs/processors_3_add_tags_tags": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_desktop_extension_api_version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_desktop_extension_icon": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_extension_additional-urls": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_extension_detailed-description": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_extension_publisher-url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_extension_screenshots": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_stack_namespace": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_node_id": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_service_id": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_service_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_task": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_task_id": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_docker_swarm_task_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "com_vmware_cp_artifact_flavor": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "description": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "io_k8s_description": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "io_k8s_display-name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "io_portainer_agent": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "io_portainer_server": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "license": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "maintainer": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_build-date": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_license": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_schema-version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_usage": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_vcs-ref": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_vcs-url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_vendor": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_label-schema_version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_base_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_created": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_description": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_documentation": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_licenses": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_ref_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_revision": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_source": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_title": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_vendor": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "org_opencontainers_image_version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "release": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "summary": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "swarm_cronjob_enable": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "swarm_cronjob_schedule": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "swarm_cronjob_skip-running": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "traefik_enable": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "url": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "vendor": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              }
            }
          },
          "memory": {
            "properties": {
              "commit": {
                "properties": {
                  "peak": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  },
                  "total": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "counter"
                  }
                }
              },
              "fail": {
                "properties": {
                  "count": {
                    "type": "scaled_float",
                    "scaling_factor": 1000,
                    "time_series_metric": "counter"
                  }
                }
              },
              "limit": {
                "type": "long",
                "meta": {
                  "unit": "byte"
                },
                "time_series_metric": "gauge"
              },
              "private_working_set": {
                "properties": {
                  "total": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  }
                }
              },
              "rss": {
                "properties": {
                  "pct": {
                    "type": "scaled_float",
                    "meta": {
                      "unit": "percent"
                    },
                    "scaling_factor": 1000,
                    "time_series_metric": "gauge"
                  },
                  "total": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  }
                }
              },
              "stats": {
                "properties": {
                  "*": {
                    "type": "object"
                  },
                  "active_anon": {
                    "type": "long"
                  },
                  "active_file": {
                    "type": "float"
                  },
                  "anon": {
                    "type": "float"
                  },
                  "anon_thp": {
                    "type": "long"
                  },
                  "file": {
                    "type": "float"
                  },
                  "file_dirty": {
                    "type": "long"
                  },
                  "file_mapped": {
                    "type": "long"
                  },
                  "file_writeback": {
                    "type": "long"
                  },
                  "inactive_anon": {
                    "type": "float"
                  },
                  "inactive_file": {
                    "type": "long"
                  },
                  "kernel_stack": {
                    "type": "long"
                  },
                  "pgactivate": {
                    "type": "long"
                  },
                  "pgdeactivate": {
                    "type": "long"
                  },
                  "pgfault": {
                    "type": "long"
                  },
                  "pglazyfree": {
                    "type": "long"
                  },
                  "pglazyfreed": {
                    "type": "long"
                  },
                  "pgmajfault": {
                    "type": "long"
                  },
                  "pgrefill": {
                    "type": "long"
                  },
                  "pgscan": {
                    "type": "long"
                  },
                  "pgsteal": {
                    "type": "long"
                  },
                  "shmem": {
                    "type": "long"
                  },
                  "slab": {
                    "type": "long"
                  },
                  "slab_reclaimable": {
                    "type": "long"
                  },
                  "slab_unreclaimable": {
                    "type": "long"
                  },
                  "sock": {
                    "type": "long"
                  },
                  "thp_collapse_alloc": {
                    "type": "long"
                  },
                  "thp_fault_alloc": {
                    "type": "long"
                  },
                  "unevictable": {
                    "type": "long"
                  },
                  "workingset_activate": {
                    "type": "long"
                  },
                  "workingset_nodereclaim": {
                    "type": "long"
                  },
                  "workingset_refault": {
                    "type": "long"
                  }
                }
              },
              "usage": {
                "properties": {
                  "max": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  },
                  "pct": {
                    "type": "scaled_float",
                    "meta": {
                      "unit": "percent"
                    },
                    "scaling_factor": 1000,
                    "time_series_metric": "gauge"
                  },
                  "total": {
                    "type": "long",
                    "meta": {
                      "unit": "byte"
                    },
                    "time_series_metric": "gauge"
                  }
                }
              }
            }
          }
        }
      },
      "ecs": {
        "properties": {
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "elastic_agent": {
        "properties": {
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "snapshot": {
            "type": "boolean"
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "event": {
        "properties": {
          "agent_id_status": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "dataset": {
            "type": "constant_keyword",
            "value": "docker.memory"
          },
          "duration": {
            "type": "long"
          },
          "ingested": {
            "type": "date",
            "format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
          },
          "module": {
            "type": "constant_keyword",
            "value": "docker"
          }
        }
      },
      "host": {
        "properties": {
          "architecture": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "containerized": {
            "type": "boolean"
          },
          "hostname": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "ip": {
            "type": "ip"
          },
          "mac": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "os": {
            "properties": {
              "codename": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "family": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "full": {
                "type": "keyword",
                "ignore_above": 1024,
                "fields": {
                  "text": {
                    "type": "match_only_text"
                  }
                }
              },
              "kernel": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024,
                "fields": {
                  "text": {
                    "type": "match_only_text"
                  }
                }
              },
              "platform": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "type": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "version": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "metricset": {
        "properties": {
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "period": {
            "type": "long"
          }
        }
      },
      "service": {
        "properties": {
          "address": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
    }
  }
}
17

Apologies @jerrac, somehow the notifications were not sent to my email.
We found the cause of the issue that you are encountering here and filed a bug for this:

2 Likes
18

Awesome! Thanks!

19

Just upgraded to 8.12 in test and the lifecycle errors have gone away. Yay!

2 Likes
20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.