_meta: Root mapping definition has unsupported parameters

Elastic Stack Elasticsearch
1

I have ES7. Currently the mapping looks like:

{
'mappings':
    {
     'properties': 
        {
        'aIndex': {'type': 'integer'},
        'AC': {'type': 'integer'},
        'AF': {'type': 'double'},
        ...
        },
    '_meta': {'gencodeVersion': '25', 'sourceFilePath': 's3://grch38_subset.vcf', 'genomeVersion': '38', 'sampleType': 'WGS', 'hail_version': '0.2.57'}
}

And it gives 400 Request error because it does not like keys inside _meta:

elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'Root mapping definition has unsupported parameters: [gencodeVersion : 25] [hail_version : 0.2.57] [genomeVersion : 38] [sampleType : WGS] [sourceFilePath : s3://grch38_subset.vcf]')

Below is the full mapping inside mappings key:

{'properties': {'aIndex': {'type': 'integer'}, 'AC': {'type': 'integer'}, 'AF': {'type': 'double'}, 'alt': {'type': 'keyword'}, 'AN': {'type': 'integer'}, 'bgi_AC': {'type': 'integer'}, 'bgi_AF': {'type': 'double'}, 'bgi_AN': {'type': 'integer'}, 'cadd_PHRED': {'type': 'float'}, 'cidr_AC': {'type': 'integer'}, 'cidr_AF': {'type': 'double'}, 'cidr_AN': {'type': 'integer'}, 'clinvar_allele_id': {'type': 'integer'}, 'clinvar_clinical_significance': {'type': 'keyword'}, 'clinvar_gold_stars': {'type': 'integer'}, 'codingGeneIds': {'type': 'keyword'}, 'contig': {'type': 'keyword'}, 'dbnsfp_SIFT_pred': {'type': 'keyword'}, 'dbnsfp_Polyphen2_HVAR_pred': {'type': 'keyword'}, 'dbnsfp_MutationTaster_pred': {'type': 'keyword'}, 'dbnsfp_FATHMM_pred': {'type': 'keyword'}, 'dbnsfp_MetaSVM_pred': {'type': 'keyword'}, 'dbnsfp_REVEL_score': {'type': 'keyword'}, 'dbnsfp_GERP_RS': {'type': 'keyword'}, 'dbnsfp_phastCons100way_vertebrate': {'type': 'keyword'}, 'docId': {'type': 'keyword'}, 'domains': {'type': 'keyword'}, 'eigen_Eigen_phred': {'type': 'double'}, 'end': {'type': 'integer'}, 'exac_AF_POPMAX': {'type': 'double'}, 'exac_AF': {'type': 'double'}, 'exac_AC_Adj': {'type': 'integer'}, 'exac_AC_Het': {'type': 'integer'}, 'exac_AC_Hom': {'type': 'integer'}, 'exac_AC_Hemi': {'type': 'integer'}, 'exac_AN_Adj': {'type': 'integer'}, 'filters': {'type': 'keyword'}, 'g1k_AC': {'type': 'integer'}, 'g1k_AF': {'type': 'double'}, 'g1k_AN': {'type': 'integer'}, 'g1k_POPMAX_AF': {'type': 'double'}, 'geneIds': {'type': 'keyword'}, 'geno2mp_HPO_Count': {'type': 'integer'}, 'genotypes': {'properties': {'num_alt': {'type': 'integer'}, 'gq': {'type': 'integer'}, 'ab': {'type': 'double'}, 'dp': {'type': 'double'}, 'ad': {'type': 'keyword'}, 'pl': {'type': 'keyword'}, 'sample_id': {'type': 'keyword'}}, 'type': 'nested'}, 'gnomad_exome_coverage': {'type': 'double'}, 'gnomad_exomes_AF': {'type': 'double'}, 'gnomad_exomes_AN': {'type': 'integer'}, 'gnomad_exomes_AC': {'type': 'integer'}, 'gnomad_exomes_Hom': {'type': 'integer'}, 'gnomad_exomes_AF_POPMAX_OR_GLOBAL': {'type': 'double'}, 'gnomad_exomes_FAF_AF': {'type': 'double'}, 'gnomad_exomes_Hemi': {'type': 'integer'}, 'gnomad_genome_coverage': {'type': 'float'}, 'gnomad_genomes_AF': {'type': 'double'}, 'gnomad_genomes_AN': {'type': 'integer'}, 'gnomad_genomes_AC': {'type': 'integer'}, 'gnomad_genomes_Hom': {'type': 'integer'}, 'gnomad_genomes_AF_POPMAX_OR_GLOBAL': {'type': 'double'}, 'gnomad_genomes_FAF_AF': {'type': 'double'}, 'gnomad_genomes_Hemi': {'type': 'integer'}, 'hgmd_accession': {'type': 'keyword'}, 'hgmd_class': {'type': 'keyword'}, 'hgmd_like': {'type': 'keyword'}, 'hgsc_wes_AC': {'type': 'integer'}, 'hgsc_wes_AF': {'type': 'double'}, 'hgsc_wes_AN': {'type': 'integer'}, 'hgsc_wgs_AC': {'type': 'integer'}, 'hgsc_wgs_AF': {'type': 'double'}, 'hgsc_wgs_AN': {'type': 'integer'}, 'mainTranscript_biotype': {'type': 'keyword'}, 'mainTranscript_canonical': {'type': 'integer'}, 'mainTranscript_category': {'type': 'keyword'}, 'mainTranscript_cdna_start': {'type': 'integer'}, 'mainTranscript_cdna_end': {'type': 'integer'}, 'mainTranscript_codons': {'type': 'keyword'}, 'mainTranscript_gene_id': {'type': 'keyword'}, 'mainTranscript_gene_symbol': {'type': 'keyword'}, 'mainTranscript_hgvs': {'type': 'keyword'}, 'mainTranscript_hgvsc': {'type': 'keyword'}, 'mainTranscript_major_consequence': {'type': 'keyword'}, 'mainTranscript_major_consequence_rank': {'type': 'integer'}, 'mainTranscript_transcript_id': {'type': 'keyword'}, 'mainTranscript_amino_acids': {'type': 'keyword'}, 'mainTranscript_domains': {'type': 'keyword'}, 'mainTranscript_hgvsp': {'type': 'keyword'}, 'mainTranscript_lof': {'type': 'keyword'}, 'mainTranscript_lof_flags': {'type': 'keyword'}, 'mainTranscript_lof_filter': {'type': 'keyword'}, 'mainTranscript_lof_info': {'type': 'keyword'}, 'mainTranscript_polyphen_prediction': {'type': 'keyword'}, 'mainTranscript_protein_id': {'type': 'keyword'}, 'mainTranscript_sift_prediction': {'type': 'keyword'}, 'mpc_MPC': {'type': 'keyword'}, 'nisc_AC': {'type': 'integer'}, 'nisc_AF': {'type': 'double'}, 'nisc_AN': {'type': 'integer'}, 'originalAltAlleles': {'type': 'keyword'}, 'pos': {'type': 'integer'}, 'primate_ai_score': {'type': 'double'}, 'ref': {'type': 'keyword'}, 'rg37_locus': {'type': 'object', 'properties': {'contig': {'type': 'keyword'}, 'position': {'type': 'integer'}}}, 'rsid': {'type': 'keyword'}, 'samples_ab_0_to_5': {'type': 'keyword'}, 'samples_ab_5_to_10': {'type': 'keyword'}, 'samples_ab_10_to_15': {'type': 'keyword'}, 'samples_ab_15_to_20': {'type': 'keyword'}, 'samples_ab_20_to_25': {'type': 'keyword'}, 'samples_ab_25_to_30': {'type': 'keyword'}, 'samples_ab_30_to_35': {'type': 'keyword'}, 'samples_ab_35_to_40': {'type': 'keyword'}, 'samples_ab_40_to_45': {'type': 'keyword'}, 'samples_gq_0_to_5': {'type': 'keyword'}, 'samples_gq_5_to_10': {'type': 'keyword'}, 'samples_gq_10_to_15': {'type': 'keyword'}, 'samples_gq_15_to_20': {'type': 'keyword'}, 'samples_gq_20_to_25': {'type': 'keyword'}, 'samples_gq_25_to_30': {'type': 'keyword'}, 'samples_gq_30_to_35': {'type': 'keyword'}, 'samples_gq_35_to_40': {'type': 'keyword'}, 'samples_gq_40_to_45': {'type': 'keyword'}, 'samples_gq_45_to_50': {'type': 'keyword'}, 'samples_gq_50_to_55': {'type': 'keyword'}, 'samples_gq_55_to_60': {'type': 'keyword'}, 'samples_gq_60_to_65': {'type': 'keyword'}, 'samples_gq_65_to_70': {'type': 'keyword'}, 'samples_gq_70_to_75': {'type': 'keyword'}, 'samples_gq_75_to_80': {'type': 'keyword'}, 'samples_gq_80_to_85': {'type': 'keyword'}, 'samples_gq_85_to_90': {'type': 'keyword'}, 'samples_gq_90_to_95': {'type': 'keyword'}, 'samples_no_call': {'type': 'keyword'}, 'samples_num_alt_1': {'type': 'keyword'}, 'samples_num_alt_2': {'type': 'keyword'}, 'sortedTranscriptConsequences': {'properties': {'biotype': {'type': 'keyword'}, 'canonical': {'type': 'integer'}, 'cdna_start': {'type': 'integer'}, 'cdna_end': {'type': 'integer'}, 'codons': {'type': 'keyword'}, 'gene_id': {'type': 'keyword'}, 'gene_symbol': {'type': 'keyword'}, 'hgvsc': {'type': 'keyword'}, 'hgvsp': {'type': 'keyword'}, 'transcript_id': {'type': 'keyword'}, 'amino_acids': {'type': 'keyword'}, 'lof': {'type': 'keyword'}, 'lof_filter': {'type': 'keyword'}, 'lof_flags': {'type': 'keyword'}, 'lof_info': {'type': 'keyword'}, 'polyphen_prediction': {'type': 'keyword'}, 'protein_id': {'type': 'keyword'}, 'protein_start': {'type': 'integer'}, 'sift_prediction': {'type': 'keyword'}, 'consequence_terms': {'type': 'keyword'}, 'domains': {'type': 'keyword'}, 'major_consequence': {'type': 'keyword'}, 'category': {'type': 'keyword'}, 'hgvs': {'type': 'keyword'}, 'major_consequence_rank': {'type': 'integer'}, 'transcript_rank': {'type': 'integer'}}, 'type': 'nested'}, 'splice_ai_delta_score': {'type': 'float'}, 'start': {'type': 'integer'}, 'topmed_AC': {'type': 'integer'}, 'topmed_AF': {'type': 'double'}, 'topmed_AN': {'type': 'integer'}, 'topmed_Hom': {'type': 'integer'}, 'topmed_Het': {'type': 'integer'}, 'transcriptConsequenceTerms': {'type': 'keyword'}, 'transcriptIds': {'type': 'keyword'}, 'utrVariantAnnotation': {'properties': {'existing_inframe_oorfs': {'type': 'integer'}, 'existing_outofframe_oorfs': {'type': 'integer'}, 'existing_uorfs': {'type': 'integer'}, 'five_prime_utr_variant_consequence': {'type': 'keyword'}, 'five_prime_utr_variant_annotation': {'type': 'keyword'}}, 'type': 'nested'}, 'variantId': {'type': 'keyword'}, 'xpos': {'type': 'long'}, 'xstart': {'type': 'long'}, 'xstop': {'type': 'long'}}, '_meta': {'gencodeVersion': '25', 'sourceFilePath': 's3://grch38_subset.vcf', 'genomeVersion': '38', 'sampleType': 'WGS', 'hail_version': '0.2.57'}}

Could you suggest on how I should change the mappings of the _meta for it to work?

2

Hi @NLSVTN Just a bit of a syntax error you have the "_meta" at the wrong level it goes inside the "mappings" you have i outside.

PUT discuss
{
  "mappings": {
    "properties": {
      "aIndex": {
        "type": "integer"
      },
      "AC": {
        "type": "integer"
      },
      "AF": {
        "type": "double"
      }
    },
    "_meta": {
      "gencodeVersion": "25"
    }
  }
}

# Result
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "discuss"
}

GET discuss

#Result

{
  "discuss": {
    "aliases": {},
    "mappings": {
      "_meta": {
        "gencodeVersion": "25"
      },
      "properties": {
        "AC": {
          "type": "integer"
        },
        "AF": {
          "type": "double"
        },
        "aIndex": {
          "type": "integer"
        }
      }
    },
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "number_of_shards": "1",
        "provided_name": "discuss",
        "creation_date": "1659155996920",
        "number_of_replicas": "1",
        "uuid": "79mz2pieRVel4u3Q9GSYYQ",
        "version": {
          "created": "8030299"
        }
      }
    }
  }
}
5

It turned out that somehow an older version of ES was used even though we installed ES7, so adding additional level between mappings and _meta solved the issue:

'mappings': {
     'variant': {
          '_meta': {...}
      },
 },
6

Glad it works.. interesting that is different than the doc show here

Unless you are going all the way back to 6.X here and that is a bit different as well here

7

Yeah, it looks like we can put all mapping under an arbitrary named _doc, not only _doc.

8

It is because version 6.X still support types and the type can be anything, but just one type per index, in your case it seems that the type name seems to be variant.