ArchLinux package: Missing commons beat fields like @timestamp in template

Hi,

I did setup a new elasticSearch with kibana and some beats, but I get an error when trying to create a indexPattern: "The indices which match this index pattern don't contain any time fields." As most discussion I found about this kind of error show that it comes from a wrong setup, I reinstalled everything again but configuring only heartbeat to test this issue. Doing more search it seems that the common beat fields are missing from the field.xml provided. (those: https://github.com/elastic/beats/blob/master/libbeat/_meta/fields.common.yml )

and here is my full "GET _template":

{
"heartbeat-6.2.3": {
"order": 1,
"index_patterns": [
  "heartbeat-6.2.3-*"
],
"settings": {
  "index": {
    "codec": "best_compression",
    "mapping": {
      "total_fields": {
        "limit": "10000"
      }
    },
    "refresh_interval": "5s",
    "number_of_routing_shards": "30",
    "number_of_shards": "1"
  }
},
"mappings": {
  "doc": {
    "dynamic_templates": [
      {
        "strings_as_keyword": {
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {
      "monitor": {
        "properties": {
          "scheme": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "host": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "ip": {
            "type": "ip"
          },
          "status": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "duration": {
            "properties": {
              "us": {
                "type": "long"
              }
            }
          }
        }
      },
      "resolve": {
        "properties": {
          "host": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "ip": {
            "type": "ip"
          },
          "rtt": {
            "properties": {
              "us": {
                "type": "long"
              }
            }
          }
        }
      },
      "socks5": {
        "properties": {
          "rtt": {
            "properties": {
              "connect": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              }
            }
          }
        }
      },
      "tls": {
        "properties": {
          "rtt": {
            "properties": {
              "handshake": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              }
            }
          }
        }
      },
      "http": {
        "properties": {
          "url": {
            "type": "text",
            "norms": false
          },
          "response": {
            "properties": {
              "status": {
                "type": "long"
              }
            }
          },
          "rtt": {
            "properties": {
              "validate": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              },
              "validate_body": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              },
              "write_request": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              },
              "response_header": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              },
              "total": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              }
            }
          }
        }
      },
      "icmp": {
        "properties": {
          "requests": {
            "type": "long"
          },
          "rtt": {
            "properties": {
              "us": {
                "type": "long"
              }
            }
          }
        }
      },
      "tcp": {
        "properties": {
          "port": {
            "type": "long"
          },
          "rtt": {
            "properties": {
              "connect": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              },
              "validate": {
                "properties": {
                  "us": {
                    "type": "long"
                  }
                }
              }
            }
          }
        }
      }
    },
    "_meta": {
      "version": "6.2.3"
    },
    "date_detection": false
  }
},
"aliases": {}
}
}

Some more info that might be relevant: I installed on a virtual box vm running Archlinux. I used the package from the Arch repo. I did the minimal config change in my second setup, mostly updating IP to get everything connected and adding some hosts to monitor in heartbeat.

In my first setup I worked around this by creating my own field.xml with the @timestamp added, but I'd like to fix this as it's cumbersome to do that for every beat on every update.

Regards,

The /etc/heartbeat/fields.yml file is generated at packaging time and should contain all of the common fields. A quick inspection of Heartbeat 6.2.3 downloaded from elastic.co or from our repos shows that common fields like @timestamp are present. Would it be possible to use one of our official packages?

I can't speak to the ArchLinux package because we don't maintain them. But it does look like the package is broken. I think you should report an issue in the archlinux issue tracker.

Thank you for this checkup. As I can confirm that the file in the package seam to be wrong, I'll open an issue with arch packager. In the mean time I'll keep going with the manual workaround, probably using the fields.xml from the official distribution.

Thank again

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.