Elastic doesn't create a new index

Elastic Stack Elasticsearch
1

Hi, and thanks for any leads in advance.
I am using the following output configuration:

<match **>
      @type                     rewrite_tag_filter
      @label                    @LOGTYPE
      <rule>
        key                     $["logtype"]
        pattern                 ^applog$
        tag                     applog
      </rule>
      <rule>
        key                     $["logtype"]
        pattern                 ^applog$
        tag                     system.${tag}
        invert                  true
      </rule>
    </match>

    <label @LOGTYPE>
    <match applog>
      @type                     relabel
      @label                    @APPLOG
    </match>
    <match system.**>
      @type                     relabel
      @label                    @SYSTEM
    </match>
    </label>

    <label @APPLOG>
    <match applog>
      @type                          elasticsearch
      @include                     out.conf
      logstash_prefix           log.applog
    </match>
    </label>

    <label @SYSTEM>
    <match system.**>
      @type                           elasticsearch
      @include                      out.conf
      logstash_prefix           fluentd.k8s
    </match>
    </label>

The symptom is the fact that system.** logs get to Kibana (so rewrite-tag-filter works), but no logs of applog origin are pushed to log.applog index. I should note that the permissions on the user used to connect are admin, and log.applog is yet-uncreated index which I expect to be created (as I've witnessed with previous configurations). Elastic is configured to allow creation of new indices.
Elastic is deployed via fluentd-elasticsearch helm chart. What steps can I perform in order to troubleshoot this issue? Can't seem to find Elastic's logs anywhere.
BTW, already tried @type stdout as well, doesn't work. Yes, there are applogs being created by the system, I see them when I reroute them to @SYSTEM label. Incredible voodoo, level 85.

2

May be it's more a question for fluentd? Not sure we can help here unless someone from the community already implemented that...

3

How and where can I see Elastic's logs? If I enable level trace, I still can't see any logs anywhere.
Where should they be?

4

I doubt it's fluentd's side. As I've noted, when I route all the logs to @SYSTEM I see them in Kibana. With the changed tag and everything. Meaning, fluentd's side is working as expected.
Where can I find logs for Elastic actions? Debug/trace/what-have-you.

5

To briefly explain my answer. I have absolutely no idea what the xml content you shared initially means. It's not coming from any Elastic product configuration AFAIK but I might be wrong.

About Elasticsearch:

It depends on how you installed elasticsearch. For example, here are the path used by debian:

So it's in /var/log/elasticsearch by default.
You can change the log levels with this:

If you want to trace everything which is happening, probably something like:

PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch": "trace"
  }
}

will do it but I don't think you want this as it will be extremely verbose.

6

Will try trace config, thanks.