# Filebeat - unknown index name/nginx log formatting

**URL:** <https://discuss.elastic.co/t/filebeat-unknown-index-name-nginx-log-formatting/248828>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 16, 2020, 12:33pm UTC](https://discuss.elastic.co/t/filebeat-unknown-index-name-nginx-log-formatting/248828 "2020-09-16T12:33:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![vinci](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vinci](https://discuss.elastic.co/u/vinci)\
**Post date:** [September 16, 2020, 12:33pm UTC](https://discuss.elastic.co/t/filebeat-unknown-index-name-nginx-log-formatting/248828/1 "2020-09-16T12:33:49Z")

</div>

Hello,

I have the following configuration file for filebeat:

```
    # filebeat.yml for harvesting docker logfiles
    #=========================== Filebeat inputs =============================
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
       - "/var/log/*.log"
      module:
        - system
      fields_under_root: true

#==================== Elasticsearch template setting ==========================
setup.ilm.enabled: true
setup.ilm.rollover_alias: "dms-logs"

#================================ General =====================================

# The name of the Beat. If this option is empty, the hostname of the server is used.
# The name is included as the agent.name field in each published transaction.
# You can use the name to group all transactions sent by a single Beat.
#name: "{{ inventory_hostname }}""

# The tags of the shipper are included in their own field with each
# transaction published.
# tags: ["dev"]

# Optional fields that you can specify to add additional information to the output.
#fields:
# env: dev

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["dev-logging.my_domain-webtools.eu:9200"]

#================================ Processors =====================================
processors:
# - add_docker_metadata: ~
  - decode_json_fields:
      fields: ["message"]
      target: "app"
      overwrite_keys: true
      add_error_key: true
# wenn fields aus den obigen json logs nicht unter dem key "alf" liegen sollen, muss file gedropped werden, da es sonst knallt
# - drop_fields:
# when:
# contains:
# container.image.name: "adb-eakte"
# fields: ["file"]
# ignore_missing: true

logging.level: info
logging.to_syslog: false
logging.to_files: true
logging.to_stderr: true

# Autodiscovery, wenn nur einzelne Docker Container gescannt werden sollen

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: nginx
          config:
            - type: container
              paths:
                - '/var/lib/docker/containers/${data.docker.container.id}/*.log'
            - module: nginx
              access:
                enabled: true
                containers:
                  stream: "stdout"
              error:
                enabled: true
                containers:
                  stream: "stderr"
        - condition:
            contains:
              docker.container.image: postgres
          config:
            - type: container
              paths:
                - '/var/lib/docker/containers/${data.docker.container.id}/*.log'
            - module: postgresql
        - condition:
            equals:
              docker.container.labels:
                eu.my_domainkom.application: "dms"
          config:
            - type: container
              paths:
                - '/var/lib/docker/containers/${data.docker.container.id}/*.log'

setup.kibana:
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify an additional path, the scheme is required: http://localhost:5601/path
  host: "http://dev-logging.my_domain-webtools.eu"

setup.dashboards.enabled: true

```

I have two questions:

1. why does filebeat create a default 'filebeat' index whenever I run set up?
2. are the module directive correctly added in the configuration? For instance, for nginx I get logs scraped from the nginx container, but the in discovery I see a whole message like:

```auto
	some_ip_address - - [16/Sep/2020:12:31:12 +0000] "https://dms-dev.breitbandausschreibungen.de" "POST /alfresco/api/-default-/public/search/versions/1/search HTTP/1.1" 200 122 "-" "-"

```

Shouldn't these chunks be formatted and separeted by IP, HTTP Method, URL and all that?  
Or where should I be able to see that? At what point do I create the separation?

Thanks!

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [September 16, 2020, 1:46pm UTC](https://discuss.elastic.co/t/filebeat-unknown-index-name-nginx-log-formatting/248828/2 "2020-09-16T13:46:55Z")

</div>

> 1. why does filebeat create a default 'filebeat' index whenever I run set up?

I don't understand your question. During the setup phase all indices and ingest pipelines must be installed.

> 1. are the module directive correctly added in the configuration? For instance, for nginx I get logs scraped from the nginx container, but the in discovery I see a whole message like:

Did you check documents stored in indices? Do they contain proper fields?

---

<div class="post-metadata">

**Author:** ![vinci](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vinci](https://discuss.elastic.co/u/vinci)\
**Post date:** [September 16, 2020, 2:10pm UTC](https://discuss.elastic.co/t/filebeat-unknown-index-name-nginx-log-formatting/248828/3 "2020-09-16T14:10:30Z")

</div>

1. The index that I'm using is called "dms-logs-\*", this is what I'm querying and currently using.  
But during the setup it adds the "filebeat" index. I would like to remove that.

2. I'm not sure how I can query the documents in the indices by the proper fields. Perhaps you could give me a hand with that.

Thanks!

---

<div class="post-metadata">

**Author:** ![vinci](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vinci](https://discuss.elastic.co/u/vinci)\
**Post date:** [September 21, 2020, 7:28am UTC](https://discuss.elastic.co/t/filebeat-unknown-index-name-nginx-log-formatting/248828/4 "2020-09-21T07:28:59Z")

</div>

Is there an unwritten rule on this forum? I've noticed this on several occasions - helping only people who kind of already know the answer?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2020, 9:29am UTC](https://discuss.elastic.co/t/filebeat-unknown-index-name-nginx-log-formatting/248828/5 "2020-10-19T09:29:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.