# Create multiple indexes

**URL:** <https://discuss.elastic.co/t/create-multiple-indexes/346629>\
**Category:** Elasticsearch\
**Created:** [November 7, 2023, 2:48pm UTC](https://discuss.elastic.co/t/create-multiple-indexes/346629 "2023-11-07T14:48:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrispos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrispos/32/116774_2.png) [@chrispos](https://discuss.elastic.co/u/chrispos)\
**Post date:** [November 7, 2023, 2:48pm UTC](https://discuss.elastic.co/t/create-multiple-indexes/346629/1 "2023-11-07T14:48:05Z")

</div>

I am trying to send two json logs from server A to my logstash server. Previously, when we only sent a json log file from server A to the logstash server,  
it was possible to create a data view and view the log in discover. But by sending two json log files it seems that I am not receiving anything now.  
Below my filebeat config and below that my logstash config.  
Does anyone have an idea what I might be doing wrong ? I am supposed to get two different indexes.

Elastic version 8.1

```auto
 input {
   beats {
     port => 5044
   }
}

filter {
   if [document_type] == "test.json" { # Controleer op het "document_type" veld
     json {
       source => "message"
     }
     mutate {
       remove_field => ["sequence", "loggerClassName", "ndc", "processName", "processId"]
     }
   }
   else if [document_type] == "abc.json" { # Controleer op het "document_type" veld
     json {
       source => "message"
     }
     mutate {
       remove_field => ["sequence", "loggerClassName", "ndc", "processName", "processId"]
     }
   }
}

output {
   if [document_type] == "test.json" { # Controleer op het "document_type" veld
     elasticsearch {
       hosts => ["XXX.XXX.XXX.XXX:9200"]
       index => "test.logs-%{+YYYY.MM.dd}"
       user => "admin"
       password => "unknown"
     }
   }
   else if [document_type] == "abc.json" { # Controleer op het "document_type" veld
     elasticsearch {
       hosts => ["XXX.XXX.XXX.XXX:9200"]
       index => "abc.server-logs-%{+YYYY.MM.dd}"
       user => "admin"
       password => "unknown"
     }
   }
}
```

filbeat.conf.yaml

```auto
filebeat.inputs:
  - type: log
    enabled: true
    paths:
     - /data/test.json
    multiline.pattern: '^{'
    multiline.negate: true
    multiline.match: after
    scan_frequency: 60000
    fields:
      type: "test.json" # Voeg hier een "type" veld toe
  - type: log
    enabled: true
    paths:
     - /data/abc.json
    multiline.pattern: '^{'
    multiline.negate: true
    multiline.match: after
    scan_frequency: 60000
    fields:
      type: "abc.json" # Voeg hier een "type" veld toe

processors:
- drop_fields:
    fields: ["sequence", "loggerClassName", "ndc", "processName", "processId"]

output.logstash:
  hosts: ["XXX.XXX.XXX.XXX:5044"]
```

Thanks in advance

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 7, 2023, 3:03pm UTC](https://discuss.elastic.co/t/create-multiple-indexes/346629/2 "2023-11-07T15:03:52Z")

</div>

Hello,

Please share a sample of your messages before logstash and how the output looks like.

But there are a couple of things that seems wrong.

First, you are not parsing the message from beats, you need to parse it.

Add a json filter as your first filter.

```auto
filter {
    json {
        source => "message"
        remove_field => "message"
    }
}

```

Second, where is the `document_type` field coming from? In filebeat you are adding a field named `type`, this will be added to your document as `fields.type`, not `document_type`.

So your conditionals should use `[fields][type]` in your output block.

Also, from what you shared you do not need those conditionals on your `filter` block, they won't work because the `document_type` field does not exist, and even if you just change it to `[fields][type]` they will also not work because the message was not parsed, and since the conditionals are used to parse the message you can change your filter block to just this:

```auto
filter {
    json {
        source => "message"
        remove_field => "message"
    }
    mutate {
        remove_field => ["sequence", "loggerClassName", "ndc", "processName", "processId"]
    }
}

```

On your filebeat configuration I don't think this does anything, you are not parsing your json document on filebat, so those fields would not exist on filebeat.

> [@chrispos](#):
>
> ```auto
> processors:
> - drop_fields:
> fields: ["sequence", "loggerClassName", "ndc", "processName", "processId"]
> 
> ```

---

<div class="post-metadata">

**Author:** ![chrispos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrispos/32/116774_2.png) [@chrispos](https://discuss.elastic.co/u/chrispos)\
**Post date:** [November 7, 2023, 3:48pm UTC](https://discuss.elastic.co/t/create-multiple-indexes/346629/3 "2023-11-07T15:48:55Z")

</div>

Hello,  
First of all thanks for your reply.  
Here are two examples. Mostly we get short / normal json messages, but sometimes also with a multiline stacktrace

```auto
{
    "timestamp": "2023-11-07T06:23:44.777+01:00",
    "sequence": 50441671,
    "loggerClassName": "logger",
    "loggerName": "test",
    "level": "DEBUG",
    "message": "hello",
    "threadName": "default task-1704",
    "threadId": 3876,
    "mdc": {
    },
    "ndc": "",
    "hostName": "google.com.just.named.something",
    "processName": "dunno",
    "processId": 1640
}
```

```auto
{
    "timestamp": "2023-11-07T06:23:44.777+01:00",
    "sequence": 50441671,
    "loggerClassName": "logger",
    "loggerName": "test",
    "level": "DEBUG",
    "message": "hello",
    "threadName": "default task-1704",
    "threadId": 3876,
    "mdc": {
    },
    "ndc": "",
    "hostName": "google.com.just.named.something",
    "processName": "dunno",
    "processId": 1640
    "stackTrace": ": org. cxf.interceptor.Fault: Unmarshalling Error: 
	cvc-pattern-valid: Value 'wat’ 
	is not facet-valid with respect to multiline stacktrace much longer then this example
}
```

The document\_type i found on the internet. I got a warning in my logstash logging “type` event field won't be used to determine the document \_type {:es\_version=\>8}” and i didnt know what to anymore, so i tryed something. I first used the type field to distinguish between two different log files for logstash, but that doesn't work.  
I will try to make changes tonight, i will keep you posted.

---

<div class="post-metadata">

**Author:** ![chrispos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrispos/32/116774_2.png) [@chrispos](https://discuss.elastic.co/u/chrispos)\
**Post date:** [November 15, 2023, 2:13pm UTC](https://discuss.elastic.co/t/create-multiple-indexes/346629/5 "2023-11-15T14:13:20Z")

</div>

Topic can get closed.

leandrojmp helped me enough with his comments got it working now. Tx!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2023, 2:13pm UTC](https://discuss.elastic.co/t/create-multiple-indexes/346629/6 "2023-12-13T14:13:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.