# Problem with indexes - Logstash dont create filebeat indexes

**URL:** <https://discuss.elastic.co/t/problem-with-indexes-logstash-dont-create-filebeat-indexes/91117>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 28, 2017, 2:21pm UTC](https://discuss.elastic.co/t/problem-with-indexes-logstash-dont-create-filebeat-indexes/91117 "2017-06-28T14:21:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![rwagner](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rwagner](https://discuss.elastic.co/u/rwagner)\
**Post date:** [June 28, 2017, 2:21pm UTC](https://discuss.elastic.co/t/problem-with-indexes-logstash-dont-create-filebeat-indexes/91117/1 "2017-06-28T14:21:10Z")

</div>

I am not able to create indexes using filebeat on windows. Apparently logstash is not even getting the logs.

I disabled the firewalld and selinux on the server that is logstash + elasticsearch, I can also see the communications coming via tcpdump, but the indicies are not created. I also disabled the firewall on windows, which is the machine that is sending the logs, but I was not successful either.

Filebeat configuration:

```
filebeat.modules:

filebeat.prospectors:
- input_type: log

  paths:
    - C:\Program Files\filebeat\daily-server.json

  fields:
  fields_under_root: true
  document_type: json
  json.keys_under_root: true
  json.overwrite_keys: true

output.logstash:
  enabled: true
  hosts: ["192.168.1.130:5001"]
  ssl.enabled: true
  ssl.certificate_authorities: ['C:\cert\logstash.crt']

```

Logstash configuration:

```
input {

  tcp {
    type => "daily"
    port => "5007"
    codec => "json"
  }

    beats {
        port => 5001
        codec => "json_lines"
        ssl => true
        ssl_certificate => "/etc/logstash/logstash.crt"
        ssl_key => "/etc/logstash/logstash.key"
    }

}
filter {

}

output {
   elasticsearch {
   hosts => "127.0.0.1:9200"
   index => "dailyserver-%{+YYYY.MM.dd}"
# document_type => "dailyserver"
}
}

```

In the configuration it is with ssl, but also I have already tested without ssl.

I made a video showing the problem. [https://youtu.be/2vhpNZC1LSo](https://youtu.be/2vhpNZC1LSo)

Can someone help?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [June 28, 2017, 8:05pm UTC](https://discuss.elastic.co/t/problem-with-indexes-logstash-dont-create-filebeat-indexes/91117/2 "2017-06-28T20:05:29Z")

</div>

> [@rwagner](#):
>
> codec =\> "json\_lines"

Try removing the json\_lines codec from the beats input in Logstash.

---

<div class="post-metadata">

**Author:** ![rwagner](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rwagner](https://discuss.elastic.co/u/rwagner)\
**Post date:** [June 29, 2017, 2:27pm UTC](https://discuss.elastic.co/t/problem-with-indexes-logstash-dont-create-filebeat-indexes/91117/3 "2017-06-29T14:27:09Z")

</div>

Hi @andrewkroh, it worked, logstash is logging in, but it is not separating the json log fields, as can be seen in kibana's print.

 ![](https://us1.discourse-cdn.com/elastic/original/3X/e/e/eed4cc46aa63fac75bffa748463635127217bf64.JPG)

I would like the fields to be divided.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [June 29, 2017, 2:34pm UTC](https://discuss.elastic.co/t/problem-with-indexes-logstash-dont-create-filebeat-indexes/91117/4 "2017-06-29T14:34:22Z")

</div>

> [@rwagner](#):
>
> fields:  
> fields\_under\_root: true  
> document\_type: json  
> json.keys\_under\_root: true  
> json.overwrite\_keys: true

There's something strange going on with your config. I think the empty `fields:` setting is causing issues (or maybe there's an indentation issue). If you are not using it just remove it or set it to null (`fields: ~`). As you can see you are getting those `fields.*` keys showing up in Elasticsearch. This is causing filebeat to never honor the JSON decoding that you enabled.

---

<div class="post-metadata">

**Author:** ![rwagner](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rwagner](https://discuss.elastic.co/u/rwagner)\
**Post date:** [June 29, 2017, 6:05pm UTC](https://discuss.elastic.co/t/problem-with-indexes-logstash-dont-create-filebeat-indexes/91117/5 "2017-06-29T18:05:20Z")

</div>

Okay @andrewkroh , I commented the lines on the field and it worked. But I have another problem, the logs are from a java application, and when the log has the stack trace it has \_jsonparsefailure, and it can not split the json log fields.

Example:  
In this log the fields have been divided:

> {"level":"INFO","timestamp":"2017-06-29T14:28:44,358","thread":"noup-3-5","file":"Abstndler.java", "line":"192","message":"Sess at -\> io.netty.channel.text@527a8fc","throwable":""}
> 
> ![](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8f5e35b5ccb0d25de0488753aed6ebe9962a76b0.JPG)

In this log the fields were not divided:

> {"level":"INFO","timestamp":"2017-06-29T13:58:22,440","thread":"pool-22-thread-1","file":"con.java", "line":"101","message":" ms.TimeoutException","throwable":"[br.com](http://br.com). ms.exception.TimeoutException: ms.TimeoutException\n at [br.com](http://br.com). ms.transport.nio.bytes.NioPeer.read(NioPeer.java:155) ~[core.jar:?]\n at [br.com](http://br.com). ms.transport.nio.bytes.NioChannel.read(NioChannel.java:65) ~[core.jar:?]\n at [br.com](http://br.com). phast.ms.transport.dec.read(dec.java:101) ~[ddd.jar:?]\n at [br.com](http://br.com). b.readMessage(b.java:360) ~[core.jar:?]\n at br.com.b.read(b.java:240) ~[core.jar:?]\n at [br.com](http://br.com). SingleSessionConnection.handleInputConnection(SingleSessionConnection.java:98) ~[core.jar:?]\n at [br.com](http://br.com). SingleSessionConnection.handle(SingleSessionConnection.java:72) ~[core.jar:?]\n at [br.com](http://br.com). bManager.connect(bManager.java:169) ~[core.jar:?]\n at [br.com](http://br.com). BaseConnector.connect(BaseConnector.java:35) ~[core.jar:?]\n at [br.com](http://br.com). ms.transport.BaseBinder.bind(BaseBinder.java:58) ~[core.jar:?]\n at [br.com](http://br.com). ms.transport.nio.bytes.NioPeer.bind(NioPeer.java:196) ~[core.jar:?]\n at [br.com](http://br.com). ms.transport.nio.netty.ServerHandler$1.run(java:52) ~[core.jar:?]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.0]\n at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.0]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.0]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.0]\n at java.lang.Thread.run(Thread.java:745) [?:1.0]\n"}

 ![](https://us1.discourse-cdn.com/elastic/original/3X/8/7/87cf78d5d7d4e9e25e7c3c5197dff9b57d98fcf0.JPG)

The developer was able to put a **\n** (marked in the image) to try to break the line in the throwable field, which is where the stack trace, but did not work. It only works if you put the **\n** along with the string where the next line would start, but the developer can not do that.

Do you have any ideas that might help @andrewkroh ?

I do not know if it was obvious, my English is a bit bad. 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 27, 2017, 6:05pm UTC](https://discuss.elastic.co/t/problem-with-indexes-logstash-dont-create-filebeat-indexes/91117/6 "2017-07-27T18:05:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.