# Need configuration to send JSON string to ELK server through Filebeat

**URL:** https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493
**Category:** Beats
**Tags:** filebeat
**Created:** [January 23, 2017, 2:39pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493 "2017-01-23T14:39:05Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![Honda\_fred\_elk](https://avatars.discourse-cdn.com/v4/letter/h/919ad9/32.png) [@Honda\_fred\_elk](https://discuss.elastic.co/u/Honda_fred_elk)
#### Post date: [January 23, 2017, 2:39pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493/1 "2017-01-23T14:39:05Z")

</div>

Hello,

I am trying to export log message (JSON) from client side log to ELK Server through FileBeat.

1. My custom log (log4j) will produce JSON output in a file through log4j.RollingFileAppender. I know, I have to have only one JSON String (Object) per line.
2. I want Filebeat to read JSON line by line from the log file and send to Remote ELK Server.
3. File beat should not miss any JSON line, even though log4j rotates the log file
4. I have installed Filebeat in my client machine.

I just need to know, a concreate settings that I have to set in filebeat.yml file.

Thanks..

---

<div class="post-metadata">

### Author: ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [January 23, 2017, 5:28pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493/2 "2017-01-23T17:28:55Z")

</div>

See [https://www.elastic.co/guide/en/beats/filebeat/5.1/configuration-filebeat-options.html#config-json](https://www.elastic.co/guide/en/beats/filebeat/5.1/configuration-filebeat-options.html#config-json) for documentation of all the options. Here's a brief example.

```auto
filebeat.prospectors:
- paths:
    - /var/log/path/to/input.json
  document_type: myapp
  json.keys_under_root: true
  json.add_error_key: true

output.elasticsearch:
  hosts: ['http://localhost:9200']

```

---

<div class="post-metadata">

### Author: ![Honda\_fred\_elk](https://avatars.discourse-cdn.com/v4/letter/h/919ad9/32.png) [@Honda\_fred\_elk](https://discuss.elastic.co/u/Honda_fred_elk)
#### Post date: [January 24, 2017, 8:26pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493/3 "2017-01-24T20:26:56Z")

</div>

Thank you so much Andrew.. Instead of elastic search I send the JSON string to logstash

I have following settings in filebeat.yml

```auto
filebeat.prospectors:
- paths:
    - <Windows path>jsonoutput.log
  document_type: log
  json.keys_under_root: true
  json.add_error_key: true

output.logstash:
  hosts: ['http://<logstashserver-ip>:6000']

```

below is the configuration settings in logstash and logstash starts without any issues.

```auto
input {
  beats {
     port => 6000
     tags => "beats"
     codec => "json_lines"
  }	
}

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout{}  
  file{
      path => "/appl/log/TestLogsOutput/LogStash_output.log"
  }
}

```

When I start Filebeat, it starts without any error. But once .log file generated JSON string as log.  
It tries to establish connection to logstash.. and throughs following error..

`WARN DNS lookup failure "http://<logstashserver-ip>:6000": lookup http://<logstashserver-ip>:6000: getaddrinfow: No such host is known. ERR Connecting error publishing events (retrying): lookup http://<logstashserver-ip>:6000: getaddrinfow: No such host is known.`

I am able telnet from my window laptop (where I run the filebeat) to logstash server..

Could you help me, what could be the reason?

Thanks

---

<div class="post-metadata">

### Author: ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [January 24, 2017, 8:35pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493/4 "2017-01-24T20:35:33Z")

</div>

Checkout the instructions for configuring Filebeat and Logstash. Let us know if you have problems after following the config examples and instructions there.

[Configuring Filebeat to Use Logstash](https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html)  
[Setting Up Logstash for Beats](https://www.elastic.co/guide/en/beats/libbeat/5.1/logstash-installation.html#logstash-setup)

---

<div class="post-metadata">

### Author: ![Honda\_fred\_elk](https://avatars.discourse-cdn.com/v4/letter/h/919ad9/32.png) [@Honda\_fred\_elk](https://discuss.elastic.co/u/Honda_fred_elk)
#### Post date: [January 25, 2017, 3:01pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493/5 "2017-01-25T15:01:58Z")

</div>

Thank you Andrew.

Everything fine except the URL.. instead of giving like below

```auto
output.logstash:
  hosts: ['http://logstashserver-ip:6000']

```

I changed to

```auto
output.logstash:
  hosts: ['logstashserver-ip:6000']

```

Now it works absolutely fine..

Thanks

---

<div class="post-metadata">

### Author: ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [January 25, 2017, 3:47pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493/6 "2017-01-25T15:47:22Z")

</div>

> [@Honda\_fred\_elk](#):
>
> codec =\> "json\_lines"

You shouldn't' need that line either since you are doing JSON decoding in Filebeat.

---

<div class="post-metadata">

### Author: ![Honda\_fred\_elk](https://avatars.discourse-cdn.com/v4/letter/h/919ad9/32.png) [@Honda\_fred\_elk](https://discuss.elastic.co/u/Honda_fred_elk)
#### Post date: [January 25, 2017, 6:22pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493/7 "2017-01-25T18:22:04Z")

</div>

I am facing another problem now.  
I have given specific file name to read JSON string. so below the config.

```auto
filebeat.prospectors:
- paths:
    - C:\appl\log\GALCLogs\Server-jsonoutput.log

```

This file written by Log4j framework.. and it rotates as soon as it attains the specified file size.  
because of this, file beat looses bottom 20 lines of log messages (JSON String) on every rotation.

Has anyone faced this problem? How this could be fixed?

Thanks

---

<div class="post-metadata">

### Author: ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [January 25, 2017, 7:31pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493/8 "2017-01-25T19:31:01Z")

</div>

Well if you use a [glob pattern](https://www.elastic.co/guide/en/beats/filebeat/5.1/configuration-filebeat-options.html#_paths) that also matches the rotated files, then filebeat can follow the rotated file and continue reading it to the end. Try this if your rotation moves x.log to x.log.1.

```auto
filebeat.prospectors:
- paths:
    - 'C:\appl\log\GALCLogs\Server-jsonoutput.log*'

```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 22, 2017, 7:31pm UTC](https://discuss.elastic.co/t/need-configuration-to-send-json-string-to-elk-server-through-filebeat/72493/9 "2017-02-22T19:31:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.