# Use logstash or filebeat for sending azure JSON logs?

**URL:** <https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 9, 2017, 2:09am UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456 "2017-02-09T02:09:16Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 9, 2017, 2:09am UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/1 "2017-02-09T02:09:16Z")

</div>

Hello, sorry for silly question but I was trying about a week to adjust logstash 5.2 config for sending azure JSON logs to elastic 5.2, but all my attempts were failed. Logstash throws many different exceptions and errors that config is wrong and JSON files can not be parsed, although I am using logstash-input-azureblob module installed, which can get this files from the blob, but can not parse them and send to elastic by some unclear reason.  
Yesterday an article about Filebeat caught my eye. But I am not sure that Filebeat 5.2 can parse JSON bunch of log files and ship them to Elastic - Kibana properly without any conversion so as I can see only \*.log extension in Filebeat.yml unfortunately.  
So, the question is what product should I choose to make it work with minimal handy changes in config files so as I have no time for it anymore and relevant knowledge how to do it? Please advise.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [February 10, 2017, 9:01am UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/2 "2017-02-10T09:01:34Z")

</div>

Have a look at this blog post here about JSON logs [https://www.elastic.co/blog/structured-logging-filebeat](https://www.elastic.co/blog/structured-logging-filebeat) and the filebeat docs: [https://www.elastic.co/guide/en/beats/filebeat/5.2/configuration-filebeat-options.html#config-json](https://www.elastic.co/guide/en/beats/filebeat/5.2/configuration-filebeat-options.html#config-json)

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 10, 2017, 12:17pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/3 "2017-02-10T12:17:16Z")

</div>

2017-02-10T12:15:37Z INFO Home path: [C:\filebeat-5.2.0-windows-x86\_64] Config path: [C:\filebeat-5.2.0-windows-x86\_64] Data path: [C:\ProgramData\filebeat] Logs path: [C:\filebeat-5.2.0-windows-x86\_64\logs]  
2017-02-10T12:15:37Z INFO Setup Beat: filebeat; Version: 5.2.0  
2017-02-10T12:15:37Z INFO Loading template enabled. Reading template file: C:\filebeat-5.2.0-windows-x86\_64\filebeat.template.json  
2017-02-10T12:15:37Z INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\filebeat-5.2.0-windows-x86\_64\filebeat.template-es2x.json  
2017-02-10T12:15:37Z INFO Elasticsearch url: [http://localhost:9200](http://localhost:9200)  
2017-02-10T12:15:37Z INFO Activated elasticsearch as output plugin.  
2017-02-10T12:15:37Z INFO Publisher name: data-vm0  
2017-02-10T12:15:37Z INFO Flush Interval set to: 1s  
2017-02-10T12:15:37Z INFO Max Bulk Size set to: 50  
2017-02-10T12:15:37Z INFO filebeat start running.  
2017-02-10T12:15:37Z INFO Registry file set to: C:\ProgramData\filebeat\registry  
2017-02-10T12:15:37Z INFO Loading registrar data from C:\ProgramData\filebeat\registry  
2017-02-10T12:15:37Z INFO States Loaded from registrar: 0  
2017-02-10T12:15:37Z INFO Loading Prospectors: 1  
2017-02-10T12:15:37Z INFO Prospector with previous states loaded: 0  
2017-02-10T12:15:37Z INFO Loading Prospectors completed. Number of prospectors: 1  
2017-02-10T12:15:37Z INFO All prospectors are initialised and running with 0 states to persist  
2017-02-10T12:15:37Z INFO Starting Registrar  
2017-02-10T12:15:37Z INFO Start sending events to output  
2017-02-10T12:15:37Z INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s  
2017-02-10T12:15:37Z INFO Starting prospector of type: log  
2017-02-10T12:16:07Z INFO No non-zero metrics in the last 30s

I have adjusted filebat.yml and filebeat.full.yml files with settings for json files, but nothing is happening.  
Still can not see any data in Kibana. [http://rgho.st/6W2FwRtMg](http://rgho.st/6W2FwRtMg)

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 10, 2017, 1:48pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/4 "2017-02-10T13:48:49Z")

</div>

Please advise why does filebeat skip directory with json files inside? It should process them but not skip.

**PS C:\filebeat-5.2.0-windows-x86\_64\> .\filebeat.exe -c filebeat.yml -e -v -d "\*"**  
2017/02/10 13:46:17.053776 beat.go:267: INFO Home path: [C:\filebeat-5.2.0-windows-x86\_64] Config path: [C:\filebeat-5.2.0-windows-x86\_64] Data path: [C:\filebeat-5.2.0-windows-x86\_64\data] Logs path: [C:\filebeat-5.2.0-windows-x86\_64\logs]  
2017/02/10 13:46:17.053776 logp.go:219: INFO Metrics logging every 30s  
2017/02/10 13:46:17.053776 beat.go:177: INFO Setup Beat: filebeat; Version: 5.2.0  
2017/02/10 13:46:17.053776 processor.go:43: DBG Processors:  
2017/02/10 13:46:17.053776 beat.go:183: DBG Initializing output plugins  
2017/02/10 13:46:17.053776 output.go:167: INFO Loading template enabled. Reading template file: C:\filebeat-5.2.0-windows-x86\_64\filebeat.template.json  
2017/02/10 13:46:17.054776 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\filebeat-5.2.0-windows-x86\_64\filebeat.template-es2x.json  
2017/02/10 13:46:17.054776 client.go:120: INFO Elasticsearch url: [http://localhost:9200](http://localhost:9200)  
2017/02/10 13:46:17.054776 outputs.go:106: INFO Activated elasticsearch as output plugin.  
2017/02/10 13:46:17.054776 publish.go:234: DBG Create output worker  
2017/02/10 13:46:17.054776 publish.go:276: DBG No output is defined to store the topology. The server fields might not be filled.  
2017/02/10 13:46:17.055775 publish.go:291: INFO Publisher name: data-vm0  
2017/02/10 13:46:17.057801 async.go:63: INFO Flush Interval set to: 1s  
2017/02/10 13:46:17.057801 async.go:64: INFO Max Bulk Size set to: 50  
2017/02/10 13:46:17.057801 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=50)  
2017/02/10 13:46:17.057801 beat.go:207: INFO filebeat start running.  
2017/02/10 13:46:17.057801 service\_windows.go:51: DBG Windows is interactive: true  
2017/02/10 13:46:17.057801 registrar.go:85: INFO Registry file set to: C:\filebeat-5.2.0-windows-x86\_64\data\registry  
2017/02/10 13:46:17.058777 registrar.go:106: INFO Loading registrar data from C:\filebeat-5.2.0-windows-x86\_64\data\registry  
2017/02/10 13:46:17.058777 registrar.go:123: INFO States Loaded from registrar: 0  
2017/02/10 13:46:17.058777 registrar.go:236: INFO Starting Registrar  
2017/02/10 13:46:17.058777 crawler.go:34: INFO Loading Prospectors: 1  
2017/02/10 13:46:17.058777 spooler.go:63: INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s  
2017/02/10 13:46:17.058777 sync.go:41: INFO Start sending events to output  
2017/02/10 13:46:17.058777 prospector\_log.go:41: DBG exclude\_files: []  
2017/02/10 13:46:17.058777 prospector\_log.go:57: INFO Prospector with previous states loaded: 0  
2017/02/10 13:46:17.058777 prospector.go:70: DBG File Configs: [G:\azurelogs\name=default\resourceId\y=2017\m=02\d=07\h=14\m=00]  
2017/02/10 13:46:17.059776 crawler.go:48: INFO Loading Prospectors completed. Number of prospectors: 1  
2017/02/10 13:46:17.059776 crawler.go:63: INFO All prospectors are initialised and running with 0 states to persist  
2017/02/10 13:46:17.059776 crawler.go:58: DBG Starting prospector 0  
2017/02/10 13:46:17.059776 prospector.go:112: INFO Starting prospector of type: log  
2017/02/10 13:46:17.059776 prospector\_log.go:62: DBG Start next scan  
**2017/02/10 13:46:17.059776 prospector\_log.go:140: DBG Skipping directory: G:\azurelogs\name=default\resourceId\y=2017\m=02\d=07\h=14\m=00**  
2017/02/10 13:46:17.059776 prospector\_log.go:83: DBG Prospector states cleaned up. Before: 0, After: 0  
2017/02/10 13:46:20.183487 service.go:32: DBG Received sigterm/sigint, stopping  
2017/02/10 13:46:20.183487 filebeat.go:168: INFO Stopping filebeat  
2017/02/10 13:46:20.183487 crawler.go:69: INFO Stopping Crawler  
2017/02/10 13:46:20.183487 crawler.go:75: INFO Stopping 1 prospectors  
2017/02/10 13:46:20.183487 prospector.go:187: INFO Stopping Prospector  
2017/02/10 13:46:20.183487 prospector.go:129: INFO Prospector channel stopped  
2017/02/10 13:46:20.183487 service.go:38: DBG Received svc stop/shutdown request  
2017/02/10 13:46:20.183487 prospector.go:153: INFO Prospector ticker stopped  
2017/02/10 13:46:20.184487 crawler.go:56: DBG Prospector 0 stopped  
2017/02/10 13:46:20.184487 crawler.go:82: INFO Crawler stopped  
2017/02/10 13:46:20.184487 spooler.go:101: INFO Stopping spooler  
2017/02/10 13:46:20.184487 spooler.go:109: DBG Spooler has stopped  
2017/02/10 13:46:20.184487 sync.go:47: DBG Shutting down sync publisher  
2017/02/10 13:46:20.184487 registrar.go:291: INFO Stopping Registrar  
2017/02/10 13:46:20.184487 registrar.go:248: INFO Ending Registrar  
2017/02/10 13:46:20.184487 registrar.go:298: DBG Write registry file: C:\filebeat-5.2.0-windows-x86\_64\data\registry  
2017/02/10 13:46:20.185488 registrar.go:323: DBG Registry file updated. 0 states written.  
2017/02/10 13:46:20.185488 logp.go:245: INFO Total non-zero values: registrar.writes=1  
2017/02/10 13:46:20.185488 logp.go:246: INFO Uptime: 3.1346887s  
2017/02/10 13:46:20.185488 beat.go:211: INFO filebeat stopped.  
PS C:\filebeat-5.2.0-windows-x86\_64\>  
PS C:\filebeat-5.2.0-windows-x86\_64\>

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 10, 2017, 2:09pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/5 "2017-02-10T14:09:58Z")

</div>

Looks like i have found an issue.  
You filebeat can read only absolute path, but not relative with \* symbols.  
And as it turned out wildcards do not work for Windows for directories.

I have added path to the file manually and got error of processing file:

2017/02/10 14:05:29.170341 client.go:432: WARN Can not index event (status=404): {"type":"type\_missing\_exception","reason":"type[log] missing","index\_uuid":"F4-Ca\_YEQDGkjPZOUFdfnQ","index":"filebeat-2017.02.10","caused\_by":{"type":"illegal\_state\_exception","reason":"trying to auto create mapping,  
dynamic mapping is disabled"}}

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 12, 2017, 1:43pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/7 "2017-02-12T13:43:06Z")

</div>

> [@111148](#):
>
> Looks like i have found an issue.You filebeat can read only absolute path, but not relative with \* symbols.And as it turned out wildcards do not work for Windows for directories.
> 
> I have added path to the file manually and got error of processing file:
> 
> 2017/02/10 14:05:29.170341 client.go:432: WARN Can not index event (status=404): {"type":"type\_missing\_exception","reason":"type[log] missing","index\_uuid":"F4-Ca\_YEQDGkjPZOUFdfnQ","index":"filebeat-2017.02.10","caused\_by":{"type":"illegal\_state\_exception","reason":"trying to auto create mapping, dynamic mapping is disabled"}}

Hello, do you have any solution for this?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [February 14, 2017, 9:23am UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/8 "2017-02-14T09:23:59Z")

</div>

It seems something got messed up with your index while you were playing around with the different options. I would recommend you to drop the index on the elasticsearch side again and restart filebeat.

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 15, 2017, 2:21pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/9 "2017-02-15T14:21:34Z")

</div>

But elastic search creates these indexes automatically and there are only indexes for ELK stack, but not for the Filebeat and Logstash as far as i know.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [February 16, 2017, 3:36pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/10 "2017-02-16T15:36:04Z")

</div>

What does your ES config look like? Reasons I ask because the error `dynamic mapping is disabled`.

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 17, 2017, 7:24pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/11 "2017-02-17T19:24:18Z")

</div>

Logstash config:

```auto
input
{
    azureblob
    {
        storage_account_name => "someaccount"
        storage_access_key => "somekey"
        container => "somecontainer"
    }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
	index => "logstash-%{+YYYY.MM.dd}"
	user => logstash_system
    password => somepassword
  }
}

```

* * *

ES config:

```auto
cluster.name: ELK

node.name: node3

network.host: ["_local_", "_site_"]

discovery.zen.ping.unicast.hosts: ["someip", "someanotherip"]

path.data: ["e:/data", "f:/data"]

```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [February 21, 2017, 8:17am UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/12 "2017-02-21T08:17:59Z")

</div>

You posted your LS config above. Do you want to send the events through LS or directly to ES?

Is the above the full ES config?

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 27, 2017, 12:07pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/13 "2017-02-27T12:07:22Z")

</div>

Hello,  
I have posted both config files.  
First one is logstash, second is elastic.

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 27, 2017, 5:03pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/14 "2017-02-27T17:03:50Z")

</div>

Have short update.  
I have removed x-pack and seems now Logstash is working.  
But the question is how to tune (create properly) indices so as now there is a mess in displaying of the data.  
[http://rgho.st/6Lb7NjMsK](http://rgho.st/6Lb7NjMsK) (json parsing failure)

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [February 28, 2017, 8:20am UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/15 "2017-02-28T08:20:39Z")

</div>

Are you still using Filebeat or now only Logstash? If only Logstash, I suggest to best ask the question in the Logstash forum.

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [February 28, 2017, 10:48am UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/16 "2017-02-28T10:48:43Z")

</div>

Logstash only.  
Thanks.

---

<div class="post-metadata">

**Author:** ![111148](https://avatars.discourse-cdn.com/v4/letter/1/eb8c5e/32.png) [@111148](https://discuss.elastic.co/u/111148)\
**Post date:** [March 1, 2017, 8:29pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/17 "2017-03-01T20:29:27Z")

</div>

Looks like logstash forum has no answer as well on this question. what a shame.

> [@Logstash can not parse azure activity logs](https://discuss.elastic.co/t/logstash-can-not-parse-azure-activity-logs/76801):
>
> Hello, I have faced with issue during parsing Azure activity logs (multiline json) by Logstash. Tried several codecs but all of them are useless. They can not parse multiline logs with coma delimiters. Please advise how to fix it and add custom fields from Json file to the logstash index (logstash-\*). Thanks. Json files sctucture is next: { "time": "2017-02-28T12:04:25.7263914Z", "resourceId": "/SUBSCRIPTIONS/...../RESOURCEGROUPS/....", "operationName": "MICROSOFT.RESOURCES/SUBSC…

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 29, 2017, 8:29pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-for-sending-azure-json-logs/74456/18 "2017-03-29T20:29:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.