# Ingest problem with filebeat and logstash **URL:** \ **Category:** Logstash\ **Created:** [November 27, 2017, 9:09am UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188 "2017-11-27T09:09:56Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Rodolphe\_Redoute](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@Rodolphe\_Redoute](https://discuss.elastic.co/u/Rodolphe_Redoute)\ **Post date:** [November 27, 2017, 9:09am UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/1 "2017-11-27T09:09:56Z")
Hi, I have an ingest problem with filebeat and logstash. I have machines with a configuration on it's own to get different type of log files. my first filebeat.yml file is like this : ``` filebeat.prospectors: - document_type: wowza paths: - /usr/local/WowzaStreamingEngine-4.0.1/logs/wowzastreamingengine_access.log* output: logstash: hosts: ["somepath:5043"] logging: to_files: true level: info files: path: /var/log/filebeat name: filebeat.log rotateeverybytes: 10485760 keepfiles: 7 ``` and the second is like this : ``` filebeat.prospectors: - document_type: elemental_live paths: - /opt/elemental_se/web/log/*0000/job_*/*_eme.log output: logstash: hosts: ["somepath:5043"] logging: to_files: true level: warning files: path: /var/log name: filebeat.log rotateeverybytes: 10485760 keepfiles: 4 ``` my output conf of logstash is this : ``` output { elasticsearch { hosts => ["somepath:9200"] index => "logstash-%{type}-%{+YYYY.MM.dd}" } } ``` my problem is, with the 2nd yml file i have indices like this one : `green open logstash-elemental_live-2017.11.22 CvnMPAClRNK9a8MxY63ccQ 1 0 141 0 128.4kb 128.4kb` but with the first file it look like this : `green open logstash-%{type}-2017.11.21 1rOpLykcTBK2RyXkLmZd7A 1 0 3004790 0 756.1mb 756.1mb` and the problem is that with kibana when i try to create a new index pattern it has a bug with "logstash-%{type}" (which is not supposed to be like this at the beginning) Anyone who has a clue why my indices look like "logstash-%{type}..." and not "logstash-wowza..." ? Thx in advance. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 27, 2017, 9:12am UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/2 "2017-11-27T09:12:27Z")
> Anyone who has a clue why my indices look like "logstash-%{type}..." and not "logstash-wowza..." ? Those events clearly don't have the `type` field set to anything. Do you really have a leading space on the `filebeat.prospectors:` line for the wowza logs? ---
**Author:** ![Rodolphe\_Redoute](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@Rodolphe\_Redoute](https://discuss.elastic.co/u/Rodolphe_Redoute)\ **Post date:** [November 27, 2017, 9:17am UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/3 "2017-11-27T09:17:49Z")
hi, no it's just the copy past that did this (just checked it) ---
**Author:** ![Rodolphe\_Redoute](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@Rodolphe\_Redoute](https://discuss.elastic.co/u/Rodolphe_Redoute)\ **Post date:** [November 27, 2017, 10:10am UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/4 "2017-11-27T10:10:12Z")
ok i just checked version, The wowza config is on filebeat 6.0 which mean that "document\_type" is ignored and has to be replaced by "fields" Is the syntax the same, like we just have to change : - document\_type: wowza to - fields: wowza ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 27, 2017, 12:38pm UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/5 "2017-11-27T12:38:34Z")
No, `fields` points to a dictionary. See [https://www.elastic.co/guide/en/beats/filebeat/6.0/configuration-filebeat-options.html#configuration-fields](https://www.elastic.co/guide/en/beats/filebeat/6.0/configuration-filebeat-options.html#configuration-fields). ---
**Author:** ![Rodolphe\_Redoute](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@Rodolphe\_Redoute](https://discuss.elastic.co/u/Rodolphe_Redoute)\ **Post date:** [November 27, 2017, 12:42pm UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/6 "2017-11-27T12:42:16Z")
so if my guessing is correct my filebeat should be like this : ``` filebeat.prospectors: - paths: - /usr/local/WowzaStreamingEngine-4.0.1/logs/wowzastreamingengine_access.log* fields: application: "wowza" ``` and my logstash output file should look like this : ``` output { elasticsearch { hosts => ["somepath:9200"] index => "logstash-%{application}-%{+YYYY.MM.dd}" } } ``` ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 27, 2017, 12:52pm UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/7 "2017-11-27T12:52:58Z")
Yeah, that looks okay. ---
**Author:** ![Rodolphe\_Redoute](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@Rodolphe\_Redoute](https://discuss.elastic.co/u/Rodolphe_Redoute)\ **Post date:** [November 27, 2017, 1:00pm UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/8 "2017-11-27T13:00:39Z")
tried it, same problem : `green open logstash-%{application}-2017.11.27 JyG-38ilQ6Wcjb8haM1atQ 1 0 469590 0 154.9mb 154.9mb` i don't think i clearly understand how 'fields" work ---
**Author:** ![Rodolphe\_Redoute](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@Rodolphe\_Redoute](https://discuss.elastic.co/u/Rodolphe_Redoute)\ **Post date:** [November 27, 2017, 2:08pm UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/9 "2017-11-27T14:08:57Z")
is there a way to configure logstash output to access fields values ? i want to do something like that : ``` output { if [fields][tags] == "wowza" { elasticsearch { hosts => ["somepath:9200"] index => "logstash-wowza-%{+YYYY.MM.dd}" } } else { elasticsearch { hosts => ["somepath:9200"] index => "logstash-%{type}-%{+YYYY.MM.dd}" } } } ``` my problem is that " if [fields][tags] == "wowza" " don't work and is never reached EDIT : here is my new filebeat.yml : ``` filebeat.prospectors: - paths: - /usr/local/WowzaStreamingEngine-4.0.1/logs/wowzastreamingengine_access.log* fields: application: "wowza" tags: ["wowza"] output: logstash: hosts: ["somepath:5043"] logging: to_files: true level: info files: path: /var/log/filebeat name: filebeat.log rotateeverybytes: 10485760 keepfiles: 7 ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 27, 2017, 8:54pm UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/10 "2017-11-27T20:54:33Z")
Comment out your elasticsearch outputs and use a `stdout { codec => rubydebug }` output. What does an example wowza event look like? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 25, 2017, 8:54pm UTC](https://discuss.elastic.co/t/ingest-problem-with-filebeat-and-logstash/109188/11 "2017-12-25T20:54:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.