# Using filebeat sending logs to logstash?

**URL:** <https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 12, 2017, 11:45am UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030 "2017-06-12T11:45:17Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yaswanth](https://avatars.discourse-cdn.com/v4/letter/y/94ad74/32.png) [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\
**Post date:** [June 12, 2017, 11:45am UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/1 "2017-06-12T11:45:18Z")

</div>

Hi,

Two types of Logs:

```
[2017-06-12 01:00:00,155][ERROR][marvel.agent.exporter.local] local exporter [default_local] - failed to delete indices
RemoteTransportException[[data-1][x.x.x.x:9200][indices:admin/delete]]; nested: IndexNotFoundException[no such index];
Caused by: [.marvel-es-1-2017.06.05] IndexNotFoundException[no such index]
        at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService$1.execute(MetaDataDeleteIndexService.java:91)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2017-06-12 07:33:08,197][DEBUG][index.search.slowlog.query] [data-0] [datacontent-jun-2017][1] took[98.6ms], took_millis[98], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[15], source[{"size":100,"query":{"nested":{"query":{"bool":{"must":[{"term":{"List.Id":{"value":136003}}},{"terms":{"List.Id":[353174427,353339434]}}]}},"path":"List"}}}], extra_source[],

```

I am sending two different logs from two different locations through filebeat to logstash . My filebeat configuration is:

```
filebeat.prospectors:

- input_type: log
  paths:
    - "/var/log/elasticsearch/escluster.log"
  document_type: exceptions

- input_type: log
  paths:
    - "/var/log/elasticsearch/escluster_index_search_slowlog.log"
  document_type: slowlogs

output.logstash:
  hosts: ["10.0.0.5:5044"]

logging:
  to_files: true
  files:
    path: /etc/filebeat/logs
  level: debug
  selectors: ["*"]

```

Here in this context the type should be `slowlogs` but it is `exceptions` i dont know where the problem is?

Thanks

---

<div class="post-metadata">

**Author:** ![Oozza](https://avatars.discourse-cdn.com/v4/letter/o/76d3ee/32.png) [@Oozza](https://discuss.elastic.co/u/Oozza)\
**Post date:** [June 12, 2017, 12:00pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/2 "2017-06-12T12:00:21Z")

</div>

> [@Yaswanth](#):
>
> "source": "/var/log/elasticsearch/kmesprodcluster.log",

Your filebeat config says nothing about the path we see in your output. This is confusing.

---

<div class="post-metadata">

**Author:** ![Yaswanth](https://avatars.discourse-cdn.com/v4/letter/y/94ad74/32.png) [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\
**Post date:** [June 12, 2017, 12:03pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/3 "2017-06-12T12:03:33Z")

</div>

sry @Oozza

It was bad copy paste. I updated my question.

Thanks

---

<div class="post-metadata">

**Author:** ![Oozza](https://avatars.discourse-cdn.com/v4/letter/o/76d3ee/32.png) [@Oozza](https://discuss.elastic.co/u/Oozza)\
**Post date:** [June 12, 2017, 12:16pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/4 "2017-06-12T12:16:37Z")

</div>

Now output says that the source file is `/var/log/elasticsearch/escluster.log` and type is `exceptions`, which is correct according to your filebeat config.

---

<div class="post-metadata">

**Author:** ![Yaswanth](https://avatars.discourse-cdn.com/v4/letter/y/94ad74/32.png) [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\
**Post date:** [June 12, 2017, 12:18pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/5 "2017-06-12T12:18:58Z")

</div>

> [@Yaswanth](#):
>
> [2017-06-12 07:33:08,197][DEBUG][index.search.slowlog.query] [data-0] [datacontent-jun-2017][1] took[98.6ms], took\_millis[98], types, stats, search\_type[QUERY\_THEN\_FETCH], total\_shards[15], source[{"size":100,"query":{"nested":{"query":{"bool":{"must":[{"term":{"List.Id":{"value":136003}}},{"terms":{"List.Id":[353174427,353339434]}}]}},"path":"List"}}}], extra\_source,

But the above log is in [quote="Yaswanth, post:1, topic:89030"]  
/var/log/elasticsearch/escluster\_index\_search\_slowlog.log  
[/quote]

The filebeat is showing the log with different `source` and `type`. Is there any problem in my logstash config file?

---

<div class="post-metadata">

**Author:** ![Oozza](https://avatars.discourse-cdn.com/v4/letter/o/76d3ee/32.png) [@Oozza](https://discuss.elastic.co/u/Oozza)\
**Post date:** [June 12, 2017, 12:26pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/6 "2017-06-12T12:26:29Z")

</div>

Alright, your multiline codec may be mixing your logs together. I would recommend to use multiline in filebeat instead.  
If you want to keep multiline in logstash, you should use one multiline codec for each `type`. But I am not sure if logstash will allow to do that.

---

<div class="post-metadata">

**Author:** ![Yaswanth](https://avatars.discourse-cdn.com/v4/letter/y/94ad74/32.png) [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\
**Post date:** [June 12, 2017, 12:52pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/7 "2017-06-12T12:52:36Z")

</div>

Thanks @Oozza

Based on your suggestion i used the multiline codec in the filebeat but it is not parsing this type of log properly

> [@Yaswanth](#):
>
> [2017-06-12 01:00:00,155][ERROR][marvel.agent.exporter.local] local exporter [default\_local] - failed to delete indices  
> RemoteTransportException[[data-1][x.x.x.x:9200][indices:admin/delete]]; nested: IndexNotFoundException[no such index];  
> Caused by: [.marvel-es-1-2017.06.05] IndexNotFoundException[no such index]  
> at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService$1.execute(MetaDataDeleteIndexService.java:91)  
> at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)  
> at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)  
> at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)  
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)  
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

I HAD used below configuration:

```
 multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
 multiline.negate: true
 multiline.match: after

```

Thanks

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [June 12, 2017, 2:26pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/8 "2017-06-12T14:26:23Z")

</div>

Can you share your filebeat config with multiline? Is multiline configuration with correct prospector and indentation correct?

According to [this go playground](https://play.golang.org/p/Q7YQCSMjp2) you pattern looks good.

---

<div class="post-metadata">

**Author:** ![Yaswanth](https://avatars.discourse-cdn.com/v4/letter/y/94ad74/32.png) [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\
**Post date:** [June 12, 2017, 2:40pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/9 "2017-06-12T14:40:20Z")

</div>

Thanks @steffens

THis my filebeat configuration

```
filebeat.prospectors:

- input_type: log
  paths:
    - "/var/log/elasticsearch/escluster.log"
  document_type: exceptions

- input_type: log
  paths:
    - "/var/log/elasticsearch/escluster_index_search_slowlog.log"
  document_type: slowlogs

multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after

output.logstash:
  hosts: ["10.0.0.5:5044"]

logging:
  to_files: true
  files:
    path: /etc/filebeat/logs
  level: debug
  selectors: ["*"]

```

I think the problem is the javastack log (i.e.first log that i had posted) only needs multiline codec the second one does not need it. When i ran the filebeat beat by giving one path in input prospector it worked fine but when i used two input prospectors the output is not coming properly.

Why i am getting some null beats without any message or anything in it(i.e.which i pasted above) ? Is it possible to run 2 filebeats by giving different input prospectors?

Thanks

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [June 12, 2017, 10:25pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/10 "2017-06-12T22:25:52Z")

</div>

The multiline configuration is per prospector and needs to be indented at the same level as the other prospector options. For example:

```auto
- input_type: log
  paths:
    - "/var/log/elasticsearch/escluster_index_search_slowlog.log"
  document_type: slowlogs
  multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 10, 2017, 10:26pm UTC](https://discuss.elastic.co/t/using-filebeat-sending-logs-to-logstash/89030/11 "2017-07-10T22:26:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.