# Filebeat ship logs verbatim **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 15, 2016, 1:40am UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096 "2016-12-15T01:40:05Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![surajs](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@surajs](https://discuss.elastic.co/u/surajs)\ **Post date:** [December 15, 2016, 1:40am UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/1 "2016-12-15T01:40:05Z")
I'm configuring filebeat 5.1.1 to output to kafka. I would like filebeat to output to the kafka topic the lines in the log file verbatim. Or move these fields under the \> beat field? This would allow us to easily migrate to using filebeat since the consumers of events from kafka need not update their logic to read from under a json field ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [December 15, 2016, 1:33pm UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/2 "2016-12-15T13:33:14Z")
See kafka [topic](https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#_topic) and [topics](https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#_topics) settings. The `topic` setting uses [format strings](https://www.elastic.co/guide/en/beats/libbeat/current/config-file-format-type.html#_format_string_sprintf) potentially accessing any field in the event. It's not clear to me how you logs are structured and how exactly you want to filter. But more advanced processing and event-routine requires logstash. ---
**Author:** ![surajs](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@surajs](https://discuss.elastic.co/u/surajs)\ **Post date:** [December 15, 2016, 7:09pm UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/3 "2016-12-15T19:09:43Z")
Thanks @steffens for your reply. Apologies for not being clear in explaining my question. Say for example, my log msg line looks like : `{"message": "hello world"}` Rather than having a response like : > { > "@timestamp": "2016-12-15T19:03:53.232Z", > "beat": { > "hostname": "surajs-host", > "name": "surajs-host", > "version": "5.1.1" > }, > "input\_type": "stdin", > "message": "hello world", > "offset": 0, > "source": "test.log", > "type": "json" > } We'd want to see something like > { > "beat": { > "@timestamp": "2016-12-15T19:03:53.232Z", > "hostname": "surajs-host", > "name": "surajs-host", > "version": "5.1.1" > "input\_type": "stdin", > "offset": 0, > "source": "test.log", > "type": "json" > }, > "message": "hello world", > } This will enable the consumer to know that all fields under the beats field are added by filebeat and everything at the top level is what is from the original message. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [December 16, 2016, 7:45am UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/4 "2016-12-16T07:45:55Z")
This is currently not possible. To change the format of the output you need logstash or ingest node. Be aware that changing the format of the events has the consequence that they stop working with the index template of elasticsearch. ---
**Author:** ![surajs](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@surajs](https://discuss.elastic.co/u/surajs)\ **Post date:** [December 16, 2016, 4:17pm UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/5 "2016-12-16T16:17:09Z")
Our current deployment looks like : ``` filebeat -> Kafka -> LS -> ES |--> Applications ``` Looks like what you're suggesting will lead us to have a sandwich LS architecture filebeat -\> LS -\> Kafka -\> LS -\> ES Do you think it would make sense to have a flag that allows a user to specify the field under which to put all the beats associated metadata ? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [December 19, 2016, 1:24pm UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/6 "2016-12-19T13:24:21Z")
As @ruflin said, this is currently not possible. Most metadata like `source`, `input_type`, `ofsset`, .... are expected to be part of the actual event. All beats related metadata are already in `beat`-namespace. Is the event structure really such a big issue, justifying a more complicate system architecture? ---
**Author:** ![surajs](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@surajs](https://discuss.elastic.co/u/surajs)\ **Post date:** [December 21, 2016, 6:28pm UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/7 "2016-12-21T18:28:28Z")
@steffens I can see how the information can be muddled if directly put under the beats namespace, but I don't see any harm of having them under beats.filebeat namespace. or have a filebeat namespace at the top level. We have a large number of consumers consuming data from kafka and my worry is if the meta-data is included at the top level, it will be impossible to distinguish it from actual event-data compared to beats data. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [December 23, 2016, 8:30am UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/8 "2016-12-23T08:30:35Z")
If you look at metricbeat, you see that we currently kind of follow the opposite approach. Everything added by the beat is namespaced under module / metricset to prevent namespace conflicts. So your best bet at the moment is probably either to update your consumers to read from json or use Logstash. ---
**Author:** ![surajs](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@surajs](https://discuss.elastic.co/u/surajs)\ **Post date:** [January 4, 2017, 10:23pm UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/9 "2017-01-04T22:23:32Z")
Sorry for the delayed reply. Thanks for the input. Just wanted to clarify my understanding about your previous suggestion. So you're saying that adding a filebeat namespace at the top level is not the right approach? So an example msg like this would be different from other beat output semantics : > { > "beat": { > "@timestamp": "2016-12-15T19:03:53.232Z", > "hostname": "surajs-host", > "name": "surajs-host", > "version": "5.1.1" > }, > "filebeat": { > "input\_type": "stdin", > "offset": 0, > "source": "test.log", > "type": "json" > }, > "message": "hello world", > } ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [January 5, 2017, 1:49pm UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/10 "2017-01-05T13:49:29Z")
@surajs I definitively don't want to say it is not the right approach. It's more two different ways of doing it and I see good reasons for both. Here is an example event from metricbeat: [https://github.com/elastic/beats/blob/master/metricbeat/module/apache/status/\_meta/data.json](https://github.com/elastic/beats/blob/master/metricbeat/module/apache/status/_meta/data.json) It resembles the above in that some info is under `metricset`, but this info is rather static. Infos that constantly change are under the `apache.status` namespace. BUT this is not really comparable to filebeat as there is only message, so message could be the namespace here. It becomse more interesting in the case of json and where the json data is written to. Summary: It is a really tricky question with lots of different options. Obviously as we have picked one model it is harder for us to change as it breaks BC. But the nice thing with LS and Ingest is that you can build your own logic to change the event to the format you need it. ---
**Author:** ![surajs](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@surajs](https://discuss.elastic.co/u/surajs)\ **Post date:** [January 5, 2017, 6:53pm UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/11 "2017-01-05T18:53:31Z")
Thanks @ruflin for the detailed answer. Will look at how we can adopt our exisiting architecture and data flow pipeline to fit our use-case. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 2, 2017, 6:53pm UTC](https://discuss.elastic.co/t/filebeat-ship-logs-verbatim/69096/12 "2017-02-02T18:53:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.