# Django logstash filter **URL:** \ **Category:** Logstash\ **Created:** [September 20, 2018, 8:06am UTC](https://discuss.elastic.co/t/django-logstash-filter/149246 "2018-09-20T08:06:14Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![esseti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/esseti/32/8383_2.png) [@esseti](https://discuss.elastic.co/u/esseti)\ **Post date:** [September 20, 2018, 8:06am UTC](https://discuss.elastic.co/t/django-logstash-filter/149246/1 "2018-09-20T08:06:15Z")
HI all, is there an "official" or standard-de-facto filter for logstash that can be used in Django? For what I know it should extend SocketHandler (TCP) or DatagramHandler (UDP), but I would love to build something from scratch to make it work. I've found [https://github.com/vklochan/python-logstash](https://github.com/vklochan/python-logstash) but it seems that it's not updated for a couple of years. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 24, 2018, 1:42pm UTC](https://discuss.elastic.co/t/django-logstash-filter/149246/2 "2018-09-24T13:42:36Z")
> is there an "official" or standard-de-facto filter for logstash that can be used in Django? No. I suggest you use a formatter (or layout or whatever it's called) that formats the log record object as a JSON string. That's trivial to parse on the Logstash side and takes care of multiline messages. > For what I know it should extend SocketHandler (TCP) or DatagramHandler (UDP), but I would love to build something from scratch to make it work. "It" being the Logstash filter? I'm confused. I don't think you should send log messages directly over the network. What happens if the Logstash server is unavailable? ---
**Author:** ![esseti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/esseti/32/8383_2.png) [@esseti](https://discuss.elastic.co/u/esseti)\ **Post date:** [September 25, 2018, 1:19pm UTC](https://discuss.elastic.co/t/django-logstash-filter/149246/3 "2018-09-25T13:19:45Z")
Ok for the formatting in JSON. Regarding sending logs, yes, if the server is down everything is lost. what could be a consistent approach? (it referred to the system to log the entries) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 25, 2018, 6:18pm UTC](https://discuss.elastic.co/t/django-logstash-filter/149246/4 "2018-09-25T18:18:26Z")
I suggest streaming the logs to disk and using Filebeat to ship them to Logstash. Sure, you can run out of disk space, but that's typically easier to handle than network and Logstash host outages. ---
**Author:** ![esseti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/esseti/32/8383_2.png) [@esseti](https://discuss.elastic.co/u/esseti)\ **Post date:** [September 26, 2018, 7:49am UTC](https://discuss.elastic.co/t/django-logstash-filter/149246/5 "2018-09-26T07:49:34Z")
The main problem is that our services are docker containers run in Kubernetes, so it would require to install a filebeat and have a file in every container. it may run out of space easily, no? is this the standard approach even for containers ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 26, 2018, 9:09am UTC](https://discuss.elastic.co/t/django-logstash-filter/149246/6 "2018-09-26T09:09:53Z")
Let the process stream its logs to stdout so that Docker picks them up and writes to disk. You can e.g. use Filebeat to ship the container logs. > **[Shipping Kubernetes logs with Filebeat](https://www.elastic.co/blog/shipping-kubernetes-logs-to-elasticsearch-with-filebeat)** > > We recently wrote about the new Filebeat features to retrieve & enrich Docker logs using Filebeat, and since the 6.0 release, you can leverage the same technology when running Kubernetes. Metadata... ---
**Author:** ![esseti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/esseti/32/8383_2.png) [@esseti](https://discuss.elastic.co/u/esseti)\ **Post date:** [October 15, 2018, 3:39pm UTC](https://discuss.elastic.co/t/django-logstash-filter/149246/7 "2018-10-15T15:39:10Z")
this one seems to take the whole docker log output and stream to elk. is this the scope? Now, in the docker log there's the JSON logs that I do plus other error/debug strings. Is there a way to seperate these logs and decide what goes to ELK? Is there a way to tell to failbeat to scrap a .log file that is inside the container? or this is not possible for kube/docker? thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 22, 2018, 6:13pm UTC](https://discuss.elastic.co/t/django-logstash-filter/149246/8 "2018-10-22T18:13:33Z")
> this one seems to take the whole docker log output and stream to elk. Yes. > Now, in the docker log there's the JSON logs that I do plus other error/debug strings. The Docker log contains the stdout and stderr of the process. I believe Docker wraps those lines into JSON objects. > Is there a way to seperate these logs and decide what goes to ELK? Filebeat has limited support for filtering out messages, see [Filter and enhance data with processors | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 19, 2018, 6:13pm UTC](https://discuss.elastic.co/t/django-logstash-filter/149246/9 "2018-11-19T18:13:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.