# Request log from python to build stats over user usage
**URL:** https://discuss.elastic.co/t/request-log-from-python-to-build-stats-over-user-usage/148997
**Category:** Logstash
**Created:** [September 18, 2018, 3:07pm UTC](https://discuss.elastic.co/t/request-log-from-python-to-build-stats-over-user-usage/148997 "2018-09-18T15:07:20Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@esseti](https://discuss.elastic.co/u/esseti)
#### Post date: [September 18, 2018, 3:07pm UTC](https://discuss.elastic.co/t/request-log-from-python-to-build-stats-over-user-usage/148997/1 "2018-09-18T15:07:20Z")
Hi all,
We have built a system that has to keep track of the number of calls a user does against our API, we also have to keep track of the logging info to build "audit" logs and give to the user the possibility to search and retrieve data.
So far, we have used graylog as a server where to stream the data, via udp.
Since I want to migrate to ELK & co (apm as well) what's the best/easiest solution to do?
We have a huge amount of data every second to stream to logstash and have them saved in elastic (we stream JSON). Should I create a UDP connection as well (does exist a library)? is there anything provided by logstash? I've check APM but it's mostly to evaluate performances and errors rather than collecting logs.
Our setup is in kubernetes and docker containers where the API codes runs. So streaming files may not be easier, while streaming directly from the app could be a bit better.
PS: is logstash able to parse the data JSON and maybe route them to different indexes? Since the data stored will be "huge" I would like to create indexed by users (this would be cool to export data, but almos impossible to remove data after X days) or divided in size of the docs or time (this makes easier to remove indexes after X days, right?). is this feasible?
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [September 24, 2018, 9:01pm UTC](https://discuss.elastic.co/t/request-log-from-python-to-build-stats-over-user-usage/148997/2 "2018-09-24T21:01:15Z")
Sure it is possible.
Logstash has a [gelf input](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-gelf.html) that can use tcp or udp and a JSON codec to decode the data.
Some people find that one Logstash instance is not enough to cope with a very high load although this is partly determined by the performance of the elasticsearch cluster.
There are [various designs for parallelisation](https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html#adding-other-sources). The elasticsearch output can use [values from a document to put that doc in an index](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index).
---
### Author:  [@esseti](https://discuss.elastic.co/u/esseti)
#### Post date: [September 25, 2018, 3:22pm UTC](https://discuss.elastic.co/t/request-log-from-python-to-build-stats-over-user-usage/148997/3 "2018-09-25T15:22:23Z")
Right now I'm using a [https://github.com/vklochan/python-logstash](https://github.com/vklochan/python-logstash) that sends JSON over udp. what would be the benefit of using GELF input?
is there any known limit of performances? let's say X messages/seconds or sort of it, just to understand the magnitude.
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [September 26, 2018, 8:20am UTC](https://discuss.elastic.co/t/request-log-from-python-to-build-stats-over-user-usage/148997/4 "2018-09-26T08:20:40Z")
That python library looks good. You will need a TCP input and a json filter. The json **codec** would need a newline as the final character in the tcp payload and I'm not sure the newline will be there.
Ignore what I said about the gelf input but the rest of of my comment is valid.
---
### Author:  [@esseti](https://discuss.elastic.co/u/esseti)
#### Post date: [September 26, 2018, 8:34am UTC](https://discuss.elastic.co/t/request-log-from-python-to-build-stats-over-user-usage/148997/5 "2018-09-26T08:34:48Z")
it actually works, i need to parse inner messages but that's a logstash pipeline configuration. i'm using UDP rather than TCP for performances ..
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 24, 2018, 8:34am UTC](https://discuss.elastic.co/t/request-log-from-python-to-build-stats-over-user-usage/148997/6 "2018-10-24T08:34:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.