# Data collection from Proxmox

**URL:** <https://discuss.elastic.co/t/data-collection-from-proxmox/295469>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [January 26, 2022, 2:54pm UTC](https://discuss.elastic.co/t/data-collection-from-proxmox/295469 "2022-01-26T14:54:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![vovan13111](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vovan13111/32/96560_2.png) [@vovan13111](https://discuss.elastic.co/u/vovan13111)\
**Post date:** [January 26, 2022, 2:54pm UTC](https://discuss.elastic.co/t/data-collection-from-proxmox/295469/1 "2022-01-26T14:54:08Z")

</div>

Good afternoon.  
When studying metricbeat for the ability to collect data from kvm, the question arose, will the ability to collect data from proxmox be added, like from vSphere? If this is already possible, then please tell me how to do it.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [January 26, 2022, 3:48pm UTC](https://discuss.elastic.co/t/data-collection-from-proxmox/295469/2 "2022-01-26T15:48:50Z")

</div>

There is no direct proxmox logging or plugin for Elasticsearch.

What I have done is create logging of proxmox to influxdb. They have plugin for it.  
and from that server you take all the data as is. I wrote python code to read from there and put it in elk and it worked without any extra effort.

if you need that code please let me know

---

<div class="post-metadata">

**Author:** ![vovan13111](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vovan13111/32/96560_2.png) [@vovan13111](https://discuss.elastic.co/u/vovan13111)\
**Post date:** [February 3, 2022, 3:58am UTC](https://discuss.elastic.co/t/data-collection-from-proxmox/295469/3 "2022-02-03T03:58:29Z")

</div>

> [@elasticforme](#):
>
> There is no direct proxmox logging or plugin for Elasticsearch.
> 
> What I have done is create logging of proxmox to influxdb. They have plugin for it.  
> and from that server you take all the data as is. I wrote python code to read from there and put it in elk and it worked without any extra effort.
> 
> if you need that code please let me know

Good afternoon.  
Please send the code. I just thought there are ways without intermediate databases.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [February 3, 2022, 2:47pm UTC](https://discuss.elastic.co/t/data-collection-from-proxmox/295469/4 "2022-02-03T14:47:09Z")

</div>

This is not complete code as I can't cut-paste due to policy.

but here is what I can. this is main part of code. and running this every 10 min via cron

```auto
#Main program
from influxdb import InfluxDBClient
 # Influx user/passoword
    host = '10.25.16.62' ; port = 8086 ; user = 'sachin' ; password = 'sachin'; dbname = 'mypxws'
    measuments = ['cpustat','memory','nics','system']
    #while True:
    for table in measuments:
        output_index = 'proxmox_'+table+'-2022'
        query = 'select * from ' + table +' where time > now() - 10m;'
        print (query, output_index)

        # open connection with Influxdb
        client = InfluxDBClient(host, port, user, password, dbname)

        result = client.query(query) # Run query
        # print(format(result))
        db_conn_data = result.get_points()

        elastic_output = Elasticsearch(elastic_hostnames, http_auth=(elastic_admin_user, elastic_admin_passwd), port=9200)
        load_to_elk(db_conn_data, elastic_output, output_index)
        client.close()

```

```auto
Here is load_to_elk stuff
# once done load in batch to elk, 3000 a time.
def load_to_elk(db_conn_data,elastic_output,output_index):
    mylist = []
    rec_count= 0
    for record in db_conn_data:
        # '2021-06-07T20:41:55Z to '2021-06-07 20:41:55
        mytime = datetime.strptime(record['time'][0:19].replace('T', ' '), '%Y-%m-%d %H:%M:%S')
        # mytime = datetime.strptime(point['time'], "%y-%m-%d %H:%M:%S")
        record['time'] = pytz.timezone(LOCAL_TZ).localize(mytime)
        record['@timestamp'] = record['time']
        record['_index'] = output_index
        #print(type(mytime), type(point['time']))
        mylist.append(record)
        if len(mylist) > 2999:
                 pb = helpers.bulk(elastic_output, mylist)
                mylist = []
          
   #lets load last batch
   pb = helpers.bulk(elastic_output, mylist)

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2022, 4:47pm UTC](https://discuss.elastic.co/t/data-collection-from-proxmox/295469/5 "2022-03-03T16:47:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.