Data collection from Proxmox

[vovan13111 (Владимир Бочаров)]

Good afternoon.
When studying metricbeat for the ability to collect data from kvm, the question arose, will the ability to collect data from proxmox be added, like from vSphere? If this is already possible, then please tell me how to do it.

[elasticforme (Sachin)]

There is no direct proxmox logging or plugin for Elasticsearch.

What I have done is create logging of proxmox to influxdb. They have plugin for it.
and from that server you take all the data as is. I wrote python code to read from there and put it in elk and it worked without any extra effort.

if you need that code please let me know

[vovan13111 (Владимир Бочаров)]

@elasticforme

There is no direct proxmox logging or plugin for Elasticsearch.

What I have done is create logging of proxmox to influxdb. They have plugin for it.
and from that server you take all the data as is. I wrote python code to read from there and put it in elk and it worked without any extra effort.

if you need that code please let me know

Good afternoon.
Please send the code. I just thought there are ways without intermediate databases.

[elasticforme (Sachin)]

This is not complete code as I can't cut-paste due to policy.

but here is what I can. this is main part of code. and running this every 10 min via cron

#Main program
from influxdb import InfluxDBClient
 # Influx user/passoword
    host = '10.25.16.62' ; port = 8086 ;  user = 'sachin' ;  password = 'sachin'; dbname = 'mypxws'
    measuments = ['cpustat','memory','nics','system']
    #while True:
    for table in measuments:
        output_index = 'proxmox_'+table+'-2022'
        query = 'select * from ' + table +' where time > now() - 10m;'
        print (query, output_index)

        # open connection with Influxdb
        client = InfluxDBClient(host, port, user, password, dbname)

        result = client.query(query) # Run query
        # print(format(result))
        db_conn_data = result.get_points()

        elastic_output = Elasticsearch(elastic_hostnames, http_auth=(elastic_admin_user, elastic_admin_passwd), port=9200)
        load_to_elk(db_conn_data, elastic_output, output_index)
        client.close()

Here is load_to_elk stuff
# once done load in batch to elk, 3000 a time.
def load_to_elk(db_conn_data,elastic_output,output_index):
    mylist = []
    rec_count= 0
    for record in db_conn_data:
        # '2021-06-07T20:41:55Z to '2021-06-07 20:41:55
        mytime = datetime.strptime(record['time'][0:19].replace('T', ' '), '%Y-%m-%d %H:%M:%S')
        # mytime = datetime.strptime(point['time'], "%y-%m-%d %H:%M:%S")
        record['time'] = pytz.timezone(LOCAL_TZ).localize(mytime)
        record['@timestamp'] = record['time']
        record['_index'] = output_index
        #print(type(mytime), type(point['time']))
        mylist.append(record)
        if len(mylist) > 2999:
                 pb = helpers.bulk(elastic_output, mylist)
                mylist = []
          
   #lets load last batch
   pb = helpers.bulk(elastic_output, mylist)