Unable to get visualize data from postgres fusing metricbeat

Dinesh_Senthil_Kumar · November 15, 2017, 1:55pm

Hi,

i am using elk 5.6, i have installed elasticsearch and kibana successfully. i have configured metric beat , i am able to get all cpu metrics but am not gettign any metrics regrding postgres.

while the metric beat log says it is sending data to elastisearch. but when i try to visualize am gettign null value for all the postgres related data. kindly help to resolve the same.
also is there a standard dashboard for postgres.

ruflin · November 16, 2017, 4:38am

Can you share your metricbeat config and the logs?

Dinesh_Senthil_Kumar · November 16, 2017, 5:44am

hi rufin pfb the error log i am getting from metric beat.
2017/11/16 05:41:27.217642 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=diskio, host=]
2017/11/16 05:41:27.217650 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=filesystem, host=]
2017/11/16 05:41:27.217666 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=core, host=]
2017/11/16 05:41:27.217631 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=load, host=]
2017/11/16 05:41:27.217672 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=cpu, host=]
2017/11/16 05:41:27.217675 wrapper.go:201: DBG Stopped metricSetWrapper[module=postgresql, name=activity, host=172.21.197.195:5444]
2017/11/16 05:41:27.217982 metrics.go:51: INFO Total non-zero values: fetches.postgresql-activity.events=2 fetches.postgresql-activity.failures=2 fetches.postgresql-bgwriter.events=4 fetches.postgresql-bgwriter.failures=4 fetches.postgresql-database.events=2 fetches.postgresql-database.failures=2 fetches.system-core.events=12 fetches.system-core.success=2 fetches.system-cpu.events=2 fetches.system-cpu.success=2 fetches.system-diskio.events=16 fetches.system-diskio.success=2 fetches.system-filesystem.events=84 fetches.system-filesystem.success=2 fetches.system-fsstat.events=2 fetches.system-fsstat.success=2 fetches.system-load.events=2 fetches.system-load.success=2 fetches.system-memory.events=2 fetches.system-memory.success=2 fetches.system-network.events=10 fetches.system-network.success=2 fetches.system-process.events=1120 fetches.system-process.success=2 libbeat.es.call_count.PublishEvents=26 libbeat.es.publish.read_bytes=12765 libbeat.es.publish.write_bytes=690304 libbeat.es.published_and_acked_events=1258 libbeat.publisher.messages_in_worker_queues=1258 libbeat.publisher.published_events=1258
2017/11/16 05:41:27.218001 metrics.go:52: INFO Uptime: 15.258655504s
2017/11/16 05:41:27.218005 beat.go:237: INFO metricbeat stopped.
2017/11/16 05:41:27.217642 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=diskio, host=]
2017/11/16 05:41:27.217650 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=filesystem, host=]
2017/11/16 05:41:27.217666 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=core, host=]
2017/11/16 05:41:27.217631 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=load, host=]
2017/11/16 05:41:27.217672 wrapper.go:201: DBG Stopped metricSetWrapper[module=system, name=cpu, host=]
2017/11/16 05:41:27.217675 wrapper.go:201: DBG Stopped metricSetWrapper[module=postgresql, name=activity, host=172.21.197.195:5444]
2017/11/16 05:41:27.217982 metrics.go:51: INFO Total non-zero values: fetches.postgresql-activity.events=2 fetches.postgresql-activity.failures=2 fetches.postgresql-bgwriter.events=4 fetches.postgresql-bgwriter.failures=4 fetches.postgresql-database.events=2 fetches.postgresql-database.failures=2 fetches.system-core.events=12 fetches.system-core.success=2 fetches.system-cpu.events=2 fetches.system-cpu.success=2 fetches.system-diskio.events=16 fetches.system-diskio.success=2 fetches.system-filesystem.events=84 fetches.system-filesystem.success=2 fetches.system-fsstat.events=2 fetches.system-fsstat.success=2 fetches.system-load.events=2 fetches.system-load.success=2 fetches.system-memory.events=2 fetches.system-memory.success=2 fetches.system-network.events=10 fetches.system-network.success=2 fetches.system-process.events=1120 fetches.system-process.success=2 libbeat.es.call_count.PublishEvents=26 libbeat.es.publish.read_bytes=12765 libbeat.es.publish.write_bytes=690304 libbeat.es.published_and_acked_events=1258 libbeat.publisher.messages_in_worker_queues=1258 libbeat.publisher.published_events=1258
2017/11/16 05:41:27.218001 metrics.go:52: INFO Uptime: 15.258655504s
2017/11/16 05:41:27.218005 beat.go:237: INFO metricbeat stopped.

Dinesh_Senthil_Kumar · November 16, 2017, 5:45am

pfb the metricbeat.yml


#==========================  Modules configuration ============================
metricbeat.modules:
  -
    module: system
    metricsets:
      - cpu
      - load
      - core
      - diskio
      - filesystem
      - fsstat
      - memory
      - network
      - process
    period: 10s
    processes:
      - .*
  -
    module: postgresql
    metricsets:
      - database
      - bgwriter
      - bgwriter
      - activity
    hosts:
      - "postgres://172.21.197.195:5444/SANTEDB"
    period: 10s

output.elasticsearch:
  hosts:
    - "172.21.197.152:9200"
#  password: changeme
#  username: elastic

logging.level: debug

ruflin · November 16, 2017, 10:17pm

Any chance you could post a little bit a longer log and from the beginning when metricbeat starts up and then wait for about 60s? For a test run could you disable the system module?

For the log best paste it into a gist and link it here.

Dinesh_Senthil_Kumar · November 17, 2017, 5:55am

Hi Ruflin

this is the error message i am getting, post commenting out the system module and trying only for the pgsql.

"error": "QueryStats: pq: unexpected error: "setting PGLOCALEDIR not supported"",
** "metricset":**

the PGLOCALEDIR dir is already set, let me know if metricbeat expects any specific locale. the below locale are present in my system. thanks in advance
zh_TW
pt_BR
ta
ja
nb
zh_CN
it
de
id
es
fr
ro
cs
ru
sv
ko
tr
pl

ruflin · November 19, 2017, 11:14pm

From the go client library we use: pq package - github.com/lib/pq - Go Packages

Most environment variables as specified at PostgreSQL: Documentation: 16: 34.15. Environment Variables supported by libpq are also supported by pq. If any of the environment variables not supported by pq are set, pq will panic during connection establishment. Environment variables have a lower precedence than explicitly provided connection parameters.

Can you try unset the env variable?

In the future please paste text instead of screenshots.

Dinesh_Senthil_Kumar · November 29, 2017, 11:45am

HI Ruflin,
post unsetting the PGLOCALEDIR env variable i am able to proceed with metricbeat. thanks fot the guidance.

system · December 27, 2017, 11:45am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.