Django Python-agent : APM Quick Start Produces “No services were found”

Elastic Observability APM
1

Hi

I work on CentOS7 VmWARE virtual machine :
Django + apm-agent : Machine 1 ip : 192.168.232.135
Apm-server + Elasticsearch + Kibana : 192.168.232.131

I try to configure stack Django == Apm-agent == Apm-Server == Elasticsearch == Kibana.

But when i click on apm menu in Kibana, I get :

No services were found

By clicking on "setup Instructions" I get :
For Server status : You have correctly setup APM-Server
For Agent status : Data successfully received from one or more agents
For Load Kibana objects : 16 saved objects successfully added

But when i click on launch APM i get nothing in APM kibana's page, no agent.

My APM-agent configuration :

My django version is :

[root@localhost ~]# python
Python 2.7.5 (default, Jul 13 2018, 13:06:57)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import django
django.VERSION
(1, 8, 18, 'final', 0)

I follow install instruction from here APM avec Elastic Observability | Elastic for django app :

I install elastic-apm in my django virtualenv :

pip install elastic-apm

version of elastic-apm :
(django_pod) [pod@localhost root]$ pip list | grep elastic-apm
elastic-apm 3.0.1

Installed applications list

INSTALLED_APPS = (
# put it in first !!
# Installation — django-modeltranslation dev documentation
'modeltranslation',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
# Pages statiques
'django.contrib.sites',
'django.contrib.flatpages',
# Applications tierces
'ckeditor',
'filer',
'easy_thumbnails',
'elasticapm.contrib.django',
# https://bitbucket.org/cpcc/django-cas -> application modifiée pour ajout
# gateway et double authentification
'django_cas_gateway',
'taggit',
'taggit_templatetags',
'djangoformsetjs',
'captcha',
'bootstrap3',
'rest_framework',
'rest_framework.authtoken',
'lti_provider',
# Applications locales
'pods',
'core',
'h5pp',
)

Activated middleware components

MIDDLEWARE_CLASSES = (
'elasticapm.contrib.django.middleware.TracingMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
# Pages statiques
'django.contrib.flatpages.middleware.FlatpageFallbackMiddleware',
)

Configuration ELASTIC_APM

ELASTIC_APM = {
'DEBUG' : True,
'SERVICE_NAME':'PODv1',
'SERVER_URL' : 'http://192.168.232.131:8200',
}

I split my post cause of post caractère limit

2

My APM-Serveur configuration :

I use elastic repo to install APM-server via yum install :

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
priority=10
protect=1

apm-server version is :

[root@localhost apm-server]# rpm -qa | grep apm-server
apm-server-6.4.1-1.x86_64

apm-server configuration:

   host: "localhost:8200"
   concurrent_requests: 5
   rum:
           enabled: true
   queue:
           mem:
                    events: 4096      
   setup.template.settings:
           index:
                     number_of_shards: 1
                     codec: best_compression
   output.elasticsearch:
          hosts: ["localhost:9200"]
          enabled: true
   indices:
- index: "apm-%{[beat.version]}-sourcemap"
  when.contains:
    processor.event: "sourcemap"

- index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "error"

- index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "transaction"

- index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "span"

- index: "apm-%{[beat.version]}-metric-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "metric"

- index: "apm-%{[beat.version]}-onboarding-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "onboarding"

#to get more info I put loglevel to debug :

logging.level: debug
logging.selectors: ["*"]
logging.metrics.enabled: false
logging.metrics.period: 30s
logging.to_files: true
logging.files:
path: /var/log/apm-server
name: apm-server
rotateeverybytes: 10485760
keepfiles: 7
permissions: 0600

Result after log activation in /var/log/apm-server/apm-server :

2018-09-25T09:04:11.112+0200 INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://localhost:9200))
2018-09-25T09:04:11.112+0200 DEBUG [elasticsearch] elasticsearch/client.go:688 ES Ping(url=http://localhost:9200)
2018-09-25T09:04:11.123+0200 DEBUG [elasticsearch] elasticsearch/client.go:711 Ping status code: 200
2018-09-25T09:04:11.124+0200 INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.4.1
2018-09-25T09:04:11.124+0200 DEBUG [elasticsearch] elasticsearch/client.go:730 HEAD http://localhost:9200/_template/apm-6.4.1
2018-09-25T09:04:11.128+0200 INFO template/load.go:129 Template already exists and will not be overwritten.
2018-09-25T09:04:11.128+0200 INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://localhost:9200)) established
2018-09-25T09:04:11.143+0200 DEBUG [elasticsearch] elasticsearch/client.go:321 PublishEvents: 1 events have been published to elasticsearch in 15.043264ms.
2018-09-25T09:04:11.143+0200 DEBUG [memqueue] memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 1]
2018-09-25T09:04:11.143+0200 DEBUG [memqueue] memqueue/eventloop.go:535 broker ACK events: count=1, start-seq=1, end-seq=1

2018-09-25T09:04:11.143+0200 DEBUG [memqueue] memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2018-09-25T09:04:11.143+0200 DEBUG [memqueue] memqueue/ackloop.go:131 ackloop: done send ack

and nothing log after .....

3

My Elasticsearch configuration :

I follow this guide for elasticsearch installation : Install Elasticsearch with RPM | Elasticsearch Guide [8.11] | Elastic

I make a yum install from same repo as apm-server

elasticsearch version :

[root@localhost apm-server]# rpm -qa | grep elasticsearch
elasticsearch-6.4.1-1.noarch

elasticsearch configuration :

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: APM-application

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: APM-1

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

Result of elasticsearch configuration :

[root@localhost ~]# curl localhost:9200
{
"name" : "APM-1",
"cluster_name" : "APM-application",
"cluster_uuid" : "p5dqXl57Qd2mdXz7QqFkhQ",
"version" : {
"number" : "6.4.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "e36acdb",
"build_date" : "2018-09-13T22:18:07.696808Z",
"build_snapshot" : false,
"lucene_version" : "7.4.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

My Kibana configuration :

I follow this guide for kibana installation : Install Kibana with RPM | Kibana Guide [8.11] | Elastic

I make a yum install from same repo as apm-server

Kibana version :

[root@localhost kibana]# rpm -qa | grep kibana
kibana-6.4.1-1.x86_64

kibana configuration :

server.port: 5601
server.host: "192.168.232.131"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"
kibana.defaultAppId: "home"
elasticsearch.shardTimeout: 0
logging.dest: /var/log/kibana/kibana.log
logging.verbose: true
apm_oss.errorIndices: apm-*
apm_oss.spanIndices: apm-*
apm_oss.transactionIndices: apm-*
apm_oss.onboardingIndices: apm-*

What did i miss in my stack configuration?

cordialy,

4

Hi,
for identifying the issue, I suggest you first check if there have been data written to your Elasticsearch. You can do so by running e.g. following query to check for transactions in the Kibana Dev Tools and check for hits.total in the response:

GET apm*/_search
{
  "query": {
    "term": {
      "processor.name": "transaction"
    }
  }
}

In case no documents could be found, please check your python agent logs for any error messages. The Server logs you are showing do not contain any hint of requests from the agent. The Server config file shows you are using a local setup for the APM Server host: "localhost:8200". You can do a curl request to the APM Server to see if it is actually reachable, e.g. curl localhost:8200.

5

hi

and thank for your answer.

I test your command in DevTools :

GET apm*/_search
{
"query": {
"term": {
"processor.name": "transaction"
}
}
}

And I get this in return :

{
"took": 7,
"timed_out": false,
"_shards": {
"total": 4,
"successful": 4,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits":
}
}

So there is no document.

First I try a telnet from my python-agent VM to my server-apm VM and i get :

[root@localhost elasticsearch]# telnet 192.168.232.131 8200
Trying 192.168.232.131...
telnet: connect to address 192.168.232.131: Connection refused

So i change the host value in apm-server configuration to :

host: "192.168.232.131:8200"

then retry telnet :

[root@localhost elasticsearch]# telnet 192.168.232.131 8200
Trying 192.168.232.131...
Connected to 192.168.232.131.
Escape character is '^]'.
^C

I try to curl but I get this :

[root@localhost ~]# curl -v 192.168.232.131:8200

  • About to connect() to 192.168.232.131 port 8200 (#0)
  • Trying 192.168.232.131...
  • Connected to 192.168.232.131 (192.168.232.131) port 8200 (#0)

GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: 192.168.232.131:8200
Accept: /

< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Wed, 26 Sep 2018 13:22:21 GMT
< Content-Length: 19
<
404 page not found

I try this too :

[root@localhost ~]# curl -v 192.168.232.131:8200/v1/transactions

  • About to connect() to 192.168.232.131 port 8200 (#0)
  • Trying 192.168.232.131...
  • Connected to 192.168.232.131 (192.168.232.131) port 8200 (#0)

GET /v1/transactions HTTP/1.1
User-Agent: curl/7.29.0
Host: 192.168.232.131:8200
Accept: /

< HTTP/1.1 405 Method Not Allowed
< Content-Type: application/json
< Date: Wed, 26 Sep 2018 13:24:41 GMT
< Content-Length: 44
<

  • Connection #0 to host 192.168.232.131 left intact
    {"error":"only POST requests are supported"}

I'm looking for how to activate python agents logs? how to do that?

cordialy

6

It sounds like there was an actual connection problem between the agent and the server, that you solved with changing the host configuration in the server. When running an earlier version than 6.4.0 it is expected to get a 404 for calling <apm-server-host>:<port>, you could try to call <apm-server-host>:<port>/healthcheck instead.

Have you checked if this actually fixed your issues and data are showing up now in Kibana?

You can find more information about how to control logging in your django app .

1 Like
7

Hi

Have you checked if this actually fixed your issues and data are showing up now in Kibana?

was a good advice, the problem come from host value configuration in apm-server.

now data are showing up in Kibana great ;).

I go continue to test elastic apm solution.

Thank a lot for your help.

cordialy

1 Like
8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.