Step 3 - Init and create the metricset failure

Elastic Stack Beats
1

Hi,
I know the same problem has been reported before but what I do is exactly what apparently solved the problem for the other developer.

I am trying to create a beat based on metricbeat and I am following the recipe described in the Beats Developer Guide.

I am using Python 2.7.12 with virtualenv 15.1.0

The problem is that during make setup, i gets a Python error when interpreting File "./vendor/github.com/elastic/beats/metricbeat/scripts/create_metricset.py", line 105
NameError: name 'raw_input' is not defined

This indicates that for whatever strange reason it is running Python 3 instead of Python 2.7.

Here is the whole recipe that I followed:

ubuntu@beats:~$ virtualenv -p python2.7 ~/workspace/python

ubuntu@beats:~$ source ~/workspace/python2.7/bin/activate
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in /home/ubuntu/workspace/python/bin/python2.7
Also creating executable in /home/ubuntu/workspace/python/bin/python
Installing setuptools, pip, wheel...done.
ubuntu@beats:~$ source ~/workspace/python2.7/bin/activate

(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat$ go get github.com/elastic/beats/metricbeat
(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat$ cd src/github.com/
(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com$ mkdir sops
(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com$ cd sops
(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com$ cd sops

(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/sops$ python ${GOPATH}/src/github.com/elastic/beats/script/generate.py --type=metricbeat
Beat Name [Examplebeat]: dpwr
Your Github Name [your-github-name]: sops
Beat Path [github.com/sops/dpwr]:
Firstname Lastname: Bob Marley

(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/sops/dpwr$ make setup
mkdir -p vendor/github.com/elastic/
cp -R /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git
make create-metricset
make[1]: Entering directory '/home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/sops/dpwr'
Traceback (most recent call last):
File "./vendor/github.com/elastic/beats/metricbeat/scripts/create_metricset.py", line 105, in
args.module = raw_input("Module name: ")
NameError: name 'raw_input' is not defined
vendor/github.com/elastic/beats/metricbeat/Makefile:78: recipe for target 'create-metricset' failed
make[1]: *** [create-metricset] Error 1
make[1]: Leaving directory '/home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/sops/dpwr'
Makefile:17: recipe for target 'setup' failed
make: *** [setup] Error 2

I tried different ways, even correcting the faulty python source file, it seems to be overwritten therefore absolutely nothing works.

Thanks

1 Like
2

I don't really know what causes the above issue but I wonder how we can make this compatible to python 3? We should probably replace raw_input by six.input()?

3

I already tried to replace raw_input () by input () in the source file "./vendor/github.com/elastic/beats/metricbeat/scripts/create_metricset.py, however, this file seems to be overwritten during the make process, therefore the raw_input () functions are back in the source file.

I'm currently investigating where and when the source file gets overwritten during the process.

4

How does your Makefile look like? I wonder if perhaps the global create_metricset is used?

Perhaps a quick fix would be to open a PR against the beats repo a fix for python 3 for the generator? We should do that anyways.

5

I've run into a similar issue when issue when python3 is set as my default python to bootstrap the beats python environment for testing.

I've looked at the generate.py, we don't do anything to mess with the environment.

Before running the generate command, can you do the following:

which python
python --version

I've created an issue for the migration to Python 3.

6

Thanks Pier-Hugues.

I want to use python 2.7, not python 3.

Here are some details about the problem:

1- I'm using Python 2.7 all the way.
I even removed all references to Python3 in .usr/bin/ as you can see
here:

which python
/usr/bin/python
python --version
Python 2.7.12

2- I've also tried using virtenv before running the command "make setup"
but I get the same result.

virtualenv -p python2.7 ~/workspace/python2.7
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in /home/ubuntu/workspace/python2.7/bin/python2.7
Not overwriting existing python script
/home/ubuntu/workspace/python2.7/bin/python (you must use
/home/ubuntu/workspace/python2.7/bin/python2.7)
Installing setuptools, pip, wheel...done.

source ~/workspace/python2.7/bin/activate
(python2.7) ubuntu@beats:~$

It is still pointing to python 2.7 as shown here:

(python2.7) ubuntu@beats:~$ which python
/home/ubuntu/workspace/python2.7/bin/python
(python2.7) ubuntu@beats:~$ python --version
Python 2.7.12

The probles occurs during the processing of "make setup"
I ran the command make setup with the debug option and I see that at one
stage python 3 gets setup in directory

3- The recipe followed is this:

go get github.com/elastic/beats/metricbeat
cd $GOPATH/src/github.com
mkdir rvigeant8
cd rvigeant8
python ${GOPATH}/src/github.com/elastic/beats/script/generate.py
--type=metricbeat
Questions answered

Now I run make in debug mode to see all details:
make -d setup > make_setup.logs
Traceback (most recent call last):
File
"./vendor/github.com/elastic/beats/metricbeat/scripts/create_metricset.py",
line 105, in
args.module = raw_input("Module name: ")
NameError: name 'raw_input' is not defined
make[1]: *** [create-metricset] Error 1
make: *** [setup] Error 2

Now if we look at the debug data, I cut a few interesting part:s:

Updating goal targets....
Considering target file 'create-metricset'.
File 'create-metricset' does not exist.
Considering target file 'python-env'.
File 'python-env' does not exist.
Considering target file
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
Looking for an implicit rule for
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
Trying pattern rule with stem 'requirements.txt'.

(more...).

In the following output section we see references to python3:
No implicit rule found for
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
Finished prerequisites of target file
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
No need to remake target
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
Finished prerequisites of target file 'python-env'.
Must remake target 'python3
python-env'.
Putting child 0x15955d0 (python-env) PID 1775 on the chain.
Live child 0x15955d0 (python-env) PID 1775
Using base prefix '/usr'
New python executable in
/home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/python3
Also creating executable in
/home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/python
Installing setuptools, pip, wheel...done.
Reaping winning child 0x15955d0 PID 1775
Live child 0x15955d0 (python-env) PID 1785
Reaping winning child 0x15955d0 PID 1785
Removing child 0x15955d0 PID 1785 from chain.
Successfully remade target file 'python-env'.
Finished prerequisites of target file 'create-metricset'.

The question is: Why is python-env set to python3 as we can see here?

ubuntu@beats: ls -l
~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat$
ll build/python-env/bin/python
lrwxrwxrwx 1 ubuntu ubuntu 7 Dec 22 17:38 build/python-env/bin/python ->
python3

Please do not hesitate if you more details
Thanks

Richard Vigeant
Programmeur-analyste/Programmer-analyst
Sécurité informatique/IT Security
Intact Corporation Financière/Intact Financial Corporation
1-855-646-8228 poste/ext. 86757

7

I wonder if there is an issue with the setup command. I assume copy-vendor already worked on your side, so can you try to run make create-metricset manually and then make update? I wonder if one of the commands fails.

8

copy-vendor and make create-metricset work succesfully.

However make update fails with the following message:

cp: cannot stat 'module/*/_meta/kibana': No such file or directory
vendor/github.com/elastic/beats/metricbeat/Makefile:23: recipe for target 'kibana' failed
make: [kibana] Error 1 (ignored)
Traceback (most recent call last):
File "./vendor/github.com/elastic/beats/script/generate_imports.py", line 7, in
from generate_imports_helper import comment, get_importable_lines
ImportError: No module named generate_imports_helper
vendor/github.com/elastic/beats/metricbeat/Makefile:57: recipe for target 'imports' failed
make: *** [imports] Error 1

9

Got rid of the fatal error:
ImportError: No module named generate_imports_helper
by copying the script directory to my own beat directory like this:
cp -r $GOPATH/src/github.com/elastic/beats/metricbeat/scripts .

Now "make setup" completes since the "make: [kibana] Error 1 is ignored

Then I can copy the _meta/kibana directory from another beat if needed.

10

I wonder if this means we need to change the generator on our side.

@pierhugues Did you have a look into this?

11

@pierhugues @ruflin I was finally able to create my module and metricset by modifying the procedure and a makefile,

Following is the revised recipe that works.
All changes that I had to do are indicated with *** in my comments.

export GOPATH=~/workspace/metricbeat 

# *** 1) Must issue this export variable to fix the mix useage of python 2.7 and python 3 during "make setup"
export VIRTUALENV_PARAMS="-p python2.7" 


mkdir ~/workspace
cd ~/workspace
rm -rf metricbeat

go get github.com/elastic/beats/metricbeat
python ${GOPATH}/src/github.com/elastic/beats/script/generate.py --type=metricbeat
    Questions:
        Beat Name [Examplebeat]: metricbeat_
        Your Github Name [your-github-name]: my-github-name
        Beat Path [github.com/my-github-name/metricbeat]: github.com/my-githubname/beats/metricbeat
        Firstname Lastname: Abc Def_

cd metricbeat/src/github.com/my-github-name/beats/metricbeat/ 

# *** 2) Must do manually the following command to fix the first fatal erreor - "ImportError: No module named generate_imports_helper"
cp -r $GOPATH/src/github.com/elastic/beats/metricbeat/scripts .   # Important to fix problem 1
  
  
# *** 3) Edit the the following file: 
    nano ${GOPATH}/src/github.com/elastic/beats/metricbeat/Makefile_
# *** Replace the following line in target section "kibana:"
    @-cp -r module/*/_meta/kibana _meta/
# ***with this line:
    @-cp -r ../../../elastic/beats/metricbeat/module/*/_meta/kibana _meta/

   

# Create the the module and first metricset
make setup
Module name: datapower
Metricset name: status

# Or create new metricset to_
When creating new metricsets:
    make create-metricset
    make update
    make collect
12

@rvigeant Thanks for the update, I haven't been able to reproduce it locally so far, I will do another shot.

I will check to add the following line to the developer doc.

export VIRTUALENV_PARAMS="-p python2.7"
13

I was having the same problem and can confirm that modifying the procedure and a makefile also solved the problem for me.

14

I was also having the exact same problem and got it working after switching to python 2.7 and executing the steps described above.

Thanks for the hints! Nevertheless it would be great to have this working under python 3 as well.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.