# Step 3 - Init and create the metricset failure

**URL:** <https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046>\
**Category:** Beats\
**Tags:** beats-development\
**Created:** [December 15, 2017, 9:05pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046 "2017-12-15T21:05:07Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![rvigeant](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rvigeant](https://discuss.elastic.co/u/rvigeant)\
**Post date:** [December 15, 2017, 9:05pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/1 "2017-12-15T21:05:07Z")

</div>

Hi,  
I know the same problem has been reported before but what I do is exactly what apparently solved the problem for the other developer.

I am trying to create a beat based on metricbeat and I am following the recipe described in the Beats Developer Guide.

I am using Python 2.7.12 with virtualenv 15.1.0

The problem is that during make setup, i gets a Python error when interpreting File "./vendor/github.com/elastic/beats/metricbeat/scripts/create\_metricset.py", line 105  
NameError: name 'raw\_input' is not defined

This indicates that for whatever strange reason it is running Python 3 instead of Python 2.7.

Here is the whole recipe that I followed:

ubuntu@beats:~$ virtualenv -p python2.7 ~/workspace/python

ubuntu@beats:~$ source ~/workspace/python2.7/bin/activate  
Running virtualenv with interpreter /usr/bin/python2.7  
New python executable in /home/ubuntu/workspace/python/bin/python2.7  
Also creating executable in /home/ubuntu/workspace/python/bin/python  
Installing setuptools, pip, wheel...done.  
ubuntu@beats:~$ source ~/workspace/python2.7/bin/activate

(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat$ go get [github.com/elastic/beats/metricbeat](http://github.com/elastic/beats/metricbeat)  
(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat$ cd src/github.com/  
(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com$ mkdir sops  
(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com$ cd sops  
(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com$ cd sops

(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/sops$ python ${GOPATH}/src/github.com/elastic/beats/script/generate.py --type=metricbeat  
Beat Name [Examplebeat]: dpwr  
Your Github Name [your-github-name]: sops  
Beat Path [[github.com/sops/dpwr](http://github.com/sops/dpwr)]:  
Firstname Lastname: Bob Marley

(python2.7) ubuntu@beats:~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/sops/dpwr$ make setup  
mkdir -p vendor/github.com/elastic/  
cp -R /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/elastic/beats vendor/github.com/elastic/  
rm -rf vendor/github.com/elastic/beats/.git  
make create-metricset  
make[1]: Entering directory '/home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/sops/dpwr'  
Traceback (most recent call last):  
File "./vendor/github.com/elastic/beats/metricbeat/scripts/create\_metricset.py", line 105, in   
args.module = raw\_input("Module name: ")  
NameError: name 'raw\_input' is not defined  
vendor/github.com/elastic/beats/metricbeat/Makefile:78: recipe for target 'create-metricset' failed  
make[1]: \*\*\* [create-metricset] Error 1  
make[1]: Leaving directory '/home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/sops/dpwr'  
Makefile:17: recipe for target 'setup' failed  
make: \*\*\* [setup] Error 2

I tried different ways, even correcting the faulty python source file, it seems to be overwritten therefore absolutely nothing works.

Thanks

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [December 19, 2017, 3:35am UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/2 "2017-12-19T03:35:14Z")

</div>

I don't really know what causes the above issue but I wonder how we can make this compatible to python 3? We should probably replace `raw_input` by `six.input()`?

---

<div class="post-metadata">

**Author:** ![rvigeant](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rvigeant](https://discuss.elastic.co/u/rvigeant)\
**Post date:** [December 19, 2017, 2:37pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/3 "2017-12-19T14:37:13Z")

</div>

I already tried to replace raw\_input () by input () in the source file "./vendor/github.com/elastic/beats/metricbeat/scripts/create\_metricset.py, however, this file seems to be overwritten during the make process, therefore the raw\_input () functions are back in the source file.

I'm currently investigating where and when the source file gets overwritten during the process.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [December 20, 2017, 3:12am UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/4 "2017-12-20T03:12:06Z")

</div>

How does your `Makefile` look like? I wonder if perhaps the global create\_metricset is used?

Perhaps a quick fix would be to open a PR against the beats repo a fix for python 3 for the generator? We should do that anyways.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [December 22, 2017, 4:33pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/5 "2017-12-22T16:33:37Z")

</div>

I've run into a similar issue when issue when python3 is set as my default python to bootstrap the beats python environment for testing.

I've looked at the `generate.py`, we don't do anything to mess with the environment.

Before running the generate command, can you do the following:

```auto
which python
python --version

```

I've created an [issue](https://github.com/elastic/beats/issues/5950) for the migration to Python 3.

---

<div class="post-metadata">

**Author:** ![rvigeant](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rvigeant](https://discuss.elastic.co/u/rvigeant)\
**Post date:** [December 22, 2017, 6:02pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/6 "2017-12-22T18:02:06Z")

</div>

Thanks Pier-Hugues.

I want to use python 2.7, not python 3.

Here are some details about the problem:

1- I'm using Python 2.7 all the way.  
I even removed all references to Python3 in .usr/bin/ as you can see  
here:

which python  
/usr/bin/python  
python --version  
Python 2.7.12

2- I've also tried using virtenv before running the command "make setup"  
but I get the same result.

virtualenv -p python2.7 ~/workspace/python2.7  
Running virtualenv with interpreter /usr/bin/python2.7  
New python executable in /home/ubuntu/workspace/python2.7/bin/python2.7  
Not overwriting existing python script  
/home/ubuntu/workspace/python2.7/bin/python (you must use  
/home/ubuntu/workspace/python2.7/bin/python2.7)  
Installing setuptools, pip, wheel...done.

source ~/workspace/python2.7/bin/activate  
(python2.7) ubuntu@beats:~$

It is still pointing to python 2.7 as shown here:

(python2.7) ubuntu@beats:~$ which python  
/home/ubuntu/workspace/python2.7/bin/python  
(python2.7) ubuntu@beats:~$ python --version  
Python 2.7.12

The probles occurs during the processing of "make setup"  
I ran the command make setup with the debug option and I see that at one  
stage python 3 gets setup in directory

3- The recipe followed is this:

go get [github.com/elastic/beats/metricbeat](http://github.com/elastic/beats/metricbeat)  
cd $GOPATH/src/github.com  
mkdir rvigeant8  
cd rvigeant8  
python ${GOPATH}/src/github.com/elastic/beats/script/generate.py  
--type=metricbeat  
Questions answered

Now I run make in debug mode to see all details:  
make -d setup \> make\_setup.logs  
Traceback (most recent call last):  
File  
"./vendor/github.com/elastic/beats/metricbeat/scripts/create\_metricset.py",  
line 105, in   
args.module = raw\_input("Module name: ")  
NameError: name 'raw\_input' is not defined  
make[1]: \*\*\* [create-metricset] Error 1  
make: \*\*\* [setup] Error 2

Now if we look at the debug data, I cut a few interesting part:s:

Updating goal targets....  
Considering target file 'create-metricset'.  
File 'create-metricset' does not exist.  
Considering target file 'python-env'.  
File 'python-env' does not exist.  
Considering target file  
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.  
Looking for an implicit rule for  
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.  
Trying pattern rule with stem 'requirements.txt'.

(more...).

In the following output section we see references to python3:  
No implicit rule found for  
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.  
Finished prerequisites of target file  
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.  
No need to remake target  
'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.  
Finished prerequisites of target file 'python-env'.  
Must remake target 'python3  
python-env'.  
Putting child 0x15955d0 (python-env) PID 1775 on the chain.  
Live child 0x15955d0 (python-env) PID 1775  
Using base prefix '/usr'  
New python executable in  
/home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/python3  
Also creating executable in  
/home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/python  
Installing setuptools, pip, wheel...done.  
Reaping winning child 0x15955d0 PID 1775  
Live child 0x15955d0 (python-env) PID 1785  
Reaping winning child 0x15955d0 PID 1785  
Removing child 0x15955d0 PID 1785 from chain.  
Successfully remade target file 'python-env'.  
Finished prerequisites of target file 'create-metricset'.

The question is: Why is python-env set to python3 as we can see here?

ubuntu@beats: ls -l  
~/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat$  
ll build/python-env/bin/python  
lrwxrwxrwx 1 ubuntu ubuntu 7 Dec 22 17:38 build/python-env/bin/python -\>  
python3

Please do not hesitate if you more details  
Thanks

Richard Vigeant  
Programmeur-analyste/Programmer-analyst  
Sécurité informatique/IT Security  
Intact Corporation Financière/Intact Financial Corporation  
1-855-646-8228 poste/ext. 86757

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [December 27, 2017, 2:45am UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/7 "2017-12-27T02:45:04Z")

</div>

I wonder if there is an issue with the `setup` command. I assume `copy-vendor` already worked on your side, so can you try to run `make create-metricset` manually and then `make update`? I wonder if one of the commands fails.

---

<div class="post-metadata">

**Author:** ![rvigeant](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rvigeant](https://discuss.elastic.co/u/rvigeant)\
**Post date:** [January 4, 2018, 5:14pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/8 "2018-01-04T17:14:30Z")

</div>

copy-vendor and make create-metricset work succesfully.

However make update fails with the following message:

cp: cannot stat 'module/\*/\_meta/kibana': No such file or directory  
vendor/github.com/elastic/beats/metricbeat/Makefile:23: recipe for target 'kibana' failed  
make: [kibana] Error 1 (ignored)  
Traceback (most recent call last):  
File "./vendor/github.com/elastic/beats/script/generate\_imports.py", line 7, in   
from generate\_imports\_helper import comment, get\_importable\_lines  
ImportError: No module named generate\_imports\_helper  
vendor/github.com/elastic/beats/metricbeat/Makefile:57: recipe for target 'imports' failed  
make: \*\*\* [imports] Error 1

---

<div class="post-metadata">

**Author:** ![rvigeant](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rvigeant](https://discuss.elastic.co/u/rvigeant)\
**Post date:** [January 5, 2018, 9:32pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/9 "2018-01-05T21:32:27Z")

</div>

Got rid of the fatal error:  
ImportError: No module named generate\_imports\_helper  
by copying the script directory to my own beat directory like this:  
cp -r $GOPATH/src/github.com/elastic/beats/metricbeat/scripts .

Now "make setup" completes since the "make: [kibana] Error 1 is ignored

Then I can copy the \_meta/kibana directory from another beat if needed.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [January 9, 2018, 5:03am UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/10 "2018-01-09T05:03:02Z")

</div>

I wonder if this means we need to change the generator on our side.

@pierhugues Did you have a look into this?

---

<div class="post-metadata">

**Author:** ![rvigeant](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@rvigeant](https://discuss.elastic.co/u/rvigeant)\
**Post date:** [January 10, 2018, 4:10pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/11 "2018-01-10T16:10:52Z")

</div>

@pierhugues @ruflin I was finally able to create my module and metricset by modifying the procedure and a makefile,

Following is the revised recipe that works.  
All changes that I had to do are indicated with \*\*\* in my comments.

```
export GOPATH=~/workspace/metricbeat 

# *** 1) Must issue this export variable to fix the mix useage of python 2.7 and python 3 during "make setup"
export VIRTUALENV_PARAMS="-p python2.7" 

mkdir ~/workspace
cd ~/workspace
rm -rf metricbeat

go get github.com/elastic/beats/metricbeat
python ${GOPATH}/src/github.com/elastic/beats/script/generate.py --type=metricbeat
    Questions:
        Beat Name [Examplebeat]: metricbeat_
        Your Github Name [your-github-name]: my-github-name
        Beat Path [github.com/my-github-name/metricbeat]: github.com/my-githubname/beats/metricbeat
        Firstname Lastname: Abc Def_

cd metricbeat/src/github.com/my-github-name/beats/metricbeat/ 

# *** 2) Must do manually the following command to fix the first fatal erreor - "ImportError: No module named generate_imports_helper"
cp -r $GOPATH/src/github.com/elastic/beats/metricbeat/scripts . # Important to fix problem 1
  
  
# *** 3) Edit the the following file: 
    nano ${GOPATH}/src/github.com/elastic/beats/metricbeat/Makefile_
# *** Replace the following line in target section "kibana:"
    @-cp -r module/*/_meta/kibana _meta/
# ***with this line:
    @-cp -r ../../../elastic/beats/metricbeat/module/*/_meta/kibana _meta/

   

# Create the the module and first metricset
make setup
Module name: datapower
Metricset name: status

# Or create new metricset to_
When creating new metricsets:
    make create-metricset
    make update
    make collect
```

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [January 11, 2018, 3:06pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/12 "2018-01-11T15:06:31Z")

</div>

@rvigeant Thanks for the update, I haven't been able to reproduce it locally so far, I will do another shot.

I will check to add the following line to the developer doc.

```auto
export VIRTUALENV_PARAMS="-p python2.7" 

```

---

<div class="post-metadata">

**Author:** ![dcroonen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dcroonen/32/27450_2.png) [@dcroonen](https://discuss.elastic.co/u/dcroonen)\
**Post date:** [January 29, 2018, 2:32pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/13 "2018-01-29T14:32:58Z")

</div>

I was having the same problem and can confirm that modifying the procedure and a makefile also solved the problem for me.

---

<div class="post-metadata">

**Author:** ![msvechla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/msvechla/32/55828_2.png) [@msvechla](https://discuss.elastic.co/u/msvechla)\
**Post date:** [January 30, 2018, 8:39pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/14 "2018-01-30T20:39:18Z")

</div>

I was also having the exact same problem and got it working after switching to python 2.7 and executing the steps described above.

Thanks for the hints! Nevertheless it would be great to have this working under python 3 as well.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 27, 2018, 8:39pm UTC](https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/15 "2018-02-27T20:39:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.