Create Custom Build image for customized kibana 6.4.3 source code

vishal_M · July 24, 2019, 5:49pm

Hello Team,

I am trying to create docker image with some customized kibana source code. So that, i can deploy it in a kubernetes cluster.

Here are the steps i followed so far.

Cloned kibana dev source code (GitHub - elastic/kibana: Your window into the Elastic Stack).
Made a Checkout 6.4.3 version into my local MacOS machine.
Customized kibana source code. // made changes in one file (filter_bar.less)
changed Node version 8.11.4
Yarn Install
yarn kbn bootstrap
yarn start
Yarn build ---> (6 packages were generated into the kibana/target directory for darwin, linux, winows versions and some OSS versions (zip files) )
Extracted darwin and windows versions and tested them using in mac and windows using run/kibana and works as expected.

And as per Jarpy's instructions in this below POST

I have cloned kibana-docker 6.4.3 ( GitHub - elastic/kibana-docker at 6.4 )into my local machine.
Also, i have installed virtualenv, Python docker in my local machine.

I know there is no need to create a docker file by myself to create a docker image. Please can you provide me steps editing the docker template (make file or docker related files under template folder) for generating the docker image using my local source code of kibana/target along with the commands.

FYI:I am pretty new to docker .

Thanks in advance and any help is highly appreciated.

Brandon_Kobel · July 24, 2019, 9:21pm

Did you run the equivalent of this command substituting the ARTIFACTS_DIR path to be relative to your cloned kibana repo?

ARTIFACTS_DIR=/home/jarpy/src/ ELASTIC_VERSION=6.4.4 make release-manager-release

vishal_M · July 29, 2019, 12:55pm

Thanks for your reply Brandon,
i tried running
env ARTIFACTS_DIR=/Users/vmacha012/Documents/Project/PluginDev/kibana (My local source code PATH) appending with ELASTIC_VERSION=6.4.3 make release-manager-release . but, it's throwing few errors
make[1]: *** [build-from-local-artifacts] Error 127
make: *** [release-manager-release] Error 2

And can see newly generated (Dockerfile-full & Docker -oss) files under /kibana-docker/build/kibana . Please can you refer to the screenshots.

vishal_M · July 29, 2019, 1:00pm

And when i tried running it as build image. It took nearly 5 mins to "sending build context to docker deamon" and it got failed in 5th step .. Please refer to snapshot 1 (It is due to the tar.gz file)

and when i tried uncompressing the file and changing the relative path in generated docker file and triggered a build ...it got exited with non-zero code:7 (Please refer to the attached screenshot 2
And the next command didn't run (I mean from step6 to Step 15)

Can some one help me in building the docker-image asap. As i have to move it to production environment early next week

Below is the file that got generated when i run env ARTIFACTS_DIR=/Users/vmacha012/Documents/Project/PluginDev/kibana ELASTIC_VERSION=6.4.3 make release-manager-release:slight_smile:

Brandon_Kobel · July 29, 2019, 5:31pm

@vishal_M, using curl against localhost isn't going to work. From the containers perspective, localhost is the container itself. You shouldn't need to modify the Dockerfile and you should just be able to specify the ARCHIVES_DIR environment variable as originally recommended by jarpy.

vishal_M · July 29, 2019, 5:35pm

Hello Brandon, I have used the below command to generate docker file.

env ARTIFACTS_DIR=/Users/vmacha012/Documents/Project/PluginDev/kibana ELASTIC_VERSION=6.4.3 make release-manager-release

I am using an auto-generated docker file to build the image. I haven't updated any line (http://localhost:8000)..

I have a question: Do i need to checkin my customized code to Kibana source code repository(https://github.com/elastic/kibana) by creating a new branch along with the target folder ?

Brandon_Kobel · July 29, 2019, 6:22pm

@vishal_M, apologies I didn't realize that the old generated Dockerfile from https://github.com/elastic/kibana-docker/tree/6.4 is using curl in this manner.

From /Users/vmacha012/Documents/Project/PluginDev/ if you run python -m SimpleHTTPServer it will host a simple web server which will allow you to use curl from within the Dockerfile.

vishal_M · July 29, 2019, 7:08pm

Hello Brandon,
Just now, i have realized that the files that have generated under (kibana source code) kibana/target folder has a different name ( [kibana-6.4.3-SNAPSHOT-linux-x86_64.tar.gz]). But, in the docker file it is generated as http://localhost:8000/kibana/target/kibana-6.4.3-linux-x86_64.tar.gz . Please refer to the attached snapshot (Left-top).
I have started running the web server using python -m SimpleHTTPServer as per your instructions and can accessible the files using url:(plz refer to the right side of the snapshot).
FYI : I have tried to build image by renaming http://localhost:8000/kibana/target/kibana-6.4.3-SNAPSHOT-linux-x86_64.tar.gz in the docker file. But, still there seems to be the same issue(left-bottom).

Brandon_Kobel · July 29, 2019, 7:47pm

You're running docker build directly instead of using the Makefile. The Makefile does some rather complex logic https://github.com/elastic/kibana-docker/blob/907a9be2a38bea7e833e2f3d4557251780aaf6e4/Makefile#L67-L79. I'd recommend instead renaming your /kibana/target/*.tar.gz to match the expected files, or augmenting the kibana-docker/templates/Dockerfile.j2 to work with your existing archive naming.

vishal_M · July 29, 2019, 9:11pm

@Brandon_Kobel ,
I believe, Make file will be first entry point that triggers when we hit ('make release-manager-release' ) it will generate us a docker file with (https://artifacts.elastic.co/downloads/kibana/kibana-6.4.3-linux-x86_64.tar.gz ) and seems to be successfully built (plz refer snapshot 1).

When we provide ARTIFACTS_DIR , it will generate us a docker file with localpath of our source code (kibana/docker ) this is where it is throwing errors. (plz refer snapshot 2)

.

vishal_M · July 30, 2019, 5:35pm

@Brandon_Kobel , can you please clone this repos and try the instructions which were in my first post (install dependencies, yarn kbn bootstrap, build)

https://github.com/elastic/kibana/tree/6.4
2.https://github.com/elastic/kibana-docker/tree/6.4 and see if you are also facing the same issue.

Else can you pull-in @jarpy . So that, i can request him to look into this asap..

Here is the complete post: Create Custom Build image for customized kibana 6.4.3 source code

Thank you very much!!

jarpy · July 31, 2019, 12:03am

Hi Vishal,

I can't see the command you ran. Please send a complete transcript from the shell, as a GitHub Gist. Screenshots of shells are hard to read, impossible to search and tend to omit details.

Also, is there any chance of using Kibana 7? It has a completely different Docker build system that is integrated with the main source code.

vishal_M · July 31, 2019, 4:32pm

Hello @jarpy && @Brandon_Kobel ,
I have tried creating kibana docker-image for version 6.4.3. Please find the attached gist url let me know if you have any issues.

I have run a web-server in my local machine using this command : python -m SimpleHTTPServer
https://gist.github.com/vishal-macha/bce7df5fc062b4ce0ef768a17a94259b .

we tried to deploy as kibana docker image7.1.1 .But, its has some issues with kubernetes cluster.

Also, We tried customizing kibana source code 7.0.0 and deploy it as a docker image. It also has a similar issue ( like generating custom docker image with kibana source code 6.4.3)(didn't tried deploying a basic kibana 7.1.1 image from elastic artifacts in kubernetes).

We were able to deploy 6.4.3 in kubernetes cluser successfully. So ended up customizing kibana source code 6.4.3. But, facing issues while trying to generate a custom docker image.

Any help is highly appreciated.

vishal_M · August 6, 2019, 1:49pm

Hello @jarpy & @Brandon_Kobel ,
Please can someone look into this gist and provide me the instructions on how to generate custom docker image successfully.

gist.github.com

https://gist.github.com/vishal-macha/bce7df5fc062b4ce0ef768a17a94259b

complete_trace.md

Last login: Wed Jul 31 10:42:56 on ttys008
HQSML-1689749:~ vmacha012$ python --version
Python 2.7.10
HQSML-1689749:~ vmacha012$ python3 --version
Python 3.7.4
HQSML-1689749:~ vmacha012$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.2.0
Thread model: posix

This file has been truncated. show original

Also, Please let me know if you require any additional information.
Any help is highly appreciated.

Brandon_Kobel · August 6, 2019, 9:06pm

@vishal_M your ARTIFACTS_DIR is incorrect, it should be ARTIFACTS_DIR=/Users/vmacha012/Documents/Project/Docker/. Also before running make release-manager-release try doing a docker pull python:3.

system · September 3, 2019, 9:07pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.