- Gitlab Ci Cd Example
- Gitlab Ci Cd Pipeline Example
- How Make A If Statement In The CI File
- Before_script Run Shell Script - GitLab CI/CD
- Before_script Not Run When Includes Are Used
- Forum.gitlab.com › T › Execute-script-python-usingExecute Script Python Using Curl - GitLab CI/CD - GitLab Forum
In the previous post I described how to run own GitLab server with CI runner.In this one, I’m going to walk through my experience of configuring GitLab-CI for one of my projects.I faced few problems during this process, which I will highlight in this post.
Some words about the project:
- Python/Flask backend with PostgreSQL as a database, with the bunch of unittests.
- React/Reflux in frontend with Webpack for bundling. No JS tests.
- Frontend and backend are bundled in a single docker container and deployed to Kubernetes
Building the first container
First thing after importing repository into gitlab we need to create .gitlab-ci.yml
.Gitlab itself has a lot of useful information about CI configuration,for example this andthis.
This script can be considered the main function of the extract, and is the file which gets run as the starting point of the extract DAG. When not to use Python Since this style guide is for the entire data team, it is important to remember that there is a time and place for using Python and it is usually outside of the data modeling phase. Setup the Gitlab CI/CD pipeline to push the Node.js code to the server and deploy it whenever code is merged to master branch. Step 1 - Clone the repository in the server using deploy token Using a deploy token ensures that we can pull the code without the need to enter the credentials manually or save the credentials in the deploy script.
Previously I used to do following steps to test/compile/deploy manually:
- npm run build - to build all js/css
- build.sh - to build docker container
- docker-compose -f docker-compose-test.yml run –rm api-test nosetests
- deploy.sh - to deploy it
Everything is pretty common for most of the web projects.So, I started with a simple .gitlab-ci.yml
file, trying to build my python code:
$CI_BUILD_TOKEN
- is a password for my docker registry, saved in projects variables in GitLab project settings
And it actually worked fine. I got my docker container built and pushed to the registry.Next step was to be able to run tests. This is where things became not so obvious.For testing, we need to add another stage to our gitlab-ci file.
Testing. First try
After reading posts like thisI thought that it will be as easy as declaring service and adding test stage:
Garmin the memory card has reached the end of its useful life. But it didn’t work. As far as I understood it, it’s because of docker-in-docker runner.Basically, we have 2 containers - docker:latest and postgresql:9.5, and they are linked perfectly fine.But then we’re bootstrapping our own container, inside this docker container.And this container can’t use docker links to access postgres, because its outside of its scope.
Testing. Second try
Then I tried to use my container as and image for the test stage and service declared in test stage itself, like this:
But it also didn’t work because of authentication.
My docker registry needs authentication which I do in before_script
stage. But before_script is being called after the services.I assume that this method should work for public images.
Testing. Third try. Kinda working solution
So I decided to try to use docker-compose in tests as I was doing manually since.Docker-compose should be able to run and link everything together.And since this page in documentation says:
GitLab CI allows you to use Docker Engine to build and test docker-based projects.
This also allows to you to use docker-compose and other docker-enabled tools.
Gitlab Ci Cd Example
I was very confused when I was not able to use docker-compose, since docker:latest image has no docker-compose installed.I spent some time googling and trying to install compose inside the container,and ended up using image jonaskello/docker-and-compose instead of the recommended one.
So my test stage changed to this:
This actually worked, but from time to time I was seeing weird race conditions during database provisioning.It’s not a big problem, and could be fixed easily. But I decided to try one more approach.
Testing. Final version
This time I decided to run postgres container during the test stage and link it to my test container.This requires to provide additional configuration to postgres container, but still this way is the closest to the original services approach.
Now, when we have our python container built and tested we need to add one more thing.We need to compile our javascript and css and put it into release container.This stage is actually going before the build.To be able to use files between stages we need to use some cache.In my case we need to cache only one directory.The one where webpack produces compiled files - src/static/dist
.
Cache is described in more details in official docs
As for release stage I’m simply going to tag container with build number and push it back to the registry.
Now we have a working CI pipeline which can build a container, run tests against this container and then push it to the registry.So far so good. The only problem is that all of this takes very long time. For my project, it takes about 20 minutes to finish all these stages.Most of the time is spent on building docker layers, downloading python and npm packages and installing it.Next post will probably be about reducing this time by using some local caching services.
Project pipelines¶
A pipeline is a group of jobs executed by GitLab CI.
Reference¶
v4 API:
gitlab.v4.objects.Project.pipelines
GitLab API: https://docs.gitlab.com/ce/api/pipelines.html
Examples¶
List pipelines for a project:
Get a pipeline for a project:
Get variables of a pipeline:
Create a pipeline for a particular reference with custom variables:
Retry the failed builds for a pipeline:
Cancel builds in a pipeline:
Delete a pipeline:
Triggers¶
Triggers provide a way to interact with the GitLab CI. Using a trigger a useror an application can run a new build/job for a specific commit.
Reference¶
v4 API:
gitlab.v4.objects.Project.triggers
GitLab API: https://docs.gitlab.com/ce/api/pipeline_triggers.html
Examples¶
List triggers:
Get a trigger:
Create a trigger:
Remove a trigger:
Full example with wait for finish:
You can trigger a pipeline using token authentication instead of userauthentication. To do so create an anonymous Gitlab instance and use lazyobjects to get the associated project:
Reference: https://docs.gitlab.com/ee/ci/triggers/#trigger-token
Pipeline schedule¶
You can schedule pipeline runs using a cron-like syntax. Variables can beassociated with the scheduled pipelines.
Reference¶
v4 API
gitlab.v4.objects.Project.pipelineschedules
gitlab.v4.objects.Project.pipelineschedules
GitLab API: https://docs.gitlab.com/ce/api/pipeline_schedules.html
Examples¶
List pipeline schedules:
Get a single schedule:
Create a new schedule:
Update a schedule:
Trigger a pipeline schedule immediately:
Delete a schedule:
List schedule variables:
Create a schedule variable:
Edit a schedule variable:
Delete a schedule variable:
Jobs¶
Jobs are associated to projects, pipelines and commits. They provideinformation on the jobs that have been run, and methods to manipulatethem.
Reference¶
v4 API
gitlab.v4.objects.Project.jobs
GitLab API: https://docs.gitlab.com/ce/api/jobs.html
Examples¶
Jobs are usually automatically triggered, but you can explicitly trigger a newjob:
List jobs for the project:
Get a single job:
List the jobs of a pipeline:
Note
Job methods (play, cancel, and so on) are not available onProjectPipelineJob
objects. To use these methods create a ProjectJob
object:
Get the artifacts of a job:
Get the artifacts of a job by its name from the latest successful pipeline ofa branch or tag:
project.artifacts(ref_name=’master’, job=’build’)
Warning
Artifacts are entirely stored in memory in this example.
You can download artifacts as a stream. Provide a callable to handle thestream:
You can also directly stream the output into a file, and unzip it afterwards:
Get a single artifact file:
Get a single artifact file by branch and job:
Mark a job artifact as kept when expiration is set:
Delete the artifacts of a job:
Get a job trace:
Warning
Traces are entirely stored in memory unless you use the streaming feature.See the artifacts example.
Cancel/retry a job:
Gitlab Ci Cd Pipeline Example
Play (trigger) a job:
Erase a job (artifacts and trace): Incredimail 2 5 plus crack.
Pipeline bridges¶
Get a list of bridge jobs (including child pipelines) for a pipeline.
Reference¶
How Make A If Statement In The CI File
v4 API
gitlab.v4.objects.ProjectPipeline.bridges
GitLab API: https://docs.gitlab.com/ee/api/jobs.html#list-pipeline-bridges
Examples¶
List bridges for the pipeline:
Pipeline test report¶
Get a pipeline’s complete test report.
Reference¶
Before_script Run Shell Script - GitLab CI/CD
v4 API
gitlab.v4.objects.ProjectPipeline.test_report
GitLab API: https://docs.gitlab.com/ee/api/pipelines.html#get-a-pipelines-test-report
Before_script Not Run When Includes Are Used
Examples¶
Forum.gitlab.com › T › Execute-script-python-usingExecute Script Python Using Curl - GitLab CI/CD - GitLab Forum
Get the test report for a pipeline: