Three Ways to Run Scripts (and Functions) in Your Gitlab CI File

Automate your script files — or just the functions

GitLab CI is a very useful dev-ops tools to automate many processes, such as building an app and publishing it (such as publishing to NPM or pub.dev), as well as updating an associated public repo (Like one in Github).

You can either write out these processes in the specific .gitlab-ci.yml file associated with your project, or you can write out separate scripts that can be referenced in your file.

Before we get into that, what is a .gitlab-ci.yml file?

The GitLab CI File

The .gitlab-ci.yml file defines scripts that should be run during the CI/CD pipeline and their scheduling, additional configuration files and templates, dependencies, caches, commands GitLab should run sequentially or in parallel, and instructions on where the application should be deployed to. — codefresh

Now, a pipeline is a process by which software development is pushed through building, testing and deployment. CI/CD stands for Continuous Integration and Continuous Deployment.

These pipelines allow the integration and deployment processes to be continuous without the developer having to repeat all of the steps involved. For example, at Socket Mobile, one of the SDKs that I work on is the Capture Flutter SDK.

Thanks to my .gitlab-ci.yml file, I don’t have to do the below items manually.

1. Build the app and it’s associated dependencies.
2. Test the build for iOS and Android.
3. Push SDK code to it’s respective Github repo.
4. Push the related sample app code to it’s respective Github repo.
5. Publish the SDK to pub.dev.

All of those objectives are defined in the various scripts and functions that get executed during the CI/CD pipeline. It’s quite convenient and makes testing, building and releasing consistent.

So how might this look in action? What does a gitlab-ci.yml file even look like? Where does it go?

Well, as for where it goes, it just goes in your project directory. By including a .gitlab-ci.yml file in your project, GitLab will execute whatever jobs you have outline in the file.

Now onto what .gitlab-ci.yml looks like and how to use it! I’ll show you three ways to run scripts — and/or functions — in your .gitlab-ci file.

1. The “In-File” Script

First, here is an example of a .gitlab-ci.yml file containing a job called my-first-job.

my-first-job:
  script:
    |
    echo "THIS JOB WAS TRIGGERED BY THE LATEST PUSH TO THE PROJECT REPO"

The goal of this job is simply to print out the line THIS JOB WAS TRIGGERED BY THE LATEST PUSH TO THE PROJECT REPO! when you push your project’s latest code to the repo.

This is an example of an “in-file” script because it’s written inline in the GitLab CI file. By declaring this job, and not including any constraints on it— such as branch restrictions and/or enforcing manual operation — this job will be run as soon as you push your latest code.

Then, in your GitLab repo, you can see a little indicator along with your most recent commit. This indicator will correspond to a pipeline, which will contain your job!

The indicator will be something like a green checkmark (pass), or a red X (fail). There are other indicators, such as a grey gear icon (manually invoked) and the an exclamation point (fail, but allowed to fail).

Bottom line, these indicators will show you whether or not your “in-file” script was executed or not. While we can run scripts this way, it can be a bit difficult to read — especially if you have a few functions to run. The second way to use scripts and/or functions ensures our .gitlab-ci.yml doesn’t get too overwhelmed.

2. The “Other-File” Script

Like other areas of programming, we can define a script and functions in another file — keeping all the logic in a separate file from our GitLab CI file.

Then, once in this file, we can “call” it from our .gitlab-ci.yml file. Let’s try to do the same thing as we previously did, but this time without having to write the bash script in the actual CI/CD file.

We can do that by creating a new file, called my_job_scripts.sh. In that file, we can move the script code from the CI/CD file into this one.

See the new my_job_scripts.sh file below.

#!/usr/bin/env bash
echo "THIS JOB WAS TRIGGERED BY THE LATEST PUSH TO THE PROJECT REPO"

Now, in our .gitlab-ci.yml, where it says script we can add the following.

my-first-job:
  script:
    - my_job_scripts.sh

This will essentially run our my_job_scripts.sh file as the script for my-first-job. Which, again, will be executed when you push your latest commit to your GitLab repo.

We can include an added layer of security by ensuring our new file is executable within the context of the job. We can do this by simply adding chmod +x my_job_scripts.sh before we have our script’s line.

See below.

my-first-job:
  script:
    - chmod +x my_job_scripts.sh 
    - my_job_scripts.sh

With this added layer of protection, we can avoid any unforseen errors where our file is unable to be executed. Nice.

But let’s take our script even further.

Let’s say we want to do two things in our my_job_scripts file. We want to echo the first line, but then also want to echo a second line. Maybe something like THIS JOB IS FUN.

How would we write that? Check it out below.

#!/usr/bin/env bash
echo "THIS JOB WAS TRIGGERED BY THE LATEST PUSH TO THE PROJECT REPO"
echo "THIS JOB IS FUN"

Perfect! Now the job will log both of those lines.

But what if, instead of running them together, you wanted to run them individually. What we can do is wrap them in their own functions. This way we can call whichever function we want, whenever we want, in our job.

#!/usr/bin/env bash
my_fun_func()
{
   echo "THIS JOB IS FUN"
}

my_latest_func()
{
   echo "THIS JOB WAS TRIGGERED BY THE LATEST PUSH TO THE PROJECT REPO"
}

The only problem now is that if we just run the file, nothing will happen. This is because we need to invoke the functions directly. Which we can do like so.

#!/usr/bin/env bash
my_fun_func()
{
   echo "THIS JOB IS FUN"
}

my_latest_func()
{
   echo "THIS JOB WAS TRIGGERED BY THE LATEST PUSH TO THE PROJECT REPO"
}

my_fun_func
my_latest_func

Now both functions will get executed when my-first-job runs.

But what if we only wanted to run one function in one job? What if we wanted to then save another function for another job.

Say, my-second-job.

my-first-job:
  script:
    - SOME LOGIC

my-second-job: 
  script: 
    - SOME MORE LOGIC

How can we execute only one function for each job? Well, we could write two separate bash files to correspond to each job. But what if we wanted to call the functions without running the whole file?

What we can do is we can source the file before running our script(s).

Once we do that, we can call whatever functions we want from it. You can think of this method as the “Other-file-AND-function” way of using scripts and/or functions in your GitLab CI/CD file.

3. The “Other-File-AND-Function” Script

In the code above, we want to run two separate functions, within two jobs, without having to declare two separate files (i.e. scripts). We can do this by first “sourcing” the file containing our functions before we even run our scripts. This can be done in the before_script section of our jobs.

my-first-job:
  before_script:
    - . my_job_scripts.sh
  script:
    - SOME LOGIC

my-second-job: 
  before_script:
    - . my_job_scripts.sh
  script:
    - SOME MORE LOGIC

The before_script section is the part of the job that sets things up and gets executed before your script (or the content in the script section) gets executed. In this case, we are sourcing the bash file before running any scripts.

We can even move our chmod +x my_job_scripts.sh in the before_script section as well.

By sourcing it, we can now reference whatever functions we want in it. See below.

my-first-job:
  before_script:
    - . my_job_scripts.sh
    - chmod +x my_job_scripts.sh
  script:
    - my_fun_fun
    - my_latest_func 

my-second-job: 
  before_script:
    - . my_job_scripts.sh
  script:
    - my_latest_func
    - my_fun_fun
    - my_fun_fun
    - my_latest_func

With the above syntax, we are able to achieve a few things. First, we remove code from our GitLab CI file, making it more readable and light. Second, we are able to reference code in another file from our GitLab CI file.

And finally, we are also able to call specific functions from this file as needed…from our GitLab CI file. The above logic seems a bit funny (why would we need to call my_fun_func and my_latest_func multiple times in one job?), so here is a more realistic scenario that could work.

I’ve also added a few other items to make it even more efficient.

default:
  before_script:
    - . my_job_scripts.sh
    - chmod +x my_job_scripts.sh

stages:
  - build
  - deploy

my-first-job:
  stage: build
  script:
    - build_sdk_github_code
    - build_sample_app_github_code
  artifacts: 
    - paths:
      - /tmp

my-second-job:
  stage: deploy
  script:
    - publish_sdk_code_to_github
    - publish_sample_app_code_to_github
  after_script: 
    - rm /tmp

Bonus Items

First, the new things I added.

• default: This is the section where you can set global defaults for some keywords. Each default keyword is copied to every job that doesn’t already have it defined. This way we don’t have to declare the before_script section in the beginning of each job.
• stages: This section is for defining “groups” of jobs. You can use stage in a job to configure it to run at a particular stage. For example, the job related to building code (my-first-job) will be executed in the build stage. This will be followed by the deploy stage (i.e. my-second-job).
• artifacts: In this section you can specify which files/directories to save as “artifacts”. These artifacts can saved in the then be used in other jobs — such as my-second-job. This is because artifacts are sent to GitLab and can be downloaded automatically by successive jobs.
• after_script: Like how before_script runs before our script section, we can declare an after_script section to execute code that we want to be run after our script is completed — such as removing a temporary directly used to store cloned and then updated code to be committed to respective GitHub repos.

As you can see, in my-first-job, I am referencing functions meant to “build” the code — i.e. prepare it for publication to GitHub. Part of this involves creating a temporary directory to store the code in (called /tmp).

Then in my-second-job we take care of actually publishing the generated code to their respective GitHub repos. Once that’s finished, we remove the /tmp folder as it is no longer needed.

The above is a more common scenario, where we have different stages of development. One stage can be seen as the build stage — containing jobs meant for building code. Another stage can be seen as the publish or deploy stage —containing jobs meant for publishing code.

By keeping of all our functions in one file and simply referencing them in our CI/CD file means we only have to make changes to one file — and to specific functions related to specific jobs. This makes our GitLab CI file readable, our code DRY and improves our single source of truth — while reducing the number of files we need to create.

Which style of writing scripts and/or jobs do you prefer? What other GitLab CI tricks do you like? Let me know in the comments!

References