Free AI web copilot to create summaries, insights and extended knowledge, download it at here

3089

Abstract

to access your resources with a Google client library. For example, let’s list all the Google Storage Buckets in Python:</p> <figure id="daa6"> <div> <div>

            <iframe class="gist-iframe" src="/gist/lynnkwong/6b34ff97750212b6cf02ad6f33af719a.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
          </div>
        </div>
    </figure></iframe></div></div></figure><h2 id="a9a2">Authenticate Docker</h2><p id="9011">Up to now, if you try to pull some images from Google Artifact/Container Registry, you will have permission issues because you need to authenticate Docker explicitly, both the default authentication and default application authentication won’t work.</p><p id="d36a">Before we get started, it’s very important to know that Docker installed with Snap on Linux may not work properly with the command shown below. It’s better to install Docker from the official site, rather than with Snap or even the “handy” <code>get_docker</code> script.</p><p id="2470">The command to authenticate Docker is:</p><div id="bf4f"><pre><span class="hljs-variable">$ </span>gcloud auth configure-docker eu.gcr.io</pre></div><p id="2568">Note it’s better to specify the regional registry as well, otherwise, <code>docker build</code> can be slow because you need to deal with all the available registries even if they are not used.</p><p id="59fd">Now, you will be able to pull from/push to Artifact/Container registry if you have been granted the permission to do so.</p><h2 id="e956">Work with service accounts</h2><p id="f8a3">If you need to check or change some resources manually, you should use your personal Google account which is added to your own or your group’s GCP project. However, if you need to do something programmatically in your code, which normally means working with Google APIs like storage, pub/sub, logging, etc, it’s better to use a service account that normally has limited and dedicated permissions/roles.</p><p id="1720">If your application is hosted on a GCP resource such as Compute Engine, App Engine, Cloud Run, etc. You can attach service accounts with specific roles to your target services and don’t need to worry about anything else.</p><p id="6eb6">However, when you develop your code locally, it’s not so straightforward to use service accounts. If your personal Google account has enough permissions to perform programmatic actions, you can run <code>gcloud auth application-default login</code> as shown above to authenticate your applications and can then start to work with Google APIs directly in your code.</p><p id="26bf">On the other hand, if your personal Google account has limited permissions and you must use a service account, you can download (or ask for…) the JSON key for the service account and use it to authenticate Google client libraries in your application. Although it’s recommended to use “Workload Identity Federation” to authenticate your applications if they are not hosted on the GCP infrastructure, it’s still quite common

Options

to use JSON keys as they are much more convenient. However, you must be extremely careful about your JSON keys and not expose them accidentally. Most importantly, never add them to your public repository. This is because anyone with these JSON keys can access and manipulate your GCP resources directly!</p><p id="1daf">A practical use case of the JSON key of a service account is for some third-party CI/CD pipelines like GitLab, where you need to build a Docker image and push it to GCP Artifact/Container Registry, which will be introduced in a later post.</p><p id="a7be">To actually authenticate your applications with a service account, you need to run the following command to set the path to the JSON key to the pre-defined environment variable <code>GOOGLE_APPLICATION_CREDENTIALS</code>:</p><div id="31bb"><pre>$ <span class="hljs-built_in">export</span> <span class="hljs-attribute">GOOGLE_APPLICATION_CREDENTIALS</span>=path/to/JSON_key.json</pre></div><p id="9b4c">Note the environment variable must be named exactly as it is shown here. It will be used by Google client libraries automatically. Besides, it should be noted that this environment variable has higher precedence than the credentials set by <code>gcloud auth application-default login</code>, which means you can have default application authentications with your own Google account and use different service accounts for different applications when needed.</p><p id="a275">With this environment variable set, you don’t need to deal with service accounts explicitly in your code anymore.</p><p id="b3bc">In this post, we introduced some common settings that are important to set up your local environment to work with GCP. If your developer’s Google account has enough permissions, which normally is the case, then the set-up should be fairly straightforward and you don’t need to bother with the service account. Nonetheless, when a service account is really needed, you can just download its JSON key and set the local path to the environment variable <code>GOOGLE_APPLICATION_CREDENTIALS</code> , and then everything will work as expected.</p><p id="cdba">Related articles:</p><ul><li><a href="https://lynn-kwong.medium.com/how-to-use-gsutil-and-python-to-deal-with-files-in-google-cloud-storage-fc4f430b3b28">How to use gsutil and Python to deal with files in Google Cloud Storage</a></li><li><a href="https://levelup.gitconnected.com/how-to-write-logs-to-google-cloud-logging-in-python-46e7b514c60b">How to Write Logs to Google Cloud Logging in Python</a></li></ul><h1 id="c4f3">Level Up Coding</h1><p id="91d5">Thanks for being a part of our community! More content in the <a href="https://levelup.gitconnected.com/">Level Up Coding publication</a>. Follow: <a href="https://twitter.com/gitconnected">Twitter</a>, <a href="https://www.linkedin.com/company/gitconnected">LinkedIn</a>, <a href="https://newsletter.levelup.dev/">Newsletter</a> <b>Level Up is transforming tech recruiting ➡️ <a href="https://jobs.levelup.dev/talent/welcome?referral=true">Join our talent collective</a></b></p></article></body>

How to Set Up Your Local Environment to Work With GCP

Learn to deal with GCP local authentications in minutes

If you work on the Google Cloud Platform (GCP), having your local environment set up properly can make your life much easier. It can be a headache to deal with all the nitty-gritty authentication issues especially when you have just started to work with GCP. Don’t worry, in this post, we will introduce the common settings for gcloud CLI, Google client libraries, and Docker, which are essential for your daily developing work. We will also introduce how to work with service accounts locally.

Install Google Cloud SDK

Please install the Google Cloud SDK according to the official documentation, which provides step-by-step instructions for different operating systems.

Authenticate Google Cloud SDK

If it’s the first time you use Google Cloud SDK, you need to initialize the gcloud CLI:

$ gcloud init

Follow the instructions on the console to create a new configuration. If you have already created a configuration for an account, but need to log in as another one, you can run the following command to log in as the other account:

$ gcloud auth login

The browser will be opened for both commands and you will be asked to log in to your Google account which is associated with a GCP project.

After you are logged in, you can check your authentication with this command:

$ gcloud auth list

Now you should be able to check the resources on GCP. For example, you can check the Google Storage Buckets with the gsutil command:

$ gsutil ls

Authenticate Google client libraries

The above authentication methods only authenticate the gcloud CLI and not Google client libraries. We need to authenticate applications that need to interact with Google APIs as well. To do this, we need to run this command:

$ gcloud auth application-default login

The browser will be opened again and you need to choose the Google account to be authenticated. Note that, this command works independently with the gcloud auth or gcloud auth login commands above. You always need to run this command explicitly after you have authenticated gcloud CLI.

After the applications are authenticated with the above command, you should be able to access your resources with a Google client library. For example, let’s list all the Google Storage Buckets in Python:

Authenticate Docker

Up to now, if you try to pull some images from Google Artifact/Container Registry, you will have permission issues because you need to authenticate Docker explicitly, both the default authentication and default application authentication won’t work.

Before we get started, it’s very important to know that Docker installed with Snap on Linux may not work properly with the command shown below. It’s better to install Docker from the official site, rather than with Snap or even the “handy” get_docker script.

The command to authenticate Docker is:

$ gcloud auth configure-docker eu.gcr.io

Note it’s better to specify the regional registry as well, otherwise, docker build can be slow because you need to deal with all the available registries even if they are not used.

Now, you will be able to pull from/push to Artifact/Container registry if you have been granted the permission to do so.

Work with service accounts

If you need to check or change some resources manually, you should use your personal Google account which is added to your own or your group’s GCP project. However, if you need to do something programmatically in your code, which normally means working with Google APIs like storage, pub/sub, logging, etc, it’s better to use a service account that normally has limited and dedicated permissions/roles.

If your application is hosted on a GCP resource such as Compute Engine, App Engine, Cloud Run, etc. You can attach service accounts with specific roles to your target services and don’t need to worry about anything else.

However, when you develop your code locally, it’s not so straightforward to use service accounts. If your personal Google account has enough permissions to perform programmatic actions, you can run gcloud auth application-default login as shown above to authenticate your applications and can then start to work with Google APIs directly in your code.

On the other hand, if your personal Google account has limited permissions and you must use a service account, you can download (or ask for…) the JSON key for the service account and use it to authenticate Google client libraries in your application. Although it’s recommended to use “Workload Identity Federation” to authenticate your applications if they are not hosted on the GCP infrastructure, it’s still quite common to use JSON keys as they are much more convenient. However, you must be extremely careful about your JSON keys and not expose them accidentally. Most importantly, never add them to your public repository. This is because anyone with these JSON keys can access and manipulate your GCP resources directly!

A practical use case of the JSON key of a service account is for some third-party CI/CD pipelines like GitLab, where you need to build a Docker image and push it to GCP Artifact/Container Registry, which will be introduced in a later post.

To actually authenticate your applications with a service account, you need to run the following command to set the path to the JSON key to the pre-defined environment variable GOOGLE_APPLICATION_CREDENTIALS:

$ export GOOGLE_APPLICATION_CREDENTIALS=path/to/JSON_key.json

Note the environment variable must be named exactly as it is shown here. It will be used by Google client libraries automatically. Besides, it should be noted that this environment variable has higher precedence than the credentials set by gcloud auth application-default login, which means you can have default application authentications with your own Google account and use different service accounts for different applications when needed.

With this environment variable set, you don’t need to deal with service accounts explicitly in your code anymore.

In this post, we introduced some common settings that are important to set up your local environment to work with GCP. If your developer’s Google account has enough permissions, which normally is the case, then the set-up should be fairly straightforward and you don’t need to bother with the service account. Nonetheless, when a service account is really needed, you can just download its JSON key and set the local path to the environment variable GOOGLE_APPLICATION_CREDENTIALS , and then everything will work as expected.

Level Up Coding

Thanks for being a part of our community! More content in the Level Up Coding publication. Follow: Twitter, LinkedIn, Newsletter Level Up is transforming tech recruiting ➡️ Join our talent collective