How to Use Google Pub/Sub with Python

Introduction:

Google Pub/Sub, an integral part of Google Cloud, is pivotal in modern application architecture, particularly for distributed systems. It excels in real-time messaging and event-driven functionalities, enhancing application responsiveness. This guide dives deep into the utilization of Google Pub/Sub with Python, a language known for its versatility and simplicity, making it ideal for interacting with Pub/Sub.

Prerequisites

• Python Environment: Make sure Python is installed on your computer.
• A Google Cloud Account: Necessary to access Google Cloud services. Create an account if you don't have one.
• Google Cloud Project: Create a project in the Google Cloud Console and enable the Pub/Sub API.
• Service Account and Permissions: Ensure you have a service account with pubsub.editor role.
• Authentication Setup: Set up authentication via a service account key file.

Step 1: Set Up Your Google Cloud Project

Begin by establishing a new project in the Google Cloud Console, ensuring to enable the Pub/Sub API for messaging services. This setup is critical for leveraging the robust capabilities of Google Cloud’s messaging infrastructure. As an alternative, particularly for those preferring command-line interfaces, the gcloud CLI offers a streamlined method for project creation and configuration. Utilizing the gcloud CLI commands not only simplifies the process but also provides a flexible and scriptable approach to managing your Google Cloud resources. This method is particularly beneficial for developers and system administrators seeking efficiency and automation in their cloud-based operations.

gcloud projects create my-python-project
gcloud config set project my-python-project
gcloud services enable pubsub.googleapis.com

Step 2: Install Google Cloud Pub/Sub Client Library

Install the Python client library for Google Cloud Pub/Sub:

pip install google-cloud-pubsub

Step 3: Configure Authentication

To configure authentication, start by creating a service account in your Google Cloud project. This account will be used to grant your application the necessary permissions to interact with Google Cloud services. Once the account is created, generate a key file for it. This key file contains the credentials that your application will use to authenticate with Google Cloud. After obtaining the key file, set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable on your system to the file path of this key. This step is crucial as it allows your application to automatically use these credentials when accessing Google Cloud services, ensuring a secure and authenticated connection.

gcloud iam service-accounts create my-service-account
gcloud projects add-iam-policy-binding my-python-project --member "serviceAccount:[email protected]" --role "roles/pubsub.editor"
gcloud iam service-accounts keys create ~/keyfile.json --iam-account [email protected]
export GOOGLE_APPLICATION_CREDENTIALS=~/keyfile.json

Step 4: Create a Topic and Subscription

Utilize the gcloud CLI to efficiently create both a topic and a subscription, streamlining your setup process in Google Cloud Pub/Sub. This approach allows for quick and easy configuration of your messaging components, enhancing the communication framework within your cloud infrastructure.

gcloud pubsub topics create my-topic
gcloud pubsub subscriptions create my-subscription --topic my-topic

Step 5: Write Python Code for Pub/Sub

Publishing a Message

To publish messages, use the following Python code:

from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient()
topic_name = 'projects/my-python-project/topics/my-topic'

def publish_message(data):
    data = data.encode("utf-8")
    future = publisher.publish(topic_name, data)
    print(f"Published message ID: {future.result()}")

publish_message("Hello, world!")

This Python code utilizes the `google.cloud.pubsub_v1` module to publish messages to a Google Cloud Pub/Sub topic. It creates a `PublisherClient`, specifies a topic, and defines a `publish_message` function to send data. The message “Hello, world!” is encoded and published, with its ID printed on completion.

Testing Publishing Messages

Create a test file, say test_pubsub_publish.py, and add the following code:

import unittest
from unittest.mock import patch
from your_pubsub_module import publish_message

class TestPublishMessage(unittest.TestCase):
    @patch('google.cloud.pubsub_v1.PublisherClient')
    def test_publish_message(self, mock_pubsub):
        # Mock the publish method
        mock_pubsub.return_value.publish.return_value = 'mocked-message-id'

        # Call the function
        result = publish_message('Test message')

        # Assertions
        self.assertEqual(result, 'mocked-message-id')

if __name__ == '__main__':
    unittest.main()

This test checks if the publish_message function is correctly calling the Pub/Sub publish method and returns the expected result.

Receiving Messages

To receive messages, use the following Python code:

from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
subscription_path = 'projects/my-python-project/subscriptions/my-subscription'

def callback(message):
    print(f"Received message: {message.data}")
    message.ack()

streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
print("Listening for messages...")
streaming_pull_future.result()

This Python script demonstrates how to receive messages from a Google Cloud Pub/Sub subscription using the `google.cloud.pubsub_v1` library. It begins by initializing a `SubscriberClient`, then specifies the path to a subscription. A function named `callback` is defined to handle incoming messages: it prints the message data and then acknowledges the message with `message.ack()`. This acknowledgment is crucial for informing Pub/Sub that the message has been successfully processed. The script then subscribes to the specified subscription path, passing the `callback` function as the handler for incoming messages. It prints a message indicating it’s listening and then enters a waiting state to continuously receive and process messages as they arrive.

Testing Receiving Messages

Create another test file, say test_pubsub_subscribe.py, and add the following code:

import unittest
from unittest.mock import patch, MagicMock
from your_pubsub_module import subscribe_to_messages

class TestSubscribeToMessages(unittest.TestCase):
    @patch('google.cloud.pubsub_v1.SubscriberClient')
    def test_subscribe_to_messages(self, mock_pubsub):
        # Setup a mock subscriber
        mock_subscriber = MagicMock()
        mock_pubsub.return_value = mock_subscriber

        # Call the function
        subscribe_to_messages()

        # Assertions
        mock_subscriber.subscribe.assert_called()

if __name__ == '__main__':
    unittest.main()

This test verifies that the subscribe_to_messages function correctly sets up a subscription and starts listening for messages.

Conclusion

Google Pub/Sub with Python is a powerful combination for building applications that require message exchange between components. By following the steps in this guide, you can set up Google Pub/Sub in a Python environment and start publishing and subscribing to messages.

For more comprehensive examples and advanced features, check out the Google Cloud Pub/Sub Python Client documentation.