Real-time Email Processing using Python and IMAP

1. Introduction

For years, developers have relied on traditional username and password authentication to access email accounts. While this method has been widely adopted, it presents inherent security vulnerabilities.

To mitigate these risks, many email providers have transitioned to OAuth 2.0, a more secure and modern authentication protocol. OAuth 2.0 eliminates the need for storing or transmitting passwords directly, instead utilizing tokens that grant specific and limited access to email resources. These tokens can be revoked at any time without requiring a password change, providing an additional layer of protection.

In this article, we will demonstrate how to utilize the IMAPClient library with OAuth 2.0 to securely and efficiently access your email account, enhancing the overall security posture of your applications.

Gmail API and Generating OAuth2 Credentials

1. Create a Google Cloud Platform project.

2. Enable the Gmail API.

3. Create OAuth 2.0 Client ID.

4. Generate OAuth 2.0 Client Secret.

5. Determine Required Scopes: https://mail.google.com/ for reading emails.

6. Configure Your Application: use the client ID and client secret to obtain an OAuth 2.0 access token.

For more information, refer to the Gmail API documentation: https://developers.google.com/gmail/api/auth/scopes

1. Introduction

For years, developers have relied on traditional username and password authentication to access email accounts. While this method has been widely adopted, it presents inherent security vulnerabilities.

To mitigate these risks, many email providers have transitioned to OAuth 2.0, a more secure and modern authentication protocol. OAuth 2.0 eliminates the need for storing or transmitting passwords directly, instead utilizing tokens that grant specific and limited access to email resources. These tokens can be revoked at any time without requiring a password change, providing an additional layer of protection.

In this article, we will demonstrate how to utilize the IMAPClient library with OAuth 2.0 to securely and efficiently access your email account, enhancing the overall security posture of your applications.

Gmail API and Generating OAuth2 Credentials

1. Create a Google Cloud Platform project.

2. Enable the Gmail API.

3. Create OAuth 2.0 Client ID.

4. Generate OAuth 2.0 Client Secret.

5. Determine Required Scopes: https://mail.google.com/ for reading emails.

6. Configure Your Application: use the client ID and client secret to obtain an OAuth 2.0 access token.

For more information, refer to the Gmail API documentation: https://developers.google.com/gmail/api/auth/scopes

Understanding IMAP and its Role in Email Processing

The Internet Message Access Protocol (IMAP) offers a robust and versatile approach to managing email messages. Unlike POP3, IMAP enables clients to retrieve and manipulate emails while leaving them on the server, facilitating access from multiple devices and real-time monitoring of incoming messages.

• Using IMAPClient: This library simplifies OAuth2 login compared to imaplib.
• OAuth2 Login: The code authenticates securely using the access token.
• IDLE Mode: This allows for efficient monitoring of new emails without constant polling.

Setting up the Environment

Before getting started, ensure you have the following:

• Python (version 3.6 or higher recommended)
• IMAPClient library for interacting with the IMAP server
• email library for processing the content of the emails

You can install IMAPClient using pip: pip install IMAPClient

Connecting to the Email Server

To establish a secure connection to Gmail’s IMAP server using OAuth 2.0 authentication, follow these steps:

from imapclient import IMAPClient
import email

HOST = 'imap.gmail.com'
USERNAME = '[email protected]'
access_token = 'your-oauth2-access-token'

server = IMAPClient(HOST, 993, use_uid=True, ssl=True)
server.oauth2_login(USERNAME, access_token, mech="XOAUTH2")
server.select_folder('Inbox', readonly=True)

Fetching Emails in Real-time

To enable real-time email processing, we leverage the IMAP IDLE command. This command instructs the server to notify the client of any new messages, eliminating the need for constant polling. The following code illustrates how to set the server in IDLE mode:

server.idle()
print("Connection is now in IDLE mode, send yourself an email or quit with ^C")

The connection enters an idle state, where it waits for incoming emails. Once an email arrives, the server sends a notification, and the script processes the email.

Processing Emails and Attachments

The function process(mail) extracts the text from the email and downloads any attachments that match a specified pattern (e.g., a PDF attachment with a specific name). Here’s the code for processing:

def process(mail) -> None:
    if mail:
        # Get text content
        for part in mail.walk():
            if part.get_content_maintype() == 'text':
                print(part.get_payload(decode=True).decode())  # Decode text content
        
        # Download attachments with specific filename pattern
        for part in mail.walk():
            if part.get_content_maintype() == 'multipart' and part.get_filename():
                if "attachment_name.pdf" in part.get_filename():  # Replace with your pattern
                    with open(f"attachments/{part.get_filename()}", 'wb') as fp:
                        fp.write(part.get_payload(decode=True))
                        print(f"Attachment saved: {part.get_filename()}")
    return None

Continuous Monitoring and Real-time Processing

In the loop below, the server checks for new messages in the inbox. If an unread email arrives, the script fetches the email content, processes it, and downloads any relevant attachments.

while True:
    try:
        # Wait for up to 30 seconds for an IDLE response
        responses = server.idle_check(timeout=30)
        print("Server sent:", responses if responses else "nothing")
        
        if responses:
            print()
            server.idle_done()
            messages = server.search(['UNSEEN'])
            latest_uid = [(max(messages))]
            for uid, message_data in server.fetch(latest_uid, 'RFC822').items():
                email_message = email.message_from_bytes(message_data[b'RFC822'])
                # if the subject meets the condition
                if email_message.get("Subject") == 'TEST':
                    process(email_message)
                    
            print()
            # start IDLE mode again 
            server.idle()
            
    except KeyboardInterrupt:
        break

This script checks for new emails every 30 seconds. If a new email arrives and its subject matches a specific condition (e.g., subject = “TEST”), the email content is processed. The loop runs continuously, waiting for new messages.

from imapclient import IMAPClient
import email

HOST = 'imap.gmail.com'
USERNAME = '[email protected]'
access_token = 'your-oauth2-access-token'

server = IMAPClient(HOST, 993, use_uid=True, ssl=True)
server.oauth2_login(USERNAME, access_token, mech="XOAUTH2")
server.select_folder('Inbox', readonly=True)

# Start IDLE mode
server.idle()
print("Connection is now in IDLE mode, send yourself an email or quit with ^c")

def process(mail) -> None:
    if mail:
        # Get text content
        for part in mail.walk():
            if part.get_content_maintype() == 'text':
                print(part.get_payload(decode=True).decode())  # Decode text content
        
        # Download attachments with specific filename pattern
        for part in mail.walk():
            if part.get_content_maintype() == 'multipart' and part.get_filename():
                if "attachment_name.pdf" in part.get_filename():  # Replace with your pattern
                    with open(f"attachments/{part.get_filename()}", 'wb') as fp:
                        fp.write(part.get_payload(decode=True))
                        print(f"Attachment saved: {part.get_filename()}")
    return None

while True:
    try:
        # Wait for up to 30 seconds for an IDLE response
        responses = server.idle_check(timeout=30)
        print("Server sent:", responses if responses else "nothing")
        
        if responses:
            print()
            server.idle_done()
            messages = server.search(['UNSEEN'])
            latest_uid = [(max(messages))]
            for uid, message_data in server.fetch(latest_uid, 'RFC822').items():
                email_message = email.message_from_bytes(message_data[b'RFC822'])
                # if the subject meet the condition
                if email_message.get("Subject") == 'TEST':
                    process(email_message)
                    
            print()
            # start IDLE mode again 
            server.idle()
            
    except KeyboardInterrupt:
        break

server.idle_done()
print("\nIDLE mode done")
server.logout()

Leveraging Python and IMAP, you can develop a robust tool for real-time email monitoring and processing. This solution enables you to automatically extract email content, download attachments, and trigger actions based on specific email criteria. This can be invaluable for business automation, notification systems, or data collection.