PyAutoGUI: Automation with Python

This article is part of the Python Libraries Series. Find more below!

PyAutoGUI is a Python library that allows you to automate tasks by controlling your mouse and keyboard. This way, you can create, bots, macros, etc…

Let me introduce you to this library.

Getting Started with PyAutoGUI

First, you need to install PyAutoGUI. It can be done easily with pip :

pip install pyautogui

Once it is installed, you can import PyAutoGUI into your Python scripts using:

import pyautogui

Controlling the Mouse

The main thing you do when it comes to controlling GUIs is move your mouse. With PyAutoGUI, you can do it with the following code:

pyautogui.moveTo(x, y)

In addition to moving the cursor, PyAutoGUI allows you to simulate mouse button clicks. The click() function enables you to perform a primary (left) click at the current cursor position. To perform a secondary (right) click, you can use the rightClick() function. These functions can be combined with moveTo() to click at specific coordinates on the screen. For example:

import pyautogui

pyautogui.moveTo(500, 500)
pyautogui.click()

If the mouse movement should take some time, you can specify its duration (in seconds):

pyautogui.moveTo(200, 300, duration=1)

PyAutoGUI also provides functions to perform other types of clicks, such as double-clicking, middle-clicking, and dragging. The doubleClick() function allows you to simulate a double-click at the current cursor position, while the middleClick() function performs a middle click. To drag the mouse, you can use the dragTo() function, specifying the target coordinates after dragging from the current position.

Besides mouse movements and clicks, PyAutoGUI offers additional functionalities for mouse-related automation. You can retrieve the current mouse position using the position() function, which returns a tuple containing the x and y coordinates. The scroll() function allows you to simulate mouse scrolling by specifying the number of units to scroll vertically and horizontally.

You can also use moveRel and dragRel if you want to perform actions depending on the current position of the mouse without having to first get the current position.

    mouse_position = pyautogui.position()
    print(mouse_position)
    
    pyautogui.dragTo(mouse_position[0] + 100, mouse_position[1], duration=1)
    pyautogui.moveRel(0, 100, duration=1)
    pyautogui.dragRel(100, 0, duration=1)

Controlling the Keyboard

The main action you can do with a keyboard is typing text. PyAutoGUI allows you to simulate typing using the typewrite() function:

pyautogui.typewrite("Hello")

If you want to introduce a delay between each keypress, you can pass an optional interval argument to the typewrite() function. The interval specifies the number of seconds to wait after typing each character.

pyautogui.typewrite("Hello", interval=0.1)

In addition to typing text, PyAutoGUI allows you to simulate the pressing and releasing of specific keys using the press() and release() functions, respectively. These functions accept key names or keycodes as input. For example, to simulate pressing the Enter key, you can use the following code:

pyautogui.press("enter")

And to release it:

pyautogui.release("enter")

PyAutoGUI also provides a handy keyDown() and keyUp() functions to hold a key down and release it, respectively. These functions are useful when simulating keyboard shortcuts.

pyautogui.keyDown("ctrl")
pyautogui.press("s")
pyautogui.keyUp("ctrl")

# you just saved a document!

Displaying Messages

If you want to provide information or feedback during the automation process, you can use some PyAutoGUI functions.

To display messages with PyAutoGUI, you can use the alert() function. This function creates a pop-up dialog box with a message and an OK button for the user to acknowledge.

pyautogui.alert("Hello world!")

The alert() function halts the execution of the program until the user clicks the OK button in the dialog box.

In addition to the alert() function, PyAutoGUI also provides the confirm() function. This function displays a dialog box with a message and two buttons: OK and Cancel. The user can choose either option, and the function returns a boolean value indicating the user's selection.

result = pyautogui.confirm("Do you want to proceed?")

if result:
    print("Proceeding with the task...")
else:
    print("Task aborted.")

Finally, you can let the user type a string using prompt() :

beverage = pyautogui.prompt("Tea or coffee?")

Working with Images

If your automation needs are a little more complex, you’ll certainly need to identify images on screen, or take screenshots. PyAutoGUI lets you do just that.

First, you have the locateOnScreen() function. This function takes an image file path as input and searches for the presence of that image on the screen. If the image is found, the function returns the coordinates of the first occurrence of the image.

image_location = pyautogui.locateOnScreen("image.png")

If you want to locate several images that are the same, you just have to use locateAllOnScreen instead. This function will return a generator for all the locations the image is found on the screen.

Alternatively, you can use locateCenterOnScreen to get the x and y coordinates of the middle of where the image is found on the screen.

pyautogui.locateCenterOnScreen('image.png')

If the image is not found, these functions return None .

Finally, you can just take a screenshot using screenshot() . It returns a Pillow/PIL Image object. If you provide this function a path, it will save the screenshot:

pyautogui.screenshot("src/images/screenshot.png")

Example: Cookie Clicker Automation

A perfect use case for this library is a cookie-clicker game. The only actions you need to do are clicking and buying upgrades. Let’s translate this into PyAutoGUI:

def cookie_clicker_automation():
    cookie_location = pyautogui.locateCenterOnScreen('cookie.png')
    
    for i in range(10):
        pyautogui.moveTo(cookie_location)
    
        for j in range(100):
            pyautogui.click()
            time.sleep(0.05)
    
        upgrade_location = pyautogui.locateCenterOnScreen('upgrade.png')
        pyautogui.moveTo(upgrade_location)
        pyautogui.click()

Final Note

When it comes to automation, there are many possible solutions. The advantage of using Python is that you have access to the whole ecosystem of libraries on the side to build something really complex, so I can only recommend PyAutoGUI!

If you want to discover other Python libraries, click below!

If you liked the story, don’t forget to clap and maybe follow me if you want to explore more of my content :)

You can also subscribe to me via email to be notified every time I publish a new story, just click here!

If you’re not subscribed to Medium yet and wish to support me or get access to all my stories, you can use my link: