Summary

Databricks workspaces are central hubs for organizing and managing resources within the Databricks platform, tailored to the size and structure of an organization, and can be created and managed through various methods.

Abstract

Databricks workspaces are essential components of the Databricks platform, serving as a unified interface for all assets, including clusters, SQL endpoints, and notebooks. They enable structured organization and collaboration, with a one-to-many relationship between users and workspaces. A Databricks account can contain multiple workspaces, which can be segmented to reflect different stages of the software development life cycle (SDLC), such as development, testing, and production. The number of workspaces a company should have depends on its size: small companies may only need one, medium-sized companies might benefit from three (Dev, Test, Prod), and large companies could require more to accommodate various departments. Workspace creation requires a Databricks account subscription and a cloud provider account, with support for AWS, Azure, and GCP. Workspaces can be created through the UI, APIs, Infrastructure as Code frameworks, or cloud provider templates. The article also notes the potential future evolution of Databricks to include serverless capabilities, which could simplify workspace setup.

Opinions

The author suggests that a single workspace is often sufficient for small companies (1–100 employees) due to the simplicity of collaboration and resource management within a single environment.
For medium-sized companies (100–1000 employees), the author recommends three workspaces to separate development, testing, and production environments, promoting an organized development process.
The author posits that large companies (over 1000 employees) may need multiple workspaces to serve different business verticals or departments, with an example given for a large banking company.
The author emphasizes the importance of considering workspace limits and constraints when determining the number of workspaces for an organization.
The author provides a pro tip: utilizing Unity Catalog at the account level allows schemas, tables, and views created in one workspace to be accessed from another within the same account, enhancing cross-workspace collaboration.
The author speculates that Databricks may introduce serverless capabilities in the future, potentially removing the need for a dedicated cloud account when creating a workspace, which would offer more flexibility and align with serverless computing advancements.

Databricks Workspaces - Explained

Databricks workspaces serve as a unified hub encompassing all features and assets(such as clusters, sql endpoints, catalogs, groups, notebooks, schemas, tables, views, repos etc) within the Databricks platform. To utilize Databricks, you must create at least one workspace. These workspaces offer a structured organization where specific users can be assigned to designated spaces. The relationship between users and workspaces follows a one-to-many model, allowing a user to access one or more workspaces. Think of a workspace as a logical environment group, providing a structured and isolated environment within a single Databricks subscription. This design ensures efficient collaboration and resource management across different tasks and user groups.

Databricks Account vs Workspace

A Databricks account represents a single entity that can include one or more workspaces. For instance, a company like ABC Inc., looking to leverage Databricks for its data and AI initiatives, would obtain a Databricks account subscription. With this account, the company can create numerous workspaces based on its requirements. Workspaces can be structured to mirror typical SDLC environments, such as development, testing, and production. This structuring ensures that developers, testers, and support personnel are part of the respective workspaces, effectively isolating different stages of the development life cycle.

Determining the Number of Workspaces

Determining the number of workspaces for a company involves assessing its size and organizational structure. Here’s a breakdown based on my experience to guide you in creating workspaces:

Note: This assumption is completely my perspective, not a standard

Small Companies (1–100 employees):

Recommendation: One Workspace
Explanation: For smaller companies, consolidating all activities into a single workspace is often efficient. This simplifies collaboration and resource management.

Medium Companies (100–1000 employees):

Recommendation: Three Workspaces (Dev, Test, Prod)
Explanation: Medium-sized companies benefit from isolating development, testing, and production environments. This promotes organized development processes.

Large Companies (>1000 employees):

Recommendation: More than Three Workspaces
Explanation: Larger enterprises may require additional workspaces to cater to various departments or business verticals. For instance, a large banking company could create workspaces for corporate banking, consumer banking, credit cards, loans, and insurances.

Note: The number of workspaces is entirely dependent on the company’s needs. It’s not uncommon for a single company to have over 70 workspaces. Additionally, consider workspace limits, as each workspace comes with specific constraints. You can find details on workspace limits here. These limits play a crucial role in determining the appropriate number of workspaces for your organization.

Pro Tip: You have the flexibility to create Unity Catalog at the account level, making it shareable across different workspaces. This allows you to generate schemas, tables, and views in one workspace and seamlessly access them from another workspace within the same account.

Prerequisites for Creating a Workspace

Let’s delve into the prerequisites for creating a workspace. As mentioned earlier, you’ll need a Databricks account subscription, essentially an admin account provided by Databricks for your company. This account acts as the administrative hub for managing your Databricks resources.

In addition to the Databricks account, you’ll also require a cloud provider account. This account is essential for provisioning the necessary compute resources within the Databricks environment. As of now, Databricks supports three major cloud providers: AWS, Azure, and GCP. Your choice of cloud provider is based on your company’s existing infrastructure and preferences.

As of the time of writing this article, it’s mandatory to associate a cloud account when creating a Databricks workspace. This integration ensures seamless access to the computational power needed for your workloads.

An important note is that, with the potential introduction of serverless capabilities, Databricks may evolve to eliminate the necessity of a dedicated cloud account for workspace creation in the future. This shift could bring about more flexibility and ease in setting up workspaces, aligning with the advancements in serverless computing technology.

Different Ways to Create a Workspace in Databricks

Databricks offers flexibility in creating workspaces, accommodating various infrastructure capabilities. Here are different methods to create workspaces:

Using UI: The Databricks admin account UI enables the straightforward creation of workspaces. Log in to the Databricks root/admin account, click on “Create Workspace,” and fill in the relevant details.
Using APIs: For teams preferring programmatic deployment, Databricks workspace APIs allow the creation, modification, and deletion of workspaces. These REST APIs can be utilized with any programming language.
Using Infrastructure as Code frameworks: Leveraging frameworks like Terraform or Bicep provides a cleaner and more readable approach. Databricks Terraform modules are actively maintained and developed.
Using Cloud Provider Templates: Various cloud providers offer templates that abstract complex details. This approach simplifies the process of creating Databricks workspaces.

Understanding Databricks workspaces is essential for optimizing their use within your organization. If you have any questions or insights, feel free to leave a comment.

Catch you later for more data wizardry! 😄📊🔮 Buy me a coffee