Free AI web copilot to create summaries, insights and extended knowledge, download it at here

Abstract

JYlbzJm0cg.jpeg)"></div> </div> </div> </a> </div><p id="8090">Databricks does not create Large Language Models like ChatGPT itself, but leaves that to specialists. Instead, the company pursues a data-centric approach centered on its Lakehouse architecture, which is hosted in its own cloud, among other things. This architecture is now being extended to Lakehouse AI and supported with new generative AI tools[2][3].</p><p id="e70b">With Lakehouse AI, Databricks is now unifying the data and AI platform so customers can develop their generative AI solutions faster and more easily. Using basic SaaS models to securely train their own custom models with their enterprise data. By bringing together data, AI models, LLM operations (LLMOps), monitoring and governance on Databricks Lakehouse Platform, enterprises could accelerate their generative AI journey[1][4].</p><figure id="eeae"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/0*6vhGlKAdXTWhZK5N.png"><figcaption>Architecture of Lakehouse AI — Image Source: Databricks[4]</figcaption></figure><p id="bf95">Lakehouse AI unifies the AI lifecycle, from data collection and preparation to model development, LLMOps, deployment and monitoring. Databricks wants to overcome key challenges when developing generative AI solutions — like[4]:</p><ul><li>Optimizing Model Quality</li><li>Cost and Complexity of Training with Enterprise Data</li><li>Trusting Models in Production</li><li>Data Security and Governance</li></ul><p id="0fca">Databricks in form of Ghodsi also repeatedly emphasized attention to privacy laws and ensuring data security in order to put a stop to misuse of AI and manipulation of AI data sets and statements

Options

[1][3].</p><p id="64d9">This is an exciting approach that Databricks is taking here. The Data Lakehouse already combines Data Warehouse and Data Lake. Now, a component of AI is added and that in a double sense: On the one hand, AI should simplify the development (e.g. in the form of ChatBot functions) and on the other hand, the creation of own AI models and services should be simplified. It is interesting that Microsoft and Databricks in particular are setting the tone here. Google and Snowflake, which also offer popular solutions, are even more reserved here.</p><h2 id="4092">Sources and Further Readings</h2><p id="1b94">[1] InfoQ, <a href="https://www.infoq.com/news/2023/07/lakehouse-ai-mosaicml/">Databricks Unveils Lakehouse AI and MosaicML Acquisition at Data + AI Summit</a> (2023)</p><p id="3eea">[2] Databricks, <a href="https://www.databricks.com/resources/ebook/state-of-data-ai?scid=7018Y000001Fi0wQAC&utm_medium=paid+search&utm_source=google&utm_campaign=20116294957&utm_adgroup=146222918142&utm_content=ebook&utm_offer=state-of-data-ai&utm_ad=658048203096&utm_term=data%20lakehouse%20databricks&gclid=Cj0KCQjwk96lBhDHARIsAEKO4xb8Z0W8u9Bps-qKAmm0KUO7isjN3fbRYSEjmIYA2J9JUwQCzhDiE0IaAu_XEALw_wcB">Data + AI in the real world — Discover emerging trends in adoption and strategy</a> (2023)</p><p id="4ea4">[3] BigData Insider, <a href="https://www.bigdata-insider.de/datenkonferenz-ganz-im-zeichen-der-ki-a-f0d3cd70bb6e1f5dd35a1a34b21ac735/">Datenkonferenz ganz im Zeichen der KI</a> (2023)</p><p id="a655">[4] Databricks, <a href="https://www.databricks.com/blog/lakehouse-ai">Lakehouse AI: A Data-Centric Approach to Building Generative AI Applications</a> (2023)</p></article></body>

Databricks Data and AI Summit 2023 under the sign of AI

Databricks announced Lakehouse AI

How Databricks wants to attack Solutions like BigQuery, Snowflake & Co.

Databricks’ CEO Ali Ghodsi is all about Artificial Intelligence. Particularly in the area of Data platforms and analysis, he has announced Lakehouse AI and plans to buy MosaicML, which offers LL models[1].

The integration of AI and tools such as chatbot functions in cloud services, development, and also Data Warehousing, Engineering and Science continues to gain momentum. Microsoft, for example, recently presented a similar concept with Microsoft Factory, and now Databricks is following suit.

What is Microsoft Fabric and why it’s a Big Deal

Microsoft’s new End-to-End Data and Analytics Platform

medium.com

Databricks does not create Large Language Models like ChatGPT itself, but leaves that to specialists. Instead, the company pursues a data-centric approach centered on its Lakehouse architecture, which is hosted in its own cloud, among other things. This architecture is now being extended to Lakehouse AI and supported with new generative AI tools[2][3].

With Lakehouse AI, Databricks is now unifying the data and AI platform so customers can develop their generative AI solutions faster and more easily. Using basic SaaS models to securely train their own custom models with their enterprise data. By bringing together data, AI models, LLM operations (LLMOps), monitoring and governance on Databricks Lakehouse Platform, enterprises could accelerate their generative AI journey[1][4].

Architecture of Lakehouse AI — Image Source: Databricks[4]

Lakehouse AI unifies the AI lifecycle, from data collection and preparation to model development, LLMOps, deployment and monitoring. Databricks wants to overcome key challenges when developing generative AI solutions — like[4]:

Optimizing Model Quality
Cost and Complexity of Training with Enterprise Data
Trusting Models in Production
Data Security and Governance

Databricks in form of Ghodsi also repeatedly emphasized attention to privacy laws and ensuring data security in order to put a stop to misuse of AI and manipulation of AI data sets and statements[1][3].

This is an exciting approach that Databricks is taking here. The Data Lakehouse already combines Data Warehouse and Data Lake. Now, a component of AI is added and that in a double sense: On the one hand, AI should simplify the development (e.g. in the form of ChatBot functions) and on the other hand, the creation of own AI models and services should be simplified. It is interesting that Microsoft and Databricks in particular are setting the tone here. Google and Snowflake, which also offer popular solutions, are even more reserved here.

Sources and Further Readings

[1] InfoQ, Databricks Unveils Lakehouse AI and MosaicML Acquisition at Data + AI Summit (2023)

[2] Databricks, Data + AI in the real world — Discover emerging trends in adoption and strategy (2023)

[3] BigData Insider, Datenkonferenz ganz im Zeichen der KI (2023)

[4] Databricks, Lakehouse AI: A Data-Centric Approach to Building Generative AI Applications (2023)