We are excited to announce that the Synapse Data Engineering public preview is shipping with ‘Runtime 1.1’ which includes Spark 3.3.1, Delta 2.2 and Python 3.10. ![]() Runtime with great default performance & robust admin controls Power BI can connect to the lakehouse data using ‘Direct Lake’ mode meaning it can read the data in the lake, with no data movement and with great performance. Every lakehouse also comes with a semantic dataset, enabling BI users to build reports directly on top of lakehouse data. The lakehouse comes with a SQL endpoint that provides data warehousing capabilities, including the ability to run T-SQL queries, create views and define functions. Since all the data in Microsoft Fabric is automatically stored in the Delta format, different data professionals can easily work together. The lakehouse also streamlines the process of collaborating on top of the same data. Ingested data comes by default in the Delta lake format, and tables are automatically created for users. Users can choose from various ways of bringing data into the lakehouse including dataflow & pipelines, and they can even use shortcuts to create virtual folders and tables without the data ever leaving their storage accounts. By making the lakehouse a first-class item in the workspace, we have made it really easy for any data engineer to create it and work with it. The Synapse Data Engineering lakehouse combines the best of the data lake and warehouse, removing the friction of ingesting, transforming, and sharing organizational data, all in an open format. Here are some of the key Synapse Data Engineering experiences that are launching as part of Microsoft Fabric at Build: Build a lakehouse for all your organizational data Instead of wasting cycles on the ‘integration tax’ of wiring together a collection of products, worrying about spinning up and managing infrastructure and stitching together disparate data sources, we want data engineers to focus on the jobs to be done. With Synapse Data Engineering, we aspire to streamline the process of working with your organizational data. What’s included in Synapse Data Engineering? With data engineering as a core experience in Fabric, data engineers will feel right at home, being able to leverage the power of Apache Spark to transform their data at scale and build out a robust lakehouse architecture. Microsoft Fabric empowers teams of data professionals to seamlessly collaborate, end-to-end on their analytics projects, ranging from data integration to data warehousing, data science and business intelligence. ![]() Today, we are excited to announce the preview of Synapse Data Engineering, one of the core experiences of Microsoft Fabric. ![]() This results in friction and project roadblocks, hampering productivity and leading to frustration. These processes are complex – data is fragmented across many sources, data sharing requires ETL jobs and synchronization, often to proprietary stores, security needs to be replicated multiple times, leading to inconsistencies. Data engineers need to tackle numerous challenges including data consolidation, security considerations as well as democratization of data, catering to different consumption needs. All this data needs to be ingested, processed at scale, and shared with the business. The amount of data that needs to be processed is growing faster than ever, ranging from tabular data to unstructured documents, images, IoT sensors and more. See Arun Ulagaratchagan’s blog post to read the full Microsoft Fabric preview announcement.ĭata engineering is playing an increasingly foundational role in every organization’s analytics journey.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |