Introducing Data Factory in Microsoft Fabric

Microsoft Fabric has unveiled its new tool, Data Factory, which is designed to meet the increasingly complex data integration needs of businesses undergoing digital transformation. The tool brings together the best of Power Query and Azure Data Factory, offering a modern data integration experience that empowers both data and business professionals. Data Factory is capable of ingesting and transforming data and orchestrating data workflows, thus facilitating more insightful business decisions.

Key Features

Data Factory is designed to cater to the needs of both citizen and professional developers, offering a platform for developing enterprise-scale data integration solutions. It provides next-generation dataflows and data pipelines, including:

• Seamless connectivity to more than 170+ data stores, including on-premises data sources, cloud databases, analytical platforms, and line of business applications.
• Next-generation Power BI dataflows as part of Data Factory in Fabric, offering 300+ out-of-the-box data transformations, including AI transformations, and scalable data flows running on Fabric compute.
• Data pipelines in Fabric, which is an evolution of Azure Data Factory pipelines, providing a rich set of integration capabilities, including a copy assistant for any copy task from data sources to data destinations.
• Built-in AI that can accelerate and automate common data integration tasks, like extracting data from unstructured data sources such as a webpage or a text file.

One of the most notable features is the ability of Data Factory to use AI to learn from examples provided by the user. This enables the extraction of specific data from unstructured sources, a feature that greatly enhances the flexibility and efficiency of the data integration process.

Partnerships and Use Cases

To ensure that Data Factory in Microsoft Fabric meets the diverse needs of its customers, Microsoft has partnered with leading industry players such as Hitachi Solutions and Delphix. Hitachi Solutions leverages the tool to integrate data from multiple sources for their clients, while Delphix has introduced Compliance Services for Data Factory on the Microsoft Fabric platform, strengthening data compliance capabilities for enterprises across industries.

Getting Started with Dataflow in Microsoft Fabric Data Factory

Dataflow is a key component of Data Factory in Microsoft Fabric. It is responsible for extracting data from the source, transforming it, and loading it into a destination. The transformation process, often referred to as ETL (Extract, Transform, and Load), gets data from variable sources, transforms it into the required shape and format, and loads it into the desired destination. Power Query, the data transformation engine used in Power BI, is now available as a standalone cloud-based data transformation service when used as Dataflow in Microsoft Fabric.

With Dataflow in Microsoft Fabric, you can have data destinations in addition to Azure Data Lake Storage (ADLS) and Dataverse. Other options include Lakehouse, Data Warehouse, Azure SQL Database, and Azure Data Explorer (Kusto). Another enhancement is the ability to export the Dataflow as a template, which can then be used to create a similar dataflow in another workspace or environment.

Creating a Dataflow Gen2 begins from the Data Factory workload inside the Fabric portal. Data can be imported from a range of sources including Excel, and then transformed using the graphical interface in the Dataflow’s Power Query Editor. Dataflow supports the application of transformations using a graphical interface, which requires no coding. All transformations convert to lines of code in the Power Query’s scripting language, known as M.

After building your transformations, you can define the data destination, where the dataI’m sorry, but it seems I wasn’t able to find an example of AI transformations in Data Factory in Microsoft Fabric. While AI transformation is mentioned as a feature of Data Factory in Microsoft Fabric, the examples and tutorials I found mainly discussed general data transformations using Power Query, which doesn’t seem to involve AI-based transformations.

However, as per the official announcement from Microsoft, AI in Data Factory can be used to analyze examples you provide on values you want to extract from unstructured data sources like web pages or text files without a well-defined format. The AI capabilities then learn from these examples and create Data Factory dataflows to extract the data you need.

For a detailed example or tutorial on this feature, I recommend reaching out to Microsoft support or consulting the official Microsoft Fabric documentation, which might have more up-to-date and in-depth resources on this topic.