Navigating the Deprecation of Export to Data Lake in D365 F&O

If you’ve been relying on the Export to Data Lake feature in Dynamics 365 Finance & Operations (D365 F&O), you’ve probably heard the news by now: Microsoft is deprecating this feature as of October 15, 2023.

Now, I know change is hard, and transitioning from something that has been a reliable workhorse for your data exports can feel daunting. But fear not, fellow dads and D365 aficionados! This blog is here to break down what’s happening, explore your options for transitioning, and highlight some critical watch-outs that you should consider during this migration.

What’s Happening?

The Export to Data Lake feature allowed users to seamlessly export data from D365 F&O to Azure Data Lake Gen2, where it could be used for all sorts of reporting, analysis, and machine learning projects. But as of November 1, 2024, this functionality will be fully retired, and new installations are already a no-go. Microsoft is steering everyone towards its new and improved solution: Azure Synapse Link.

Azure Synapse Link is more than just a replacement; it’s an upgrade designed to offer more robust features, including Delta Table functionality, which significantly improves data management, query performance, and reduces latency.

Your Transition Options

As you prepare to make the switch, you have several options to consider, each with its own set of pros, cons, and potential pitfalls. Let’s dive into each one.

Option 1: Export to Azure Data Lake in CSV Format (BYOL CSV)

This option is the most familiar to those of us used to the old Export to Data Lake feature. It exports your D365 F&O data to Azure Data Lake in CSV format within your Azure subscription. Here’s what you need to know:

Technology No Longer Needed:
- CDMUtil: This tool was previously used to create SQL views in a serverless SQL pool from the CSV files stored in Azure Data Lake. With the switch to Synapse Link, this manual process becomes obsolete.
Technology Being Added:
- Azure Synapse Link: This new service replaces the need for manually creating SQL views. It directly exports data to Azure Data Lake in the CSV format, simplifying the setup process.
Watch-Outs:
- Storage Costs: While this method might seem straightforward, keep an eye on storage costs. Large volumes of CSV files can quickly add up, especially if you’re dealing with high data refresh frequencies.
- Integration Complexity: If you’re integrating this data with Power BI or other tools, remember that CSV files might require additional transformation work, increasing the time and effort needed for data analysis.

Option 2: Export to Azure Data Lake in Delta Parquet Format

For those looking for better performance and scalability, exporting data in Delta Parquet format is a strong contender. Delta files support ACID transactions and schema evolution, making them more robust for big data scenarios.

Technology No Longer Needed:
- Serverless SQL Pools with CSV-based Data: The older system of handling CSVs through serverless SQL pools is no longer required since Delta Parquet files are optimized for direct consumption in Synapse Analytics.
Technology Being Added:
- Apache Spark Pool: Required for converting CSV files to Delta format, this adds another layer of infrastructure to manage but enhances data processing capabilities.
- Azure Synapse Workspace: A dedicated space within Azure Synapse Analytics where the converted Delta files are stored and processed.
Watch-Outs:
- Cost: The cost of running a Synapse workspace and Apache Spark pool (needed for CSV-to-Delta conversion) can be significant. If your data refreshes frequently or involves large volumes, these costs can spike.
- Complexity: Setting up and maintaining this environment requires more technical expertise, so ensure your team is ready for the challenge.

Option 3: Export to OneLake with Microsoft Fabric

The most integrated and streamlined option is exporting directly to OneLake, which is part of Microsoft Fabric. This method is designed to minimize the need for downstream applications to manage data conversions, offering a more seamless experience.

Technology No Longer Needed:
- Manual Data Conversion Processes: With OneLake managing data conversions, many of the manual steps required in previous setups are no longer necessary.
- External Data Warehouses: Since OneLake serves as a centralized repository, the need for external storage and data warehousing is reduced.
Technology Being Added:
- Microsoft Fabric: A comprehensive data platform that integrates with OneLake to provide a fully managed solution for storing and analyzing data.
- OneLake: The new storage solution that simplifies data access and integrates with Microsoft Fabric for enhanced analytics capabilities.
Watch-Outs:
- Limited Support: As of now, OneLake primarily supports D365 F&O and CE entities. If your data needs extend beyond these, this might not be the right option for you.
- Dependency on Microsoft Fabric: Since OneLake is a relatively new and evolving service, you might encounter growing pains as Microsoft continues to develop this offering. Keep an eye on updates and be prepared for potential changes.

A Word of Advice

Transitioning from a feature that’s been part of your workflow for years is never easy, but it’s also an opportunity to rethink and optimize how you handle data in D365 F&O. Whether you choose to stick with a CSV-based export, embrace the Delta Parquet format, or go all-in with OneLake and Microsoft Fabric (my preference), make sure you understand the implications, both technical and financial.

My advice? Start planning your migration now. Test out the different options in a sandbox environment, and consult with your IT and data teams to ensure a smooth transition.

And here’s your dad joke for the day

Why did the data file go to therapy? It had too many CSV issues.