Skip to content

Daniel Crawford's Data Blog

Tips for performant database designs

Menu

Home
Blog
Resources
About
Contact

Menu

Contact

So this is where I put some contact information for myself…

I can be found through the links below but am most active on LinkedIn:

LinkedIn
Twitter

Categories

Architecture Patterns
Fabric
Performance Tuning
Synapse
Top 10 Performance Considerations

Archives

July 2023
January 2023
December 2022
November 2022
September 2022
August 2022

Recent Synapse Videos

With high concurrency mode, we’re bringing a game-changing way to orchestrate your data ingestion and transformation processes in enterprise workflows. Notebooks in pipelines now leverage shared, high-performance sessions, combining speed with cost-efficiency—all while maintaining top-tier security.

Imagine a pipeline with five notebooks, each running 5 minutes. Normally, the 3-minute Spark start time per step would push your pipeline to 40 minutes. With high concurrency, the total runtime drops to 28 minutes—a 30% speed improvement.

Unlock faster workflows, lower costs, and a seamless data journey with high concurrency mode. Get ready to experience the next level of pipeline orchestration! 🎉

https://blog.fabric.microsoft.com/en-US/blog/introducing-high-concurrency-mode-for-notebooks-in-pipelines-for-fabric-spark/

🎙 Meet the Speakers:

👤 Guest from Product Group: Santhosh Kumar Ravindran, Senior Product Manager

Santhosh Ravindran currently leads Spark Compute and Settings for Microsoft Fabric Spark. He focuses on building capabilities that meet the data engineering needs like Spark Pools, Queueing and Scheduling, and Job orchestration for big data workloads in large enterprises using Spark. Prior to this, Santhosh was a Product Manager and Engineering Lead building Metadata scanners, access policy orchestration, lineage and data catalog systems for enterprise data estates as part of the Microsoft Governance Platform (Microsoft Purview).

LinkedIn: https://www.linkedin.com/in/thisissanthoshkumar/

Twitter: https://www.twitter.com/iamsanthoshkr

👤 Host: Estera Kot, Principal Product Manager at Microsoft.

LinkedIn: https://www.linkedin.com/in/esterakot/
Twitter: https://twitter.com/estera_kot

👍 Liked this video? Don't forget to hit the 'Like' button and share it with your peers!

High Concurrency Mode for Notebooks in Pipelines for Fabric Spark

Hyperparameter tuning involves determining the optimal values for parameters not learned by the machine learning model during training but set by the user prior to the initiation of the training process. These parameters, known as hyperparameters, include the learning rate, the number of hidden layers in a neural network, the regularization strength, and the batch size. The performance of a machine learning model can be highly sensitive to the selection of hyperparameters, with the optimal set varying significantly based on the specific problem and dataset. Consequently, hyperparameter tuning is a crucial step in the machine learning pipeline, substantially influencing the model's accuracy and generalization performance.

In the context of Fabric, data scientists can utilize FLAML, a lightweight Python library designed for the efficient automation of machine learning and AI operations, to meet their hyperparameter tuning needs. Within Fabric notebooks, users can employ `flaml.tune` for cost-effective hyperparameter tuning.

https://learn.microsoft.com/en-us/fabric/data-science/hyperparameter-tuning-fabric

https://blog.fabric.microsoft.com/en-US/blog/introducing-code-first-automl-and-hyperparameter-tuning-now-in-public-preview-for-fabric-data-science/

🎙 Meet the Speakers:

👤 Guest from Microsoft Fabric Product Group: Misha Desai, Senior Program Manager at Microsoft

Misha is a Senior Product Manager based in Seattle, WA, specializing in model tracking, training, and governance within the Fabric Data Science team.

Linkedin: www.linkedin.com/in/misha-desai-6034a362

👤 Host: Estera Kot, PhD, Principal Product Manager at Microsoft.

LinkedIn: https://www.linkedin.com/in/esterakot/

Twitter: https://twitter.com/estera_kot

👍 Liked this video? Don't forget to hit the 'Like' button and share it with your peers!

Hyperparameter tuning in Fabric Data Science

The environment serves as a comprehensive container for both your hardware and software settings within Spark. Within this unified interface, you have the ability to select the desired Spark runtime, install libraries, and configure Spark compute settings and properties. It simplifies the process of managing, maintaining, and sharing these configurations. Environment is now generally available and ready for production workloads.

https://learn.microsoft.com/en-us/fabric/data-engineering/create-and-use-environment

https://blog.fabric.microsoft.com/en-us/blog/environment-is-now-generally-available/

🎙 Meet the Speakers:

👤 Guest Expert: Shuaijun Ye | Fabric Data Engineering

LinkedIn: https://www.linkedin.com/in/shuaijun-ye-133805125/

👤 Host: Estera Kot, PhD, Principal Product Manager at Microsoft.

LinkedIn: https://www.linkedin.com/in/esterakot/

Twitter: https://twitter.com/estera_kot

👍 Liked this video? Don't forget to hit the 'Like' button and share it with your peers!

Microsoft Fabric environment - A consolidated item for all your hardware and software settings

Microsoft Fabric supports job queueing for Spark compute when capacity limits are reached, ensuring jobs are retried until they either execute or expire. Admins allocate resources within Azure-based capacities, affecting how jobs are queued. The system supports queueing for pipeline-triggered Notebook jobs and Spark job definitions but not for interactive or public API-triggered Notebook jobs. Jobs are processed in a First-In-First-Out manner.

https://learn.microsoft.com/en-us/fabric/data-engineering/job-queueing-for-fabric-spark

🎙 Meet the Speakers:

👤 Guest from Product Group: Santhosh Kumar Ravindran, Senior Product Manager

Santhosh Ravindran currently leads Spark Compute and Settings for Microsoft Fabric Spark. He focuses on building capabilities that meet the data engineering needs like Spark Pools, Queueing and Scheduling, and Job orchestration for big data workloads in large enterprises using Spark. Prior to this, Santhosh was a Product Manager and Engineering Lead building Metadata scanners, access policy orchestration, lineage and data catalog systems for enterprise data estates as part of the Microsoft Governance Platform (Microsoft Purview).

Linkedin: https://www.linkedin.com/in/thisissanthoshkumar/

Twitter: https://www.twitter.com/iamsanthoshkr

👤 Host: Estera Kot, Principal Product Manager at Microsoft.

LinkedIn: https://www.linkedin.com/in/esterakot/
Twitter: https://twitter.com/estera_kot

👍 Liked this video? Don't forget to hit the 'Like' button and share it with your peers!

Job Queueing for Notebook in Microsoft Fabric | Spark Compute

Apache Airflow job in Microsoft Fabric provides an integrated, cloud-based platform for development, scheduling, and monitoring python-based data workflows, articulated as Directed Acyclic Graphs (DAGs). This innovation delivers a Software-as-a-Service (SaaS) experience for data pipeline development and management using Apache Airflow, making Apache Airflow runtime readily accessible for the development and operationalization of your data workflows.

https://blog.fabric.microsoft.com/blog/introducing-data-workflows-in-microsoft-fabric

🎙 Meet the Speakers:

👤 Guest Expert: Abhishek Narain, Principal Program Manager in Microsoft. Abhishek serves as a Product Manager at Microsoft Fabric's Data Factory. His primary area of focus is Data Factory, a tool that empowers data engineers to construct scalable, robust, and reliable ETLs.

LinkedIn: https://www.linkedin.com/in/narain-abhishek/

Twitter: https://x.com/narainabhishek

👤 Host: Estera Kot, Principal Product Manager at Microsoft.

LinkedIn: https://www.linkedin.com/in/esterakot/
Twitter: https://x.com/estera_kot

👍 Liked this video? Don't forget to hit the 'Like' button and share it with your peers!

Apache Airflow Job in Microsoft Fabric

Load More... Subscribe