Brett Haralson

Apache Spark has a powerful scheduling system that provides default FIFO and Fair scheduling algorithms. But this looks a bit limiting to use only these algorithms in some use cases. So you can develop a custom scheduler. Here's a brief guide on writing your own scheduler: 1. Understand the Default Schedulers: Spark uses "TaskSchedulerImpl" to manage task scheduling. By default, it supports FIFO and Fair schedulers. 2. Define Your Custom Algorithm: Create a new class that implements the "SchedulingAlgorithm" trait. This class should define how tasks are prioritized by overriding its "comparator" method. 3. Build Your Scheduler: Write an implementation for "SchedulableBuilder" to create and manage pools and task sets using your custom algorithm. You override its "buildPools", "addTaskSetManager" and "addSchedulable" methods. 4. Integrate with Spark: Modify the TaskSchedulerImpl to instantiate your custom scheduler based on a new configuration option. For a deeper understanding, check Spark's org.apache.spark.scheduler package. You can fork the project in this link: https://lnkd.in/dsrUh9K9 #BigData #ApacheSpark #DataEngineering #Scheduler #Scala #DistributedComputing #Spark

12

My SQL Server instance uses SSDs. Can I shrink database files automatically without any performance impact? It's common to encounter data or log files that have grown significantly larger than necessary. This could be due to a long-running transaction or a temporary increase in database size due to bulk operations. To reclaim space, I've encountered several methods that are either scheduled to shrink the data files or auto-shrink is enabled. This can cause fragmentation where data pages are not stored contiguously. 𝐂𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞: - Does fragmentation matter when using SSDs? Why or Why not? Can this be proven? - List 2 other methods that could be used instead of automatically shrinking the database files. - List scenarios in which it would be appropriate to shrink database files. - How can you ensure you're notified anytime Auto Shrink is enabled? Join 430+ professionals improving their skills through automation challenges and hands-on learning: https://buff.ly/40SeFDp Found this useful? Hit 👍 Something to add? Drop it in the comments ✍️ Think others would benefit? Repost ♻️ #dbachallenges #automation

8

5 Comments

🚨 Is your data engineering workflow holding you back? 🚨 MotherDuck and Dagster Labs are teaming up to discuss the secrets of modern data engineering. We'll cover: - Crafting an end-to-end data platform with Bluesky data, MotherDuck, and Dagster - Proactively solving problems with "shift-left" testing to catch issues early - Practical automation strategies to streamline your workflow and boost productivity 📅 January 14th ⏰ 9:00 AM PT ☑️ Registration link in the comments

13

1 Comment

🎉 29 Days of Airflow 2.9: Day 23 🎉 📈 Today's feature update in #ApacheAirflow 2.9 focuses on enhancing data visibility: adding datasets to the DAG graph. This improvement, delivered via pull request #37604, integrates datasets directly into the visual #DAG graph in the #Airflow UI. This allows users to see how datasets are connected to various tasks, providing a clearer understanding of how data flows through and impacts workflows. 🔍 Why is this significant? Incorporating datasets into the DAG graph helps users visualize and comprehend the dependencies and relationships between tasks and data more intuitively. This addition is crucial for troubleshooting and optimizing data pipelines, ensuring a more efficient and effective data handling process. 🔗 Learn more about the benefits of visualizing datasets in your DAG graphs: https://bit.ly/458xvqh #Airflow29 #DataVisualization #29daysofairflow29 #DataEngineering

19

Analyzing massive datasets doesn't have to be a challenge. KX Data Scientist Ryan Siegler shares how KDB.AI’s partitioning feature makes it easy to scale vector search operations without sacrificing performance.​ In his latest blog, you’ll learn:​ 💡How partitioning accelerates query performance and scales massive datasets​ 💡Key metadata types for organizing datasets (date, integer, and symbol)​ 💡The versatility of partitioning across index types like qHNSW and TSS​ 💡Practical benefits for time-series analysis and enterprise AI applications​ Discover how partitioning can transform your data operations: https://lnkd.in/eaKTV4ah​ #KX #KDBAI #DataAnalytics #KDBAI #VectorSearch #TimeSeriesData #BigData #EnterpriseAI

11

Learn all about “Real-time Predictions with PostgreSQL and PL/Python”with Drishti Jain at PGConf NYC 2024! https://buff.ly/3yzf1TF @drishtijjain @PostgreSQL #PostgreSQL #postgres #pgconfnyc

1

Awesome post is up on the blog by Arjun Anandkumar on his journey migrating away from Control-M to Airflow. Definitely worth a read!!!

58

1 Comment

❓ Do you use dbt Labs in Apache Airflow? 🤗 We value your input and want to hear from YOU! 🙏🏽 Please take a few minutes to fill out this survey, and you will help improve the user experience of dbt users in Airflow: https://bit.ly/4egyJnf

79

Super cool new features for our SQL Server Capture connector! 💪 Lower impact on the source DB We’ve restructured the connector to only query tables with new changes which results in fewer overall queries. ⏩ Lower latency Lower latency in extracting data when many tables are being captured 🧹 Automatic CDC table clean-up As soon as we’ve read data, we have the option to delete it from your source, resulting in lower disk usage. 👨‍💼 Automatic CDC instance management We can automatically create new change tables and remove old ones as your schema changes so you no longer have to do anything in that case. Check it out: https://lnkd.in/dyqzBPia

12

1 Comment

Dolt works with most workbenches, ORMs and SQL connectors that are available on the Internet and we've been writing blogs showing how to connect those tools to Dolt server. This blog by Jennifer Purevsuren goes over how to use MySQL Connector/Python to connect to Dolt database. https://lnkd.in/er-jaYCG

2

#ReductStore is a feature-rich time series object store, but that doesn't mean you have to be a #database expert to start working with it. See how easy it is to start storing and querying data with ReductStore in #Python 😍 . https://lnkd.in/dgnUMNm5

1

New feature release - Job metrics timeline view! Ever wonder why your Databricks job performance changes over time? You may ask yourself questions such as: “Why is my runtime increasing since last week?” “Is the cost of this job also increasing?” “Is the input data size changing?” “Is my job in danger of crashing?” Understanding why your job performance is changing is critical to both optimize costs and prevent crashes. In this post we show a new visualization timeline features to help engineers find root causes quickly. Try it out today with Gradient! #dataengineering https://lnkd.in/ebPjtU7y

7

5 Comments

How do You Become a Performance Tuner? Yesterday's post on becoming a SQL Server performance tuning type person seemed to get some attention. So the next question is how? What resources (aside from experienced teammates) would point a DBA to? Assume they have 3-5 years of solid DBA experience. Books? Videos? Websites with labs? If you ARE that person...how best do you learn? Let's hear from my Senior DBA crowd! And everyone else too! 😂 ----------------- Follow me to get notified when I post I post regularly, but not every day 😁 #SQLServer  #FractionalDBA  #PocketDBA #Coffee Get a Free SQL Server Health Check! https://lnkd.in/dYNteiKg

23

11 Comments