In my GitHub collection of SQL Scripts is now available a new stored procedure sp_tblCleanupRetentionWindow under the TablesManagement/Partitioning folder. Help for the function is available in the repository Wiki.
Scenario Overview
You have huge tables in the MS SQL Database and to be able to efectively manage the amount of data, you have partitioned the table.
As the amount of the data grows over the time, you start thinking about regular maintenance of such huge table and drop old not needed data.
You need to clean the oldest partitions and MERGE the partitions in the partitoin fuction to keep the number of partitions reasonable (especially if new partitions are created as new data are arriving).
To cleanup partitions prior SQL Server 2016 you have to perform partition switching to cleanup the partitions and then you could MERGE the partitions. On SQL Server 2016 and above the situation is simplified as you can use the TUNRCATE TABLE WITH PARTITIONS. But again to merge partition, you must put exact partition boundary values etc.
To automate regular maintenance you need to write sripts to handle all the needed tasks.
sp_tblCleanupRetentionWindow
As mentioned in the scenario above, the sp_tblCleanupRetentionWindow is the procedure which takes care about all the operations needed to cleanup the table prior specified retention window.
Parameters
| ParameterName | Data Type | Default | Description |
|---|---|---|---|
@pfName | nvarchar(128) | NULL | Name of the partition function. All associated partition schemes and tables will be cleaned according specified @retentionWindow |
@retentionWindow | sql_variant | NULL |
|
@infoOnly | bit | 1 | When 1 prints only information about affected tables and partitions and does not perform any cleanup. When 0 does the actual cleanup |
How It Operates
Procedure takes @pfName parameter, which identifies partition function used by partitioned table(s). Then based on the @retentionWindow parameter it idetnfifies partition containing the @retentionWindow value.
The @retentionWindow parameter identifies the lowest value which must be kept in the table. All records prior that one can be cleared.
If the @retentionWindow value is part of the first (leftmost) partition, the procedure ends and does not perform any cleanup as there is nothing to cleanup. It cannot clean the partition containing the @retentionWindow value.
If the @retentionWindow is part of second and higher partitions, it scan for all Partitions Shemes using the @pfName partition function.
After identification of the partition schemes it identifies all the tables using the identified partitions shemes.
Once the tables are identified, it loops through all the tables and TRUNCATES all the partitions prior the partition containing the @retentionWindow value.
After the truncation the procedure ALTERS the parition function and merges all the partitions prior the partition containing the @retentionWindow value to the first (leftmost) partition.
It keeps always the first leftmost partition empty.
By the scrip the procedure is deployed into the [master] database and marked as system stored procedure. This allows simple call of the procedure from within all databases on the SQL Server instance and also thanks to that the procedure is running in the context of the database from within it is being called.
Sample
The sample below does cleanup fo all tables associated with the pf_partDate partition function. Cleanup and merge all partitions prior partition containign value of @retentionWindow = '2020-12-15'. pf_partDate is using the date data type.
If we take a closer look on the pf_partDate partition function by using the sp_HelpPartitionFunction we will find, it has multiple partitions and is used by multipel partition schemes and tables.
sp_HelpPartitionFunction 'pf_partDate', 1
Info Only
If we execute the procedure with @infoOnly=1 then only information about affected partitions and tables will be provided.
DECLARE @retentionWindow date = '2020-12-05'
EXEC sp_tblCleanupRetentionWindow
@pfName = 'pf_partDate'
,@retentionWindow = @retentionWindow
,@infoOnly = 1
Info Only Output
Below is sample output of the procedure which informs about actions which will be performed if the @infoOnly would be set to 0.
sp_tblCleanupRetentionWindow v0.10 (2021-05-07) (C) 2017-2021 Pavel Pawlowski
=============================================================================
Cleans retention window for all tables associated with partition function
Feedback mail to: pavel.pawlowski@hotmail.cz
Repository: https://github.com/PavelPawlowski/SQL-Scripts
-----------------------------------------------------------------------------
Cleaning retention window for partition function [pf_partDate]
Partitions to Cleanup:
----------------------
[ ] [x] < [2020-11-01]
[2020-11-01] <= [x] < [2020-12-01]
Affected Partition Schemes And Tables:
--------------------------------------
[ps_date1]
- [dbo].[tblDate1]
- [dbo].[tblDate11]
[ps_date2]
- [dbo].[tblDate2]From the output we can see, that there are two partitions which would be affected. First partition containing data prior 2020-10-01 and second partition containing data between the 2020-10-01 and 2020-11-01.
Also it informs, that there are two partitions schemes. Partition scheme [ps_date1] which is used by tables [dbo].[tblDate1] and [dbo].[tblDate11] and partition scheme [ps_date2] which is used by table [dbo].[tblDate2].
Cleanup
If we execute the procedure with @infoOnly=0 then information about affected partitions and tables will be provided as well as actual cleanup will be done.
DECLARE @retentionWindow date = '2020-12-05'
EXEC sp_tblCleanupRetentionWindow
@pfName = 'pf_partDate'
,@retentionWindow = @retentionWindow
,@infoOnly = 0
Cleanup output
sp_tblCleanupRetentionWindow v0.10 (2021-05-07) (C) 2017-2021 Pavel Pawlowski
=============================================================================
Cleans retention window for all tables associated with partition function
Feedback mail to: pavel.pawlowski@hotmail.cz
Repository: https://github.com/PavelPawlowski/SQL-Scripts
-----------------------------------------------------------------------------
Cleaning retention window for partition function [pf_partDate]
Partitions to Cleanup:
----------------------
[ ] [x] < [2020-11-01]
[2020-11-01] <= [x] < [2020-12-01]
Affected Partition Schemes And Tables:
--------------------------------------
[ps_date1]
- [dbo].[tblDate1]
- [dbo].[tblDate11]
[ps_date2]
- [dbo].[tblDate2]
CLEANUP PROCESS
---------------------------------------------------------------
2021-05-18 23:54:11.4600180 +02:00 - Starting CLEANUP Process
2021-05-18 23:54:11.4610182 +02:00 - [dbo].[tblDate2] PARTITION 1 not empty. Starting TRUNCATE
2021-05-18 23:54:11.4620186 +02:00 - [dbo].[tblDate2] PARTITION 1 TRUNCATE completed
2021-05-18 23:54:11.4710215 +02:00 - [dbo].[tblDate11] PARTITION for range [2020-11-01] <= [x] < [2020-12-01] not empty. Starting TRUNCATE
2021-05-18 23:54:11.4760227 +02:00 - [dbo].[tblDate11] PARTITION for range [2020-11-01] <= [x] < [2020-12-01] TRUNCATE completed
2021-05-18 23:54:11.4770230 +02:00 - [pf_partDate] range [2020-11-01] <= [x] < [2020-12-01] : Start MERGE into PARTITION 1
2021-05-18 23:54:11.4790237 +02:00 - [pf_partDate] range [2020-11-01] <= [x] < [2020-12-01] : MERGE into PARTITION 1 completed. New range of PARTITION 1: [x] < [2020-12-01]
2021-05-18 23:54:11.4790237 +02:00 - CLEANUP Process COMPLETEDCompared to the @infoOnly=1 output we see additional section “CLEANUP PROCESS’, which informs about the actual cleanup steps performed.
Procedure identified, that partition 1 of the [dbo].[tblDate2] was not empty and therefore performed TRUNCATE of the partition 1. Then it identified that the table [dbo].[tblDate11] has data in partition covering range between dates 2020-11-01 and 2020-12-01, therefore performed also truncate of that partitions. As there is no information about any truncate of the table [dbo].[tblDate1] it means the table had both affected partitions empty. The same is related to the other afected partitions of the [dbo].[tblDate2] and [dbo].[tblDate11].
After the partition cleanup it started the multi-step partition merge process. It merged each affected partition into the left most partition and informs about the new range of the leftmost partition 1.
If we again utilize the sp_HelpPartitionFunction we can see, that the partitions were properly cleared.
Azure support
The procedure supports also execution in Azure SQL Databases as well as Instances or Azure Synapse Analytics.
When deploying to Azure SQL Database or Synapse Analytics, comment-out the unsuported USE statement or ignore the eventual error as stated in the header of the procedure source file.
/* *****************************************************************************************
AZURE SQL DB Notice
Comment-out the unsupported USE [master] when running in Azure SQL DB/Synapse Analytics
or ignore error caused by unsupported USE statement
******************************************************************************************** */
USE [master]
GO
SQL Server Version Limitation
Because the procedure internally is using TRUNCATE TABLE WITH PARTITION , procedure is limited only to the SQL Server 2016 and above or Azure SQL Database, Azure Managed Instance and Azure Synapse Analytics.
Summary
The sp_tblCleanupRetentionWindow heavily simplifies the management of huge partitioned table and moves the complete maintenance of retention window into a single call of a stored procedure. Hopefully you will find the procedure usefull.