Backing up Microsoft Planner with dlt and Azure Blob

TL/DR: Microsoft Planner has no native backup export (other than manual exports in the UI), so I built a scheduled Microsoft Graph extraction that writes immutable JSONL snapshots to Azure Blob Storage.

The NPO I work for, Pro Juventute, organizes their work not in hierarchical teams, but in circles. This org-structure requires decisions to be documented and available to all employees. And when it came to decide where those decisions should live, the org went with Microsoft Planner.

I had a simple-sounding requirement: keep a recoverable copy of Planner plans. However, Planner has no native backup export, and the supported extraction path is Microsoft Graph. That meant dealing with OAuth, pagination, retries, related endpoint families, and enough task-detail calls to make a coffee break feel optimistic.

Starting with Graph permissions

I used an existing Azure Entra app registration and had two application permissions admin-consented before writing the pipeline:

• Group.Read.All to enumerate Microsoft 365 groups and discover their plans.
• Tasks.Read.All to read plans, details, buckets, tasks, and task details.

Letting dlt own the REST work

My first prototype used custom requests code for OAuth, retries, pagination, and endpoint traversal. It worked, which is often how these things become production code before anybody has had time to think about them.

I replaced that prototype with dlt's declarative REST API source:

@dlt.source(name="planner_backup", max_table_nesting=0)
def planner_source() -> Any:
    auth = OAuth2ClientCredentials(...)
    config: RESTAPIConfig = {...}
    yield from rest_api_resources(config)

dlt handles OAuth2 client-credentials token acquisition and refresh, HTTP retries, Graph @odata.nextLink pagination, parent-to-child endpoint resolution, and parallel extraction. It also manages extraction packages, normalization, pipeline state, and Blob loading.

Why keep this declarative? Every custom request loop becomes another place where a backup can silently miss data. I personally prefer keeping production code focused on describing Graph resources and their relationships. And my coding agent agrees - it's on a strict token diet 😅

Modeling related Planner resources

Graph exposes the data through separate endpoint families. I modeled those relationships as related dlt resources instead of creating one custom nested document per plan:

groups.id
  -> plans._groups_id
     -> plan_details._plans_id
     -> buckets._plans_id
        -> tasks._buckets_id
           -> task_details._tasks_id

Use the Planner REST API - Microsoft Graph v1.0

You can use the Planner API in Microsoft Graph to create tasks and assign them to users in a group in Microsoft 365.

Microsoft LearnTarkanSevilmis

The resolved-resource prefixes preserve join keys. A restore process can use them to connect groups, plans, buckets, tasks, and details after extraction.

The six resource files are groups, plans, plan_details, buckets, tasks, and task_details. Assignments stay as fields on tasks. Checklists and references stay as nested JSON fields on task details. With max_table_nesting=0, dlt keeps those structures in JSON.

Choosing JSONL snapshots

I tested Parquet, then removed it. A pure declarative dlt setup writes one loader format per resource and load package. Producing both formats would require a second Graph extraction or custom conversion code.

JSONL is a better source of truth for this backup. Each line is one JSON object, so dlt can stream large resources without holding an entire resource in memory. The nested Graph fields also remain JSON values. The resulting files:

planner/<yyyyMMdd_hhmm>/
  groups.jsonl.gz
  plans.jsonl.gz
  plan_details.jsonl.gz
  buckets.jsonl.gz
  tasks.jsonl.gz
  task_details.jsonl.gz
  _dlt_loads/
  _dlt_pipeline_state/
  _dlt_version/
  init

Every resource uses write_disposition="replace". Each timestamped folder is a complete snapshot (which makes eventual restore processes much easier), merge or incremental state would add complexity without helping the backup.

Making the Blob layout predictable

The filesystem destination requires a dataset prefix. I wanted the timestamp itself to be the folder name, without another normalized prefix:

destination = dlt.destinations.filesystem(
    bucket_url="az://planner",
    layout="{table_name}.{ext}",
    enable_dataset_name_normalization=False,
)
pipeline = dlt.pipeline(
    pipeline_name="planner_to_blob",
    dataset_name=datetime.now(timezone.utc).strftime("%Y%m%d_%H%M"),
)

enable_dataset_name_normalization=False matters here. dlt normally prefixes a numeric dataset name with an underscore. Disabling normalization preserves the timestamp folder exactly.

The verified local run created planner/20260714_1424/. The load step completed in 3.04 seconds after extraction and normalization.

Finding the slow part

The full local run extracted 803 groups, 309 plans, 1,700 buckets, and 22,276 tasks with matching task details. Extraction took about 50 minutes. Normalization and Blob loading took a few seconds.

The bottleneck is clear: roughly 22,276 task-detail requests at about 7.5 requests per second. My default workspace has four extraction workers, and dependent resources are marked parallelized=True. In production, this job runs with twelve workers:

pipeline.extract(
    planner_source(),
    workers=12,
    loader_file_format="jsonl",
)
pipeline.load()

This is balanced between running into Graph 429 responses and wait time. Graph $batch could reduce request overhead too, although that would require custom Graph request code. I left it out deliberately... for now.

Deploying the backup

The dltHub deployment completed with:

uv run dlthub deploy --non-interactive

The new job runs twice per day. And the corresponding restore process is separate work. It would need to create missing group membership, plans, buckets, tasks, assignments, and details in dependency order. I'm not pretending that restore is free, but accidentally deleting a task from Planner shall not be free anyway 😅

The full dlt pipeline:

"""Microsoft Planner backup to Azure Blob Storage.

Each run exports Graph resources as JSONL:
    planner/yyyyMMdd_hhmm/<resource>.jsonl.gz

The dlt REST API source owns Microsoft Graph authentication, retries,
pagination, and parent-resource resolution.
"""

from datetime import datetime, timezone
from typing import Any

import dlt
from dlt.sources.helpers.rest_client.auth import OAuth2ClientCredentials
from dlt.sources.rest_api import rest_api_resources
from dlt.sources.rest_api.typing import RESTAPIConfig


@dlt.source(name="planner_backup", max_table_nesting=0)
def planner_source() -> Any:
    auth = OAuth2ClientCredentials(
        access_token_url=(
            f"https://login.microsoftonline.com/{dlt.secrets['sources.azure_graph.tenant_id']}"
            "/oauth2/v2.0/token"
        ),
        client_id=dlt.secrets["sources.azure_graph.client_id"],
        client_secret=dlt.secrets["sources.azure_graph.client_secret"],
        access_token_request_data={"scope": "https://graph.microsoft.com/.default"},
    )
    config: RESTAPIConfig = {
        "client": {
            "base_url": "https://graph.microsoft.com/v1.0/",
            "auth": auth,
        },
        "resource_defaults": {
            "primary_key": "id",
            "write_disposition": "replace",
            "max_table_nesting": 0,
        },
        "resources": [
            {
                "name": "groups",
                "endpoint": {
                    "path": "groups",
                    "params": {"$top": 999, "$select": "id,displayName"},
                    "data_selector": "value",
                    "paginator": {
                        "type": "json_link",
                        "next_url_path": "$['@odata.nextLink']",
                    },
                },
            },
            {
                "name": "plans",
                "endpoint": {
                    "path": "groups/{group_id}/planner/plans",
                    "params": {
                        "group_id": {"type": "resolve", "resource": "groups", "field": "id"},
                    },
                    "data_selector": "value",
                    "paginator": {
                        "type": "json_link",
                        "next_url_path": "$['@odata.nextLink']",
                    },
                },
                "include_from_parent": ["id"],
                "primary_key": ["_groups_id", "id"],
                "parallelized": True,
            },
            {
                "name": "plan_details",
                "endpoint": {
                    "path": "planner/plans/{plan_id}/details",
                    "params": {
                        "plan_id": {"type": "resolve", "resource": "plans", "field": "id"},
                    },
                    "data_selector": "$",
                    "paginator": "single_page",
                    "response_actions": [{"status_code": 404, "action": "ignore"}],
                },
                "include_from_parent": ["id"],
                "primary_key": "id",
                "parallelized": True,
            },
            {
                "name": "buckets",
                "endpoint": {
                    "path": "planner/plans/{plan_id}/buckets",
                    "params": {
                        "$top": 200,
                        "plan_id": {"type": "resolve", "resource": "plans", "field": "id"},
                    },
                    "data_selector": "value",
                    "paginator": {
                        "type": "json_link",
                        "next_url_path": "$['@odata.nextLink']",
                    },
                },
                "include_from_parent": ["id"],
                "primary_key": ["_plans_id", "id"],
                "parallelized": True,
            },
            {
                "name": "tasks",
                "endpoint": {
                    "path": "planner/buckets/{bucket_id}/tasks",
                    "params": {
                        "$top": 200,
                        "bucket_id": {"type": "resolve", "resource": "buckets", "field": "id"},
                    },
                    "data_selector": "value",
                    "paginator": {
                        "type": "json_link",
                        "next_url_path": "$['@odata.nextLink']",
                    },
                },
                "include_from_parent": ["id"],
                "primary_key": ["_buckets_id", "id"],
                "parallelized": True,
            },
            {
                "name": "task_details",
                "endpoint": {
                    "path": "planner/tasks/{task_id}/details",
                    "params": {
                        "task_id": {"type": "resolve", "resource": "tasks", "field": "id"},
                    },
                    "data_selector": "$",
                    "paginator": "single_page",
                    "response_actions": [{"status_code": 404, "action": "ignore"}],
                },
                "include_from_parent": ["id"],
                "primary_key": "id",
                "parallelized": True,
            },
        ],
    }
    yield from rest_api_resources(config)


def load_planner_backup() -> None:
    timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M")
    pipeline = dlt.pipeline(
        pipeline_name="planner_to_blob",
        destination=dlt.destinations.filesystem(
            bucket_url="az://planner",
            layout="{table_name}.{ext}",
            enable_dataset_name_normalization=False,
        ),
        dataset_name=timestamp,
        progress="log",
    )
    print(pipeline.run(planner_source(), loader_file_format="jsonl"))


if __name__ == "__main__":
    load_planner_backup()