SQL & Python Patterns

Tool-belt series · SQL · Python · Vocabulary

Twelve shapes that solve
most of the actual work.

Every real data problem is a recombination of about a dozen patterns. This page names each one in plain English first — what you're trying to do and how to recognise it — then gives you the SQL, then the Python where Python is genuinely better. The point isn't to memorise syntax. It's vocabulary: you can't ask an AI, a colleague, or a search box for something you can't name.

How to use this if you learn by doing

Don't read it end to end. Skim the twelve headings once so the names land, then come back when you hit a problem and think "this is the one where…". Naming the pattern is 80% of solving it — once you know you need a window function, the syntax is a two-minute lookup or one prompt away.

The SQL is written to run on Fabric's SQL endpoint, a Warehouse, or Snowflake with minimal changes. Where a dialect differs meaningfully, it says so.

01

Window functions — the single highest-leverage thing to learn

If you learn one thing on this page, learn these. A window function looks at other rows while staying on the current row — that's the whole idea, and it's what separates people who write awkward self-joins from people who write four clean lines. Five of the twelve patterns below are just window functions wearing different hats.

01

Deduplicate — keep one row per thing

You want this when: an extract has the same record more than once and you need exactly one — usually the most recent. This is the most-used pattern in this entire page.

The idea in words

Number the rows within each group, ordered so the one you want is number 1. Then keep number 1. That's it — and once you see it, you'll use it constantly.

WITH ranked AS (
  SELECT *,
    ROW_NUMBER() OVER (
      PARTITION BY customer_id        -- one per customer
      ORDER BY updated_at DESC       -- newest wins
    ) AS rn
  FROM raw_customer
)
SELECT * FROM ranked WHERE rn = 1;
Read it aloud"Partition by" = start counting again for each. "Order by" = which one gets to be number 1. Change those two lines and this same shape solves a dozen different問 problems.
Watch outIf two rows tie on updated_at, which one wins is arbitrary and can change between runs — producing a table that differs each load for no visible reason. Add a tiebreaker to the ORDER BY (a surrogate key, a load timestamp) whenever the result has to be stable.
02

Compare to the previous period

You want this when: anyone says "versus last month", "change on prior", "movement", or "delta". Which in a finance org is roughly every second request.

The idea in words

Reach backwards a fixed number of rows and pull that value onto the current row, so you can subtract them side by side. LAG looks back; LEAD looks forward.

SELECT
  period,
  cost_centre,
  amount,
  LAG(amount) OVER (PARTITION BY cost_centre ORDER BY period) AS prior_amount,
  amount - LAG(amount) OVER (PARTITION BY cost_centre ORDER BY period) AS movement
FROM monthly_actuals;
The classic bugIf a cost centre has no rows in March, LAG gives you February — and silently calls it "prior month". If gaps are possible, join to a date spine first (pattern 07) so missing periods exist as zero rather than vanishing.
03

Running totals and rolling averages

You want this when: "cumulative year to date", "3-month rolling average", "balance over time". Anything that accumulates.

The idea in words

Same window, but instead of one specific row you define a range of rows to aggregate over — "everything up to here", or "these three".

SELECT
  period, amount,
  SUM(amount) OVER (
    ORDER BY period
    ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW      -- running total
  ) AS ytd,
  AVG(amount) OVER (
    ORDER BY period
    ROWS BETWEEN 2 PRECEDING AND CURRENT ROW            -- 3-month rolling
  ) AS rolling_3m
FROM monthly_actuals;
In Power BI, don'tIf the destination is a semantic model, do this in DAX with time-intelligence measures instead — it stays responsive to whatever the user filters. Precompute it in SQL only when the grain is fixed and the cost of recalculating is real.
04

Rank within a group — top N per category

You want this when: "top 5 customers per region", "biggest three variances per department". Note this is pattern 01 with a different filter — same machinery.

WITH r AS (
  SELECT region, customer, revenue,
    DENSE_RANK() OVER (PARTITION BY region ORDER BY revenue DESC) AS rk
  FROM sales
)
SELECT * FROM r WHERE rk <= 5;
Which ranking functionROW_NUMBER never ties (1,2,3,4). RANK ties then skips (1,1,3). DENSE_RANK ties without skipping (1,1,2). For "top 5" you almost always want DENSE_RANK, otherwise a tie silently costs you a row.
05

Gaps and islands — find consecutive runs

You want this when: "how many days in a row was this breached", "find the missing invoice numbers", "how long was each outage". Sounds exotic, comes up constantly once you can name it.

The idea in words

The trick that makes it click: if you number the rows sequentially and subtract that from the date, every row in a consecutive run gets the same answer. Group by that answer and you've found your runs. It's a genuinely clever trick and it's worth sitting with for a minute.

WITH g AS (
  SELECT breach_date,
    DATEADD(day,
      -ROW_NUMBER() OVER (ORDER BY breach_date),
      breach_date) AS grp                    -- constant within a run
  FROM daily_breaches
)
SELECT MIN(breach_date) AS run_start,
       MAX(breach_date) AS run_end,
       COUNT(*)        AS days_in_run
FROM g GROUP BY grp
ORDER BY run_start;
02

Reshaping — making data the right shape

Most "the data's wrong" problems are actually "the data's the wrong shape" problems. These four fix that.

06

Unpivot — columns that should be rows

You want this when: a finance extract has a column per month (Jan, Feb, Mar…). Humans love this layout; every analytical tool hates it. Almost every spreadsheet-sourced dataset needs this first.

The idea in words

Turn twelve columns into two: one saying which month, one saying the value. Long and boring beats wide and clever — every time, for anything downstream.

-- portable version: works everywhere
SELECT account, 'Jan' AS period, jan AS amount FROM budget
UNION ALL SELECT account, 'Feb', feb FROM budget
UNION ALL SELECT account, 'Mar', mar FROM budget;

-- T-SQL has a built-in for it
SELECT account, period, amount
FROM budget
UNPIVOT (amount FOR period IN (jan, feb, mar)) AS u;
Python is better hereIf there are 36 columns, don't write 36 UNION ALLs. One line of pandas: df.melt(id_vars=['account'], var_name='period', value_name='amount'). This is exactly the kind of job where a notebook beats SQL.
07

Date spine — make the missing rows exist

You want this when: a month with no activity vanishes from your report instead of showing zero, so charts have gaps and prior-period comparisons quietly lie (see pattern 02).

The idea in words

Build a table of every date you care about, then LEFT JOIN your data onto it. Now absence is visible as a zero rather than invisible as a missing row — which is nearly always what people actually meant.

WITH spine AS (
  SELECT CAST('2024-01-01' AS date) AS d
  UNION ALL
  SELECT DATEADD(month, 1, d) FROM spine WHERE d < '2026-12-01'
)
SELECT s.d AS period, COALESCE(a.amount, 0) AS amount
FROM spine s
LEFT JOIN monthly_actuals a ON a.period = s.d
OPTION (MAXRECURSION 0);
Do it once, properlyBetter still: keep a permanent date dimension table with month, quarter, fiscal year and period-end flags. Every model you build will use it, and your fiscal calendar lives in exactly one place instead of being reinvented in each report.
08

Banding — turn numbers into categories

You want this when: "group these into small / medium / large", ageing buckets, materiality tiers. Any time a continuous number needs to become something you can group by.

SELECT invoice_id, days_overdue,
  CASE
    WHEN days_overdue <= 0  THEN 'Current'
    WHEN days_overdue <= 30 THEN '1-30'
    WHEN days_overdue <= 60 THEN '31-60'
    ELSE '60+'
  END AS ageing_bucket
FROM receivables;
Better as a tableHard-coded CASE logic gets copied into six reports and then diverges. Put the bands in a small lookup table and join to it — then changing "60+" to "90+" is one row, not a search across your estate.
09

Slowly changing dimensions — keeping history

You want this when: a customer moves region, a cost centre is renamed, an account is remapped — and last year's report must still show what was true then. This is the pattern that makes you sound senior, because most people never learn it.

The idea in words

Instead of overwriting the row, close the old one off with an end date and insert a new one. Each row becomes "this was true between these dates", and you join on the date to get the version that applied at the time.

-- one row per version, not per entity
customer_key | customer_id | region | valid_from | valid_to   | is_current
1            | C100        | North  | 2024-01-01 | 2025-06-30 | 0
2            | C100        | South  | 2025-07-01 | 9999-12-31 | 1

-- "what was true when the transaction happened"
SELECT t.*, c.region
FROM transactions t
JOIN dim_customer c
  ON c.customer_id = t.customer_id
 AND t.txn_date BETWEEN c.valid_from AND c.valid_to;
Decide deliberatelyOverwriting (type 1) is fine for typos. Versioning (type 2) is required when history must be restatement-proof. Getting asked "why did last year's number change?" is what happens when you picked type 1 and should have picked type 2.
03

Comparing — the reconciliation toolkit

Three patterns for "these two things should match and don't" — the most valuable class of work you can do in a finance organisation.

10

Anti-join — what's in A but not B

You want this when: "which invoices didn't make it across", "which employees have no cost centre", "what's missing". The single fastest way to find a data problem.

-- readable and fast: no rows on the right
SELECT a.*
FROM source_system a
LEFT JOIN target_system b ON b.doc_id = a.doc_id
WHERE b.doc_id IS NULL;
NOT IN will hurt youWHERE id NOT IN (SELECT id FROM b) returns zero rows if that subquery contains a single NULL. It's silent, it's baffling, and it has burned everyone once. Use LEFT JOIN…IS NULL or NOT EXISTS instead.
11

Full reconciliation — both sides, with reasons

You want this when: two systems disagree and you need to explain the gap, not just measure it. This is the deliverable that ends arguments — and the one people remember you for.

The idea in words

FULL OUTER JOIN keeps rows that exist on either side, so you catch all three failure modes at once: missing on the left, missing on the right, and present on both but different. Then categorise every difference — that categorisation is the actual product.

SELECT
  COALESCE(a.doc_id, b.doc_id)          AS doc_id,
  a.amount                              AS source_amount,
  b.amount                              AS target_amount,
  COALESCE(a.amount,0) - COALESCE(b.amount,0) AS diff,
  CASE
    WHEN b.doc_id IS NULL            THEN 'Missing in target'
    WHEN a.doc_id IS NULL            THEN 'Missing in source'
    WHEN ABS(a.amount-b.amount) < 0.01 THEN 'Match'
    WHEN a.posted_date <> b.posted_date THEN 'Timing difference'
    ELSE 'Unexplained — investigate'
  END AS reason
FROM source_ledger a
FULL OUTER JOIN target_ledger b ON b.doc_id = a.doc_id;
Why this winsNobody expects zero difference. What they want is that every difference has a name, and the "Unexplained" bucket is small and shrinking. Report that bucket as your metric — it's honest, trackable, and it makes the work visible.
12

Cleaning keys before you join

You want this when: a join returns far fewer rows than expected and you can't see why. The answer is almost always invisible characters — and you'll lose an hour to this if you don't check it first.

-- the usual suspects, in order of how often they're the culprit
SELECT
  UPPER(TRIM(doc_id))                      AS clean_id,   -- case + spaces
  LEN(doc_id) - LEN(TRIM(doc_id))          AS pad_chars,  -- hidden padding
  TRY_CAST(amount_text AS decimal(18,2))     AS amount      -- NULL = bad rows
FROM raw_extract;

-- diagnose a bad join fast: how many keys actually overlap?
SELECT
  (SELECT COUNT(DISTINCT doc_id) FROM a)                        AS keys_a,
  (SELECT COUNT(DISTINCT doc_id) FROM b)                        AS keys_b,
  (SELECT COUNT(DISTINCT a.doc_id) FROM a JOIN b ON b.doc_id=a.doc_id) AS overlap;
The row-count explosionIf a join returns more rows than you started with, you have duplicates on one side — you've made a many-to-many by accident. Check the count before and after every join; when it jumps, go back to pattern 01.
04

When to reach for Python instead

You have a notebook. The useful question isn't "which is better" — it's "which one is this job shaped like?"

Use SQL whenUse Python when
Set-based work: joining, filtering, aggregating
The data's already in tables
Others will need to read and maintain it
The engine can optimise it better than you can
Looping over files, folders or API pages
Anything with 30+ columns to reshape
Calling an API, parsing JSON, reading Excel
Fuzzy matching, regex, or genuine algorithms
Anything you'd struggle to express as one set operation

The four Python moves worth having by heart

You don't need to learn Python properly to be effective in a notebook. These four cover most of what a data job actually needs, and the rest you can prompt your way through.

# 1. read many files as one table — the folder loop, done properly
df = spark.read.option("header",True).csv("Files/landing/2026/*.csv")

# 2. reshape wide to long — the unpivot from pattern 06, in one line
long = wide.melt(id_vars=["account"], var_name="period", value_name="amount")

# 3. sanity-check anything before you trust it
print(df.count(), len(df.columns))
df.groupBy("source_system").count().show()      # where did rows come from?
df.summary().show()                              # ranges, nulls, outliers

# 4. write it back as a table others can use
df.write.mode("overwrite").saveAsTable("silver_invoices")
The vibe coder's edgeYou don't have to know the syntax — you have to know that the pattern exists and what it's called. "Unpivot these month columns into rows" gets you working code from any assistant instantly. "Fix this spreadsheet" does not. The vocabulary is the skill, and that's what this page is for.

Mixing them in one notebook

You don't have to choose. In a Fabric notebook you can run SQL against your lakehouse and hand the result to Python, or register a dataframe so SQL can see it. Use each for what it's good at within a single job.

# SQL for the set-based part…
df = spark.sql("""
  SELECT customer_id, SUM(amount) AS total
  FROM silver_invoices
  WHERE posted_date >= '2026-01-01'
  GROUP BY customer_id
""")

# …Python for the part SQL is bad at
df.createOrReplaceTempView("totals")      # and back to SQL again
spark.sql("SELECT * FROM totals WHERE total > 100000").show()
05

The naming sheet

The actual deliverable of this page: symptom on the left, name on the right. When you can say the right-hand column, you can ask anyone — human or AI — for exactly what you need.

When you catch yourself saying…You want…
"There are duplicates" / "I only want the latest one"ROW_NUMBER + PARTITION BY (01)
"Versus last month" / "movement" / "change on prior"LAG (02)
"Year to date" / "rolling three months"Windowed SUM/AVG with a frame (03)
"Top 5 per region"DENSE_RANK (04)
"How many days in a row" / "find the missing numbers"Gaps and islands (05)
"There's a column for every month"Unpivot / melt (06)
"Empty months disappear from the chart"Date spine + LEFT JOIN (07)
"Group these into buckets"Banding, ideally via a lookup table (08)
"Last year's number changed and it shouldn't have"Type 2 slowly changing dimension (09)
"What's missing?"Anti-join (10)
"These two systems don't agree"FULL OUTER JOIN + reason categories (11)
"The join isn't matching and I can't see why"Key cleaning + overlap count (12)

Three habits that prevent most bugs

  • Count before and after every join. A row count that grows means accidental many-to-many; one that shrinks means a key problem. Thirty seconds, catches most silent errors.
  • Never trust a number you haven't seen at row level. Before you publish a total, look at ten of the rows behind it. It's astonishing how often that alone finds the problem.
  • Write the check query alongside the build query. If you can't write a query that would prove it's wrong, you don't yet understand what "right" means here.
← back to the tool belt