Tool-belt series · SQL · Python · Vocabulary
Every real data problem is a recombination of about a dozen patterns. This page names each one in plain English first — what you're trying to do and how to recognise it — then gives you the SQL, then the Python where Python is genuinely better. The point isn't to memorise syntax. It's vocabulary: you can't ask an AI, a colleague, or a search box for something you can't name.
Don't read it end to end. Skim the twelve headings once so the names land, then come back when you hit a problem and think "this is the one where…". Naming the pattern is 80% of solving it — once you know you need a window function, the syntax is a two-minute lookup or one prompt away.
The SQL is written to run on Fabric's SQL endpoint, a Warehouse, or Snowflake with minimal changes. Where a dialect differs meaningfully, it says so.
If you learn one thing on this page, learn these. A window function looks at other rows while staying on the current row — that's the whole idea, and it's what separates people who write awkward self-joins from people who write four clean lines. Five of the twelve patterns below are just window functions wearing different hats.
You want this when: an extract has the same record more than once and you need exactly one — usually the most recent. This is the most-used pattern in this entire page.
The idea in wordsNumber the rows within each group, ordered so the one you want is number 1. Then keep number 1. That's it — and once you see it, you'll use it constantly.
WITH ranked AS (
SELECT *,
ROW_NUMBER() OVER (
PARTITION BY customer_id -- one per customer
ORDER BY updated_at DESC -- newest wins
) AS rn
FROM raw_customer
)
SELECT * FROM ranked WHERE rn = 1;
updated_at, which one wins is arbitrary and can change between runs — producing a table that differs each load for no visible reason. Add a tiebreaker to the ORDER BY (a surrogate key, a load timestamp) whenever the result has to be stable.You want this when: anyone says "versus last month", "change on prior", "movement", or "delta". Which in a finance org is roughly every second request.
The idea in wordsReach backwards a fixed number of rows and pull that value onto the current row, so you can subtract them side by side. LAG looks back; LEAD looks forward.
SELECT
period,
cost_centre,
amount,
LAG(amount) OVER (PARTITION BY cost_centre ORDER BY period) AS prior_amount,
amount - LAG(amount) OVER (PARTITION BY cost_centre ORDER BY period) AS movement
FROM monthly_actuals;
You want this when: "cumulative year to date", "3-month rolling average", "balance over time". Anything that accumulates.
The idea in wordsSame window, but instead of one specific row you define a range of rows to aggregate over — "everything up to here", or "these three".
SELECT
period, amount,
SUM(amount) OVER (
ORDER BY period
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW -- running total
) AS ytd,
AVG(amount) OVER (
ORDER BY period
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW -- 3-month rolling
) AS rolling_3m
FROM monthly_actuals;
You want this when: "top 5 customers per region", "biggest three variances per department". Note this is pattern 01 with a different filter — same machinery.
WITH r AS (
SELECT region, customer, revenue,
DENSE_RANK() OVER (PARTITION BY region ORDER BY revenue DESC) AS rk
FROM sales
)
SELECT * FROM r WHERE rk <= 5;
ROW_NUMBER never ties (1,2,3,4). RANK ties then skips (1,1,3). DENSE_RANK ties without skipping (1,1,2). For "top 5" you almost always want DENSE_RANK, otherwise a tie silently costs you a row.You want this when: "how many days in a row was this breached", "find the missing invoice numbers", "how long was each outage". Sounds exotic, comes up constantly once you can name it.
The idea in wordsThe trick that makes it click: if you number the rows sequentially and subtract that from the date, every row in a consecutive run gets the same answer. Group by that answer and you've found your runs. It's a genuinely clever trick and it's worth sitting with for a minute.
WITH g AS (
SELECT breach_date,
DATEADD(day,
-ROW_NUMBER() OVER (ORDER BY breach_date),
breach_date) AS grp -- constant within a run
FROM daily_breaches
)
SELECT MIN(breach_date) AS run_start,
MAX(breach_date) AS run_end,
COUNT(*) AS days_in_run
FROM g GROUP BY grp
ORDER BY run_start;
Most "the data's wrong" problems are actually "the data's the wrong shape" problems. These four fix that.
You want this when: a finance extract has a column per month (Jan, Feb, Mar…). Humans love this layout; every analytical tool hates it. Almost every spreadsheet-sourced dataset needs this first.
Turn twelve columns into two: one saying which month, one saying the value. Long and boring beats wide and clever — every time, for anything downstream.
-- portable version: works everywhere
SELECT account, 'Jan' AS period, jan AS amount FROM budget
UNION ALL SELECT account, 'Feb', feb FROM budget
UNION ALL SELECT account, 'Mar', mar FROM budget;
-- T-SQL has a built-in for it
SELECT account, period, amount
FROM budget
UNPIVOT (amount FOR period IN (jan, feb, mar)) AS u;
df.melt(id_vars=['account'], var_name='period', value_name='amount'). This is exactly the kind of job where a notebook beats SQL.You want this when: a month with no activity vanishes from your report instead of showing zero, so charts have gaps and prior-period comparisons quietly lie (see pattern 02).
The idea in wordsBuild a table of every date you care about, then LEFT JOIN your data onto it. Now absence is visible as a zero rather than invisible as a missing row — which is nearly always what people actually meant.
WITH spine AS (
SELECT CAST('2024-01-01' AS date) AS d
UNION ALL
SELECT DATEADD(month, 1, d) FROM spine WHERE d < '2026-12-01'
)
SELECT s.d AS period, COALESCE(a.amount, 0) AS amount
FROM spine s
LEFT JOIN monthly_actuals a ON a.period = s.d
OPTION (MAXRECURSION 0);
You want this when: "group these into small / medium / large", ageing buckets, materiality tiers. Any time a continuous number needs to become something you can group by.
SELECT invoice_id, days_overdue,
CASE
WHEN days_overdue <= 0 THEN 'Current'
WHEN days_overdue <= 30 THEN '1-30'
WHEN days_overdue <= 60 THEN '31-60'
ELSE '60+'
END AS ageing_bucket
FROM receivables;
You want this when: a customer moves region, a cost centre is renamed, an account is remapped — and last year's report must still show what was true then. This is the pattern that makes you sound senior, because most people never learn it.
The idea in wordsInstead of overwriting the row, close the old one off with an end date and insert a new one. Each row becomes "this was true between these dates", and you join on the date to get the version that applied at the time.
-- one row per version, not per entity
customer_key | customer_id | region | valid_from | valid_to | is_current
1 | C100 | North | 2024-01-01 | 2025-06-30 | 0
2 | C100 | South | 2025-07-01 | 9999-12-31 | 1
-- "what was true when the transaction happened"
SELECT t.*, c.region
FROM transactions t
JOIN dim_customer c
ON c.customer_id = t.customer_id
AND t.txn_date BETWEEN c.valid_from AND c.valid_to;
Three patterns for "these two things should match and don't" — the most valuable class of work you can do in a finance organisation.
You want this when: "which invoices didn't make it across", "which employees have no cost centre", "what's missing". The single fastest way to find a data problem.
-- readable and fast: no rows on the right
SELECT a.*
FROM source_system a
LEFT JOIN target_system b ON b.doc_id = a.doc_id
WHERE b.doc_id IS NULL;
WHERE id NOT IN (SELECT id FROM b) returns zero rows if that subquery contains a single NULL. It's silent, it's baffling, and it has burned everyone once. Use LEFT JOIN…IS NULL or NOT EXISTS instead.You want this when: two systems disagree and you need to explain the gap, not just measure it. This is the deliverable that ends arguments — and the one people remember you for.
The idea in wordsFULL OUTER JOIN keeps rows that exist on either side, so you catch all three failure modes at once: missing on the left, missing on the right, and present on both but different. Then categorise every difference — that categorisation is the actual product.
SELECT
COALESCE(a.doc_id, b.doc_id) AS doc_id,
a.amount AS source_amount,
b.amount AS target_amount,
COALESCE(a.amount,0) - COALESCE(b.amount,0) AS diff,
CASE
WHEN b.doc_id IS NULL THEN 'Missing in target'
WHEN a.doc_id IS NULL THEN 'Missing in source'
WHEN ABS(a.amount-b.amount) < 0.01 THEN 'Match'
WHEN a.posted_date <> b.posted_date THEN 'Timing difference'
ELSE 'Unexplained — investigate'
END AS reason
FROM source_ledger a
FULL OUTER JOIN target_ledger b ON b.doc_id = a.doc_id;
You want this when: a join returns far fewer rows than expected and you can't see why. The answer is almost always invisible characters — and you'll lose an hour to this if you don't check it first.
-- the usual suspects, in order of how often they're the culprit
SELECT
UPPER(TRIM(doc_id)) AS clean_id, -- case + spaces
LEN(doc_id) - LEN(TRIM(doc_id)) AS pad_chars, -- hidden padding
TRY_CAST(amount_text AS decimal(18,2)) AS amount -- NULL = bad rows
FROM raw_extract;
-- diagnose a bad join fast: how many keys actually overlap?
SELECT
(SELECT COUNT(DISTINCT doc_id) FROM a) AS keys_a,
(SELECT COUNT(DISTINCT doc_id) FROM b) AS keys_b,
(SELECT COUNT(DISTINCT a.doc_id) FROM a JOIN b ON b.doc_id=a.doc_id) AS overlap;
You have a notebook. The useful question isn't "which is better" — it's "which one is this job shaped like?"
| Use SQL when | Use Python when |
|---|---|
| Set-based work: joining, filtering, aggregating The data's already in tables Others will need to read and maintain it The engine can optimise it better than you can |
Looping over files, folders or API pages Anything with 30+ columns to reshape Calling an API, parsing JSON, reading Excel Fuzzy matching, regex, or genuine algorithms Anything you'd struggle to express as one set operation |
You don't need to learn Python properly to be effective in a notebook. These four cover most of what a data job actually needs, and the rest you can prompt your way through.
# 1. read many files as one table — the folder loop, done properly
df = spark.read.option("header",True).csv("Files/landing/2026/*.csv")
# 2. reshape wide to long — the unpivot from pattern 06, in one line
long = wide.melt(id_vars=["account"], var_name="period", value_name="amount")
# 3. sanity-check anything before you trust it
print(df.count(), len(df.columns))
df.groupBy("source_system").count().show() # where did rows come from?
df.summary().show() # ranges, nulls, outliers
# 4. write it back as a table others can use
df.write.mode("overwrite").saveAsTable("silver_invoices")
You don't have to choose. In a Fabric notebook you can run SQL against your lakehouse and hand the result to Python, or register a dataframe so SQL can see it. Use each for what it's good at within a single job.
# SQL for the set-based part…
df = spark.sql("""
SELECT customer_id, SUM(amount) AS total
FROM silver_invoices
WHERE posted_date >= '2026-01-01'
GROUP BY customer_id
""")
# …Python for the part SQL is bad at
df.createOrReplaceTempView("totals") # and back to SQL again
spark.sql("SELECT * FROM totals WHERE total > 100000").show()
The actual deliverable of this page: symptom on the left, name on the right. When you can say the right-hand column, you can ask anyone — human or AI — for exactly what you need.
| When you catch yourself saying… | You want… |
|---|---|
| "There are duplicates" / "I only want the latest one" | ROW_NUMBER + PARTITION BY (01) |
| "Versus last month" / "movement" / "change on prior" | LAG (02) |
| "Year to date" / "rolling three months" | Windowed SUM/AVG with a frame (03) |
| "Top 5 per region" | DENSE_RANK (04) |
| "How many days in a row" / "find the missing numbers" | Gaps and islands (05) |
| "There's a column for every month" | Unpivot / melt (06) |
| "Empty months disappear from the chart" | Date spine + LEFT JOIN (07) |
| "Group these into buckets" | Banding, ideally via a lookup table (08) |
| "Last year's number changed and it shouldn't have" | Type 2 slowly changing dimension (09) |
| "What's missing?" | Anti-join (10) |
| "These two systems don't agree" | FULL OUTER JOIN + reason categories (11) |
| "The join isn't matching and I can't see why" | Key cleaning + overlap count (12) |