Tool-belt series · Microsoft Fabric · Data engineering
Ten minutes a day. One idea, one thing you do in your own tenant, one question to ask about your organisation. By Day 7 you can defend an architecture in a meeting — not just build a pipeline. Written for someone who has already put a Delta table in a lakehouse from a notebook and wired a workspace to Git, and wants the layer above that.
One day per sitting, in order — each is 5–10 minutes. Every day has the same three parts: the idea (why this exists and when it's the wrong choice), do this (something small in your own tenant — that's where it sticks), and for your org (a question to bring to work). Tick items as you go; progress and collapsed days save in this browser on this device.
You need a Fabric-enabled workspace on any capacity, including a trial. Everything here works on an F2 or a trial SKU — nothing requires a big capacity. Where a feature is preview or still rolling out, it says so.
You already did the thing this whole platform is built around, and it probably felt routine: a notebook wrote a Delta table into a lakehouse. Here's what actually happened. That table landed in OneLake — one storage account for the entire tenant — as open Delta/Parquet files. The moment it landed, it became queryable by every other engine in Fabric without being copied: T-SQL through the lakehouse's SQL analytics endpoint, DAX through a Direct Lake semantic model, Spark from any other notebook, and external tools through the OneLake APIs.
That is the whole pitch. Traditional stacks copy data between a lake, a warehouse, and a BI model — three copies, three refresh windows, three chances to disagree. Fabric separates storage (OneLake, one copy, open format) from compute (Spark, T-SQL, KQL, DAX, Power Query). Every Fabric item you see in the portal is one of those engines pointed at that store, or something that orchestrates them.
The item zoo, grouped so it stops being noise
| Group | Items | What it's for |
|---|---|---|
| Store | Lakehouse, Warehouse, Eventhouse, SQL database, Mirrored database | Where tables live. Differ by engine and by who's meant to write to them. |
| Move | Data pipeline, Dataflow Gen2, Copy job, Eventstream, Shortcut, Mirroring | Getting data in — or deliberately not moving it (Day 2). |
| Transform | Notebook, Spark job definition, Stored procedure, Materialized lake view | Turning raw into usable. |
| Serve | Semantic model, Report, SQL endpoint, GraphQL API, Data agent, Fabric App | How humans and systems actually consume it. |
| React | Activator, Alerts | Doing something when a number crosses a line. |
| Govern | Domains, Workspaces, Deployment pipelines, Variable libraries, OneLake catalog | Keeping it findable, shippable and trustworthy (Day 6). |
Do this — 4 minutes
Most people's instinct is "write a pipeline to copy it in." In Fabric that should be your third instinct. The first question is always: do I need a copy at all? Every copy you avoid is a schedule you don't maintain, a latency you don't explain, and storage you don't pay for.
The ladder — work down it, stop at the first one that fits
| Option | What it does | Reach for it when |
|---|---|---|
| Shortcut | A pointer. Data stays where it is (another lakehouse, ADLS Gen2, S3, Google Cloud Storage, Dataverse) and appears in your lakehouse as if local. | The data already sits in a lake somewhere and you just need to read it. Zero copy, zero schedule. |
| Mirroring | Near-real-time managed replication of an operational database (Azure SQL, SQL Server, Cosmos DB, Snowflake, Databricks catalog, PostgreSQL) into OneLake as Delta. | You need an operational system's data continuously, without building or owning CDC yourself. |
| Copy job | A guided copy with batch or incremental modes, no pipeline authoring. | Straight source-to-destination movement where a full pipeline is overkill. |
| Data pipeline | Orchestration: copy activities, notebook/stored-proc invocation, control flow, parameters, scheduling, failure handling. | You need orchestration — several steps in order, with dependencies and retries. |
| Dataflow Gen2 | Power Query in the service. Now on a modern .NET-based evaluation engine with far better performance across 80+ connectors. | Messy connector-shaped sources and Power Query is genuinely the fastest way to shape it — or the author lives in Power Query, not Python. |
| Eventstream | Streaming ingestion into Eventhouse/lakehouse from event sources. | The data arrives continuously and lateness matters in seconds, not hours. |
A rule that holds up in review meetings: shortcut what you can, mirror what you must, pipeline what you have to transform, and only use a dataflow when Power Query is genuinely the right tool. "Notebook everything" is as wrong as "dataflow everything" — the skill is matching the tool to the shape of the source and to who has to maintain it after you.
Do this — 5 minutes
Four things in Fabric hold tables. They are not interchangeable, and picking correctly is most of what "architecture" means day to day.
| Store | Write with | Choose it when |
|---|---|---|
| Lakehouse | Spark, pipelines, dataflows | Files and tables, Python/Spark transformation, semi-structured or large data. Its SQL endpoint is read-only. |
| Warehouse | T-SQL (full DML) | The team is SQL-first, or you need multi-table transactions and stored procedures. Full read/write T-SQL over the same OneLake Delta. |
| Eventhouse / KQL | Eventstream, KQL | High-volume time-series, logs, telemetry — append-heavy data queried by time window. |
| SQL database | T-SQL, apps | You need genuine OLTP — an application writing rows. It auto-replicates into OneLake for analytics. |
The honest heuristic: Lakehouse if your team writes Python, Warehouse if your team writes SQL, Eventhouse if the data is events, SQL database if an application needs to write. All four end up as Delta in OneLake, so the choice is about the authoring experience and the write pattern — not about where the bytes go.
What Delta actually is — the 90-second version
A Delta table is a folder of Parquet files plus a transaction log (_delta_log) recording every commit. That log is what gives you ACID transactions, time travel, and safe concurrent writes on plain cloud storage. Consequences you actually feel:
OPTIMIZE compacts them.VACUUM removes them — which also sets how far back you can look.TIMESTAMP clause — query the table as it looked before last night's load.# in your notebook — run these on the table you already built
spark.sql("DESCRIBE HISTORY your_table").show(10, truncate=False) # every commit
spark.sql("OPTIMIZE your_table") # compact small files
spark.sql("DESCRIBE DETAIL your_table").show(truncate=False) # file count & size
# read it as it was 3 versions ago
df = spark.read.format("delta").option("versionAsOf", 3).table("your_table")
Do this — 5 minutes
Your notebook works. The gap between that and a notebook that runs unattended at 3am is four things: where its dependencies come from, how it gets its inputs, how it loads data the second time, and what it costs.
1. Environments — stop installing libraries in cell one
An Environment is a Fabric item holding your Spark runtime version, libraries, and Spark configuration, attachable to notebooks and Spark job definitions. It's how you stop %pip install at the top of every notebook and get reproducible runs. It's also the honest answer to "why did it work yesterday?"
2. Parameters — the toggle cell that makes it callable
Mark a cell as the parameter cell (cell menu → Toggle parameter cell). Values declared there are overridden when a pipeline or another notebook calls it. This is the single change that turns a script into a component.
# parameter cell — defaults for interactive runs, overridden by the caller
load_date = "2026-08-01"
source_system = "sap"
full_reload = False
# elsewhere: chain notebooks and read pipeline values
import notebookutils
notebookutils.notebook.run("transform_silver", 300, {"load_date": load_date})
notebookutils.notebook.exit("success") # returned to the calling pipeline
3. Incremental loads — MERGE, not overwrite
The move from beginner to practitioner is loading changes rather than rewriting the table. Delta's MERGE does upserts in one atomic commit:
from delta.tables import DeltaTable
target = DeltaTable.forName(spark, "silver_customer")
(target.alias("t")
.merge(updates.alias("s"), "t.customer_id = s.customer_id")
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute())
Add .option("mergeSchema", "true") on writes when upstream columns legitimately change — but know that you've just made schema drift silent. Most teams want it on in bronze and off in gold.
4. Cost — because someone will ask
Everything in Fabric consumes Capacity Units from one shared pool: Spark, SQL, pipelines, Power BI, all of it. Spark sessions are the classic silent burner — an idle attached session still holds resources. Autoscale billing for Spark lets Spark jobs bill serverlessly rather than competing for the capacity, which is worth knowing exists when a Spark job starts throttling everyone's reports.
Do this — 6 minutes
Medallion architecture is three layers, and the value is entirely in the discipline of not skipping one:
Why star schema, still, in 2026? Because VertiPaq — the engine behind every Power BI semantic model — is built for it. One wide flat table looks simpler and performs worse: it compresses badly, it makes measures ambiguous, and it makes every filter scan more data. If your organisation is on Power BI, gold being a star schema is not a stylistic preference, it's the difference between a report that returns instantly and one that doesn't.
Materialized lake views — the layer that automates the layers
Rather than hand-orchestrating bronze→silver→gold refreshes, materialized lake views let you declare the transformation and let Fabric manage materialisation and lineage. As of mid-2026 lineage extends across layers and workspaces: define one refresh schedule on your gold lakehouse and it cascades back through silver and bronze automatically. That's a meaningful reduction in pipeline plumbing — and a very current thing to be able to name.
Direct Lake — and the trap that will bite you
A Direct Lake semantic model reads Delta files from OneLake straight into memory. No import refresh, no DirectQuery round-trip — import-like speed on data that's current as of the last commit. It's the reason gold-layer modelling matters so much.
Knowing that one paragraph puts you ahead of most people who use Fabric daily. It is the most common invisible performance regression in the platform.
Do this — 6 minutes
You've connected a workspace to GitHub, which is further than most Fabric users ever get. Here's the shape of the full lifecycle, and specifically what Git doesn't solve.
The three pieces
What Git does not capture — the part people learn painfully
For automation beyond the UI there's the Fabric REST API, the Fabric CLI, and the fabric-cicd Python library for deploying items from a repo in a build pipeline. You don't need these on day one, but knowing they exist is what separates "we click Deploy" from "we have a release process."
Do this — 6 minutes
A pipeline nobody consumes is a cost centre. Fabric has five distinct ways to serve what you've built, and knowing all five means you can answer "how would people use this?" before someone asks.
Governance, briefly, because it's what makes any of it trustworthy
Do this — 6 minutes
Everything above, compressed to what you'd actually say out loud in a design review. Screenshot this one.
"Where should this data live?"
| Signal | Answer |
|---|---|
| Team writes Python / Spark; files as well as tables | Lakehouse |
| Team writes SQL; needs INSERT/UPDATE and stored procs | Warehouse |
| Events, logs, telemetry; queried by time window | Eventhouse / KQL |
| An application writes single rows (OLTP) | SQL database |
"How do we get it in?" — stop at the first yes
| Ask | Then use |
|---|---|
| Is it already in a lake we can reach? | Shortcut — no copy, no schedule |
| Is it an operational DB we need continuously? | Mirroring |
| Is it a straight A→B move? | Copy job |
| Are there several steps with dependencies? | Data pipeline |
| Is Power Query genuinely the right shaping tool? | Dataflow Gen2 |
| Does lateness matter in seconds? | Eventstream |
The five things that will bite you
| Trap | What happens | Fix |
|---|---|---|
| A view in a Direct Lake model | The whole model silently falls back to DirectQuery — everything gets slower | Materialise the view as a Delta table |
| Lakehouse SQL endpoint is read-only | The "we'll write back through SQL" plan dies late | Use a Warehouse if you need writes |
| Shortcuts skip deployment pipelines | Promotion to test/prod quietly loses them | Recreate per workspace, or automate via API |
| Small files | Thousands of tiny Parquet files make reads crawl | OPTIMIZE; don't partition small tables |
| Idle Spark sessions | Burn CUs and throttle other people's reports | Stop sessions; know about autoscale billing |
Three sentences that make you sound like you've done this before
Seven days gets you the map. These turn it into the reputation you're actually after.
Pick any real dataset your team cares about — small is fine. Build it properly, once. Doing this end to end teaches more than another twenty hours of reading, and it gives you something to show.
You don't want to pay for a certification — fine. The genuinely valuable part of DP-700 (Fabric Data Engineer Associate) is free anyway: the skills-measured outline in the study guide is a published syllabus of everything Microsoft thinks a Fabric data engineer should know, and every linked Learn module is free. Read the outline, tick off what you can already do, and let the gaps set your next month.
DP-700 covers three roughly equal areas: implementing and managing an analytics solution (workspaces, security, lifecycle), ingesting and transforming data (batch and streaming, SQL/PySpark/KQL), and monitoring and optimising. If Power BI modelling is more your angle, DP-600 covers the analytics-engineer side — semantic models, DAX, Direct Lake — and the same trick applies.