AI on your own terms: a deep dive into Foundry Local

We’ve grown accustomed to the idea that AI lives in the cloud; you send a prompt to a server, and you get an answer back.
But what if you could bring that power to your own machine?

I mostly talk about the shift toward modern architectures.
Today, we’re diving into a tool that embodies that shift: Microsoft Foundry Local.

For IT professionals and developers, this is one of the most interesting developments within the Microsoft ecosystem right now.
It allows us to run powerful AI models (like Phi-4, Llama, or Qwen) directly on our own hardware, without a single byte traveling to the cloud.

What is Microsoft Foundry Local?

Simply put: it is a local runtime for AI models.
Whereas you would normally make an API call to Microsoft Foundry, you are now running a lightweight service on your own Windows/Mac laptop or desktop.

The beauty of it is that ‘under the hood’ it utilizes your own GPU, NPU, or CPU, but on the front end, it offers an API that is 100% compatible with OpenAI.
This means your existing code or scripts can often work locally with just one adjustment (changing the base_url to localhost).

Why would you want this?

When we look at practical applications, there are specific scenarios where “Local” beats “Cloud.”
These are the use cases I am most enthusiastic about:

1. Privacy & Compliance (The “air-gapped” scenarios)

Imagine working with specific banking data or highly sensitive internal IP.
With Foundry Local, the data never leaves the device.
You can build a RAG (Retrieval-Augmented Generation) solution that searches and summarizes your local documents without having to worry about data sovereignty or cloud compliance regulations.

In my professional life, I create solutions that are data sovereign and are compliant with every law, but that doesn’t change that there are still solutions with data that normally don’t leave devices for this specific reason.

2. Low latency & offline use

Think of Field Service technicians visiting locations with poor connectivity.
With a local implementation on their Surface or laptop, they can still use an AI assistant to search through manuals or generate incident reports, even in a basement without 5G.
Additionally, the latency is often lower because there is no network roundtrip; the speed is determined purely by your own hardware.

How do you get started?

Installation is surprisingly “modern” and simple via the command line (winget):

winget install Microsoft.FoundryLocal
foundry model list
foundry model run qwen2.5-coder-1.5b-instruct-generic-cpu:4

Next, you can connect directly to your local model using the AI Toolkit in VS Code.
It feels like magic to see a full language model running on the machine right in front of you, completely disconnected from the internet.

Conclusion: the future is hybrid

Does this mean the end of Microsoft Foundry in the cloud? Certainly not.
For heavy models and massive scalability, the cloud remains king.
But for edge cases, privacy, and fast dev loops, local is the way forward.

To me, Microsoft Foundry Local is proof that as we head into 2026, we are growing into a Hybrid AI world: smart endpoints at the edge, and brute computing power at the core.

Have you experimented with local models yet?