Building a local-first AI ops platform on a prize GPU (ROCm 10, 30B coder, multi-agent)
I stopped treating local LLMs as a side project and started routing real product work through them.
I'm building InnerOS / InnerOps, a local-first control plane for running development and operational work through AI agents without sending every task to cloud APIs.
The basic idea is simple:
- portal + commercial/workforce modules
- MCP-based orchestration
- Cursor, Codex and chat agents as interchangeable workers
- automatic routing to the machine/model that best fits the task
- cloud models only when they add something the local stack cannot
Why I pushed this now
I received a Radeon AI PRO R9700 (gfx1201, 32 GB VRAM) after finishing 10th in the AMD Dev Quest at AMD Advancing AI in San Francisco on July 22–23.
That was the push I needed to stop treating “local inference” as a weekend experiment and make it part of the development loop.
The stack I actually use
AMD node
- Radeon AI PRO R9700
- official ROCm 10 vLLM container
- PyTorch 2.12.0+rocm10.0.0
- vLLM 0.27.x
- Qwen3-Coder-30B-A3B-Instruct-AWQ on port 8000
- systemd-managed so it survives reboots
Intel node
- Ollama for lighter chat/classification work
- smaller Qwen/Phi-class models
Routing
- coding / refactor / review → 30B model on the AMD GPU
- lighter ops / classification / chat → smaller local models
Supporting services
- Lemonade Server for local TTS/STT on CPU so voice workloads do not eat the GPU’s VRAM
- MCP server + agent inbox
- Dev Swarm for heavier implementation work
- IDE task bridge for Cursor/Codex-style workers
What changed this month
The biggest improvement was not raw tokens/sec.
My previous vLLM container was named as if it were ROCm 10. When I inspected the actual runtime, it was still ROCm 7.14, PyTorch 2.11 and vLLM 0.23.x.
That was a useful lesson:
Container names are not evidence. torch.__version__ is.
I cut over to the official ROCm 10 stack and made the serving layer persistent.
Since then I have also:
- documented rollback
- wired heavy coding tasks to the local AMD endpoint
- verified the Codex adapter path
- expanded the A2A agent registry to 55 functional agents plus control/execution cards
- added MCP-based ops updates through Discord
- started measuring whether agent automation is actually returning human time instead of merely generating activity
Before vs now
| | Old path | Current |
| ---------------- | ------------------------------------ | --------------------------------------------------- |
| Stack | ROCm 7.14 / vLLM 0.23 / PyTorch 2.11 | ROCm 10 / vLLM 0.27 / PyTorch 2.12 |
| Operations | manual Docker | systemd-managed |
| Smoke throughput | ~25–40 tok/s mixed | ~19–41 tok/s prompt-dependent |
| Main gain | local inference worked | local inference is part of the development workflow |
The throughput did not magically double.
The win is reliability, routing and utilization.
The GPU is now doing useful work for hours instead of appearing in benchmark screenshots once a week.
What is still hard
32 GB means one large model at a time.
VRAM budgeting is part of the architecture.
The infrastructure is also ahead of the visible product. The browser portal and the real Workforce product integration still need work.
And multi-IDE coordination is messy. Durable inboxes, ops tasks and evidence help, but this is still early software rather than a polished “autonomous company” fantasy.
Business angle
The initial market is SMB operations in Ecuador/LATAM: workforce, quotes, WhatsApp operations and internal company workflows.
The bet is that local-first AI can reduce recurring model spend and keep more sensitive operational work on owned infrastructure, while still using Gemini or other cloud models for specialized lanes when there is a real reason to pay for them.
One thing I’m increasingly convinced of:
The interesting benchmark is not tokens per second. It is how many hours per day the hardware becomes useful enough that you trust it with real work.
The InnerOS repository is currently private, so I’m not using the GitHub link as a public proof point yet.
I’m building in public from the server rack upward and will keep sharing the parts that are actually verified.
If you’re running vLLM on consumer AMD hardware at roughly 30B scale, I’m especially interested in comparing notes on VRAM budgeting, AWQ/MoE behavior and long-running agent workloads.
Building a local-first AI ops platform sounds like a fascinating endeavor, especially with the growing need for efficient data processing on-premises. In my journey with a project that integrates AI into content strategies, I've experienced some common roadblocks that might resonate with your work.
When developing a system that relies on local computation, one challenge you may face is ensuring smooth performance without the vast resources cloud solutions can offer. I've dealt with this by focusing on optimizing algorithms for local execution. For instance, by leveraging lighter models and more efficient data handling, we managed to reduce inference times significantly, which is crucial for real-time applications. In our case, reducing latency improved user engagement metrics by nearly 30%—important when processing large amounts of content.
Another consideration is user onboarding and support. With so many tools relying on cloud environments, getting users comfortable with local installs can be a hurdle. Providing clear documentation and engaging with users through a forum or community channel has proven invaluable. We saw a significant reduction in support tickets when we created concise setup guides and shared user-generated tips.
Lastly, don’t underestimate the value of benchmarking performance. When we first launched, we made the mistake of not adequately testing our tools against various workloads. Once we implemented consistent benchmarking, we gathered actionable insights that led to continuous improvements in performance and user satisfaction.
As you develop InnerOS / InnerOps, keeping these points in mind could enhance both your product and user experiences. Good luck with your build!
Thanks Brenda, this resonates a lot with what we’re seeing.
Performance has probably been one of the biggest lessons for us. InnerOS currently runs a 30B-class coding/reasoning model locally on our own AMD infrastructure using ROCm and vLLM. One thing we learned quickly is that “local-first” cannot mean “send everything to the biggest model.” We’re moving toward routing tasks by complexity: deterministic code and smaller/local models for routine operations, and the larger model only when deeper reasoning is actually needed.
I completely agree on onboarding as well. Our goal is that users should not need to understand ROCm, GPUs, vLLM, containers, or model configuration. The local infrastructure should eventually feel like an appliance or managed runtime, while the user interacts through a normal web interface.
Benchmarking is also becoming a core part of InnerOS rather than an afterthought. We’re tracking things such as inference latency, VRAM usage, concurrency, queue behavior, model availability, task completion, and when a workload needs to fall back to another resource.
The interesting part for us is that local-first is becoming less about “no cloud ever” and more about owning the intelligence layer: deciding what stays local, what can leave the environment, which model handles each task, and how much external compute is actually necessary.
Your point about benchmarking across different workloads is especially relevant. That’s one area we want to make much more visible inside InnerOS itself.
I like that framing local-first isn’t necessarily about rejecting the cloud, but about controlling where each workload runs and why. Making those routing decisions visible could also help users understand the real cost, privacy, and performance tradeoffs without needing to understand the underlying infrastructure.
How are you deciding when a task should remain local versus fall back to a cloud model fixed rules, live performance metrics, or an agent evaluating each task?
We’re moving toward a hybrid approach rather than one fixed rule or letting an agent make an unconstrained routing decision.
Local is the default. Hard policy comes first: privacy, data locality, allowed providers, cost limits, and whether a task is permitted to leave the local environment at all.
After that, routing can consider things like capability, context requirements, latency, current availability and eventually observed performance for that type of task. Cloud is a fallback when it genuinely gives the workflow an advantage that the local stack can’t provide.
The part I’m especially interested in is making the routing decision itself observable.
Instead of only recording “local model” or “cloud model,” I want the trace to answer: why was it routed there, what policy was active, how long did it take, what did it cost, and how much human intervention did the overall workflow still require?
That last metric is becoming more important to me than model speed itself. We’re starting to measure it as Human Time Returned.
The architecture only matters if it gives meaningful time back to the person using it.
“Human Time Returned” is a powerful metric because it measures the value of the entire workflow, not just the model’s speed. It could reveal cases where a slower local model delivers more value by reducing cost, protecting sensitive data, and requiring less human correction.
How are you planning to measure Human Time Returned by comparing the automated workflow with the time the same task previously took manually?
This comment was deleted 3 days ago
The shift from measuring tokens/sec to measuring human time returned is the interesting part.
Curious whether the local-first setup is already producing a measurable workflow advantage, or if that’s still mostly an infrastructure hypothesis.
It’s already more than an infrastructure hypothesis.
We’re still early and the sample size is not large enough for me to claim a universal productivity number yet, but we have started measuring actual “human time returned.”
One workflow we tracked had an estimated manual baseline of about 120 minutes and was completed with roughly 10 minutes of human involvement using InnerOS. That’s about 110 minutes returned, or roughly a 12x speedup for that specific task.
More broadly, over the last week our system recorded hundreds of agent executions, including hundreds of coding tasks handled on our own local vLLM infrastructure instead of being manually executed step by step or routed to external models.
The bigger advantage we’re seeing is not raw tokens/sec. It’s parallelism and reduced supervision: agents can work, test, inspect services, coordinate tasks, and keep context while I move on to something else.
So yes, there is already measurable workflow advantage. What we don’t have yet is enough longitudinal data to responsibly say something like “InnerOS returns X hours per employee per week.”
That is exactly the KPI we’re now building around: human baseline time, assisted time, rework, handoffs, autonomous completion rate, and ultimately human hours returned.
I think that metric will matter much more than model speed.
The real benchmark is not how fast the model answers. It’s how long the human is no longer waiting, supervising, or repeating work.
That’s a much stronger signal than the infrastructure story — especially the 120→10 minute workflow. I’d be interested in digging into whether that time returned holds across different task types and rework levels. Happy to continue privately — what’s the best email to reach you on?
Thanks Aryan. I’d be very happy to continue the conversation privately.
The 120→10 minute example came from a real internal workflow, and we’re now instrumenting the same measurements across different task classes, rework levels and degrees of human intervention so we can see where the gains actually hold — and where they don’t.
You can reach me at rlopez@innerchispa.us.
Would be great to compare notes.
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.