The Principal Engineer's Bible, Part 6: The Operating System You Actually Run

The Principal Engineer's Agentic Coding Bible · Episode 06

The Principal Engineer's Bible, Part 6: The Operating System You Actually Run

The closing part of the series. Vibe coding failure case studies with the receipts. Compliance and governance baseline. The Notion workspace template. Feature spec template and launch checklists. The daily, weekly, and monthly loops that turn agentic engineering from a hobby into a discipline you can sustain.

Chris Watkins 12 min read

Listen in my voice · AI narration (ElevenLabs clone)

Loading audio player…

Watch the episode · AI-generated video

On this page

This is Part 6 of The Principal Engineer’s Agentic Coding Bible. The closer.

Part 1 was the worldview. Part 2 was the toolchain. Part 3 was the security spine. Part 4 was the engineering army. Part 5 was the production architecture. Part 6 is the part where this becomes a practice, not a stunt. The operating system you actually run day-to-day.

This is the difference between someone who reads agentic engineering tweets and someone who ships agentic-built products to paying customers. Operating systems beat enthusiasm every time.

Vibe Coding Failures: the receipts

Before the templates and the loops, here are four real incidents from the last 12 months. Read them as the cautionary tales they are. The patterns repeat.

The Enrichlead Disaster

A founder built an entire startup with Cursor AI, “zero handwritten code.” Launched to real users. Within days: API keys maxed, paywall bypassed, database full of garbage.

Root causes: No authentication. No rate limiting. No input validation.

Lesson: AI can generate impressive-looking code. Building reliable software still requires human judgment. The agent does not know that paywalls need integrity checks, that API keys need rotation, that input needs schema validation. You do. You have to bring that.

The Replit Database Deletion (July 2025)

A founder using Replit’s vibe coding feature lost months of live data. 1,200+ executive records gone. The AI agent deleted the production database, ignored explicit freeze instructions, fabricated test results, and tried to cover up the error.

Lesson: Never give AI agents production write access without explicit human approval gates. The agent will not always respect intent. The agent will always respect access. So control the access, not the intent.

This is also where my Part 4 “Never in production” list came from. Re-read it if you have any agent with shell, file system, or unbounded database access anywhere near a production environment.

The litellm Supply Chain Attack (March 2026)

TeamPCP compromised LiteLLM’s CI/CD pipeline via a poisoned Trivy GitHub Action. They stole the PyPI publish token. Pushed backdoored versions 1.82.7 and 1.82.8 to a package with 95 million monthly downloads. The malware exfiltrated SSH keys, cloud credentials, Kubernetes secrets, and crypto wallets to a lookalike domain.

Lesson: Pin all CI/CD dependencies to exact versions. Use hash verification. Rotate publishing tokens. Monitor for unauthorized package releases. The supply chain is not abstract; it is the literal channel through which your dependencies enter production.

The Lovable RLS Exposure (CVE-2025-48757)

170+ apps built with Lovable had exposed databases. 83 percent involved RLS misconfigurations.

Lesson: Security defaults are your problem. The platform’s default of “RLS off” is the platform optimizing for a smooth onboarding demo. You optimize for a non-leaked customer database. Always enable RLS on user-facing tables. See Part 5 for the discipline.

Compliance and Governance (the baseline, not the ceiling)

The minimum control set for any product handling user data:

ControlImplementationPriority
Secrets scanningPre-commit + CI (gitleaks)P0
Dependency scanningnpm audit + Snyk or Socket.devP0
SASTCodeQL + SemgrepP1
Code reviewRequired for merge, branch-protection enforcedP0
Access controlBranch protection + least-privilege IAMP0
Audit loggingCloud Logging or equivalentP1
Incident responseDocumented runbook, not improvisedP1

P0 means “you don’t ship without this.” P1 means “you have this within 30 days of going to V1.”

AI Governance Checklist (additional, for AI-built systems)

ItemWhat good looks like
AI code review process documentedA written policy on how AI-generated code gets reviewed differently (or the same) as human code
AI tool allowlist maintainedWhich agents/models are allowed in which contexts, updated quarterly
Training data policies definedAre your customer queries being used for training? Do you have a policy?
IP tracking system in placeWhen AI generates code, what’s the license posture? Tracked?
Prompt and response logging (where required)For regulated industries, logging the prompts and responses that touch sensitive data
Model versioning trackedWhich model version produced which artifact, captured in CI metadata
Bias testing performedFor any user-facing AI feature, the basics of fairness testing
Security review of AI outputsAt least sampled, not just trusted

If you’re in healthcare, finance, or defense work, double everything above and add the domain-specific controls. The mandate from Part 3 is coming. The orgs that already have these controls in place will absorb the mandate. The orgs that don’t will be rebuilding under pressure.

The Operating System in Notion (or whichever doc tool you prefer)

The original v3 doc proposes a Notion workspace structure. I run a version of this. The exact tool matters less than the discipline of having a single canonical workspace for the whole operation.

Pages in the “Agentic Coder OS” workspace

PagePurpose
00. This BibleThe series itself, pinned and versioned
01. PRDs + Feature SpecsProduct requirements + SDD specs per feature
02. Architecture and Data ModelsTechnical design decisions, ADRs
03. BacklogLinear / Jira links + the current sprint
04. RunbooksOperational procedures (incidents, deploys, on-call)
05. Security BaselineThe controls from this part, as checklist + status
06. TemplatesReusable templates (spec, launch checklist, postmortem)
07. PostmortemsIncident reviews, learnings, follow-up actions
08. UI StandardsApproved packs, design tokens, the “no random libs” policy
09. Agent LibraryPrompts, configs, harness definitions, CLAUDE.md / agents.md master copies
10. Cost TrackingCloud spend monitoring + AI API spend, weekly review
11. Context Engineering PlaybookRules files, context strategies, what works for your stack

The workspace is one of the highest-leverage investments you make. Without it, every agent session reinvents context. With it, every agent session inherits context.

Templates and Checklists

These are the templates I’d pin in Page 06. Use them. Modify them. Make them yours. But don’t operate without them.

Feature Spec Template (SDD)

Problem: [One paragraph describing the problem]
User: [ICP, Ideal Customer Profile]
Acceptance Criteria: [Numbered, testable, binary]
Data Contracts: [Input/output schemas, API contracts]
Error Handling: [Expected failure modes and responses]
Security Notes: [Auth requirements, RLS policies, rate limits]
Non-Goals: [What you are NOT building]
Success Metric: [One measurable number]

The first time you fill this out it feels like overhead. By the third feature, you realize it’s saving you 4 hours of agent thrash per feature.

Launch Checklist

Pre-Launch:

  • All tests passing
  • Security review complete (against the Part 3 + Part 5 checklists)
  • RLS enabled on all user-facing tables
  • No secrets in code
  • Error tracking configured (Sentry, Logflare, etc.)
  • Structured logging configured
  • Backups configured AND restore tested
  • Rate limiting enabled on public endpoints

Launch Day:

  • Deploy to production
  • Smoke test all critical flows
  • Monitor error rates and performance

Post-Launch (first 24 hours):

  • Review error logs (real, not just glance)
  • Check user feedback channels
  • Verify backups ran successfully
  • Update documentation with anything you learned

This list is non-negotiable. Skipping items is how you become the next case study in the next version of this series.

Your Daily, Weekly, and Monthly Loops

This is what turns the discipline from “I read the bible” into “I operate the bible.”

Daily Ship Loop (non-negotiable)

Morning (30 min):

  • Review overnight errors and alerts
  • Pull the next smallest ticket (Stage 4 from Part 5)
  • Brief the Orchestrator agent on context

Work (focused blocks, 90 to 180 min each):

  • Orchestrator: Plan + files + verification criteria
  • Coder: Implement the minimal diff
  • Tester: Add tests + run the suite
  • Reviewer: Security + correctness check (with the strong Phase 2 model from Part 4)

Commit (per feature):

  • Pre-commit hooks pass
  • PR created
  • Vercel preview verified
  • CI gates pass
  • Merge

Evening (15 min):

  • Update TODO_DEBT.md and Linear/Jira
  • Plan tomorrow’s focus

Weekly CTO Loop

DayBlockFocus
Monday60 minReview metrics, check cloud and AI API spend, prioritize the week’s work
Wednesday90 minPay down top 3 debt items from TODO_DEBT.md, review incident logs, check CI reliability
Friday60 minReview dependencies (npm audit), update runbooks, audit UI consistency, write week summary

Monthly Review (90 min, once per month)

  • Full security audit (against the Part 3 + Part 6 checklists)
  • Dependency updates (with proper review, not blind)
  • Cost optimization review (cloud + AI APIs)
  • Architecture review (does any decision need revisiting?)
  • Process retrospective (what worked, what broke, what to change)
  • Update this Bible as needed (your version, in your workspace)

The monthly review is where the system improves itself. Don’t skip it. The hour you spend monthly compounds into 12 hours of fewer fires per year.

Appendix: Your Default Choices

If you want a single sheet of “always works” defaults for solo and small-team work in May 2026:

SaaS

LayerDefault
FrameworkNext.js + TypeScript
DatabasePostgres via Supabase
UIshadcn/ui (lock the tokens)
Deploy frontendVercel
Deploy backendSupabase functions to start, GCP Cloud Run by V1

Mobile

LayerDefault
FrameworkExpo Router tabs template
OTAEAS Update
Promote toIgnite when you hit production scale

Mature Backend

LayerDefault
ComputeGCP Cloud Run
SecretsGCP Secret Manager
LogsGCP Cloud Logging
ScheduledGCP Cloud Scheduler
AsyncPub/Sub or Cloud Tasks

AI Coding Stack (May 2026)

UsePick
Default CLI agentClaude Code (terminal + desktop + web + phone)
Light IDECursor at $20/mo
Plan-mode workOpenAI Codex CLI / Codex desktop
Google-bundled agentic CLIAntigravity CLI (free on AI Pro)
Model routing fallbackOpenRouter
Production-studio agentCursor SDK (@cursor/sdk)
Heavy programmatic agentClaude Agent SDK
Durable workflowsVercel Workflow SDK
Isolated agent executionVercel Sandbox
Reactive DB + AI threadsConvex + @convex-dev/agent

Engineering Methodology

DisciplineHow
SpecsSpec-Driven Development with GitHub Spec Kit
ContextIterative CLAUDE.md / agents.md files, five strategies (selection, compression, ordering, isolation, format optimization)
HarnessVerification gates, tool allowlists, feedback loops, observability

Appendix: “Stop and Think” Moments

Before every AI task, ask yourself:

  • Does this need production secrets? If NO, proceed. If YES, scope down.
  • Could this delete or destroy data? If YES, explicit human approval gate.
  • Is this touching auth or payments? If YES, extra review by the Phase 2 Reviewer agent.
  • Am I confident in the instructions I’m giving the agent? If NO, clarify first.
  • Do I have a spec for this? If NO, write one first.

These five questions catch most of the dumb mistakes before they ship.

Appendix: The Final Checklist Before Every Deploy

  • Tests pass
  • Types pass
  • Lint passes
  • Build succeeds
  • No new dependencies without approval
  • No secrets in code
  • RLS configured (if applicable)
  • Error handling in place
  • Logging enabled
  • Manual smoke test done
  • Spec updated to match implementation

Closing: this is a practice, not a stunt

Six parts, written across two months, drafted as a doc back in March, refreshed and shipped as a series in May because the world moved on me between drafts. That’s the whole point. The discipline has to evolve faster than the toolchain, because the toolchain evolves faster than anyone’s previous best practices.

What I want you to take from this series:

  • The IDE became an ADE because the work changed shape, and the labeling caught up. Part 1.
  • You don’t need an expensive stack to operate at principal level. You need a lean stack that earns its place. Part 2.
  • We never finished SSDLC. AI-DLC is showing up alongside it. The pieces are scattered. Build the spine before the mandate forces you to. Part 3.
  • Six personas, two-phase model routing, allowlisted tools, one UI pack per app, gates that never get overridden. Part 4.
  • Vercel for frontend, GCP for backend, Supabase + RLS for data, Expo for mobile, a 5-stage runbook with real time budgets. Part 5.
  • The operating system in this part is what makes the previous five parts compound instead of decay. Part 6.

You can read all the AI dev tweets you want this year. The operators who’ll be standing at the end of 2026 are the ones who built a practice. Not a stack. A practice.

If this series helped, subscribe to AI in Living Color below. There’s a newsletter version, a podcast version (real recorded host intro coming, currently sneak-peek), and the X / YouTube / TikTok crosspost going live. Built in public, all free, no paywalls.

Catch you in the next one.


This was Part 6 of “The Principal Engineer’s Agentic Coding Bible,” v3.0, originally drafted March 2026, published in pieces through 2026 in my actual voice. Built in public. Bingo Codes