The Software Factory: From Bug Report to Production Code — Davis Palmie, Factory

Summary: AI engineering → “Software Factory” (agents + operational workflow)

Davis Palmi (Factory technical specialist; formerly at Lumetric) explains how AI engineering has evolved—from basic code autocomplete, to file-level generation, to “agents” that can gather context, call tools, and debug code. He argues the next shift is moving from engineers writing code to engineers managing agent systems—organized into a “software factory” that converts organizational inputs (bug reports, feedback, monitoring alerts) into ready-to-run production code.

---

“Software Factory” concept (what it does)

A system of agents that drives the workflow from input → deployment in production.

Agents take on functions beyond coding, including:

• Incident triage
  • Reading Sentry alerts
  • Collecting traces/context
  • Publishing updates to Slack
• Creating and updating plans and documentation
• Executing, validating, testing, deploying, and tracking releases
• Supporting testing/monitoring and handling outdated artifacts
  • e.g., stale documentation and dead code

Humans remain critical as validation checkpoints: monitoring outputs, catching mistakes, and correcting issues—especially before the system runs in closed loops.

---

Why cost control and model choice matter

Factory addresses two major concerns when AI produces more code than humans can reliably review.

1) Budget overspending

• Factory is model-agnostic and not tied to a single provider.
• It emphasizes choosing the best price-performance option per task.
• It cites examples of achieving similar results at lower cost by selecting different model families, including open-source alternatives.

2) Access to best model “right now”

• The approach discourages betting everything on a single favorite model.
• It supports selecting the best tool per task and/or dynamic routing.

---

Organizational change / reducing bottlenecks

Product work and engineering blur as engineers build and steer the factory system.

To make it work, team boundaries must blur so that context flows across org units (engineering, QA, product, documentation, support, ops).

Palmi argues “coding” is usually not the bottleneck. Delays often come from:

• PR review time
• Reproducing user errors
• Testing and debugging time
• Documentation aging/outdating
• Fragmented team communication and geographic separation

The factory is positioned as a way to unify these functions and reduce coordination overhead, aiming to shift from O(N) to O(1)-style management overhead (standardize once, reuse everywhere).

---

Safe rollout strategy (learning + monitoring)

• Start narrow (e.g., incident triage).
• Add more stages only after monitoring shows reliability.
• Before enabling feedback loops, each stage needs “careful control and monitoring.”
• Emphasize gradual automation to build trust and understanding.

---

Key “factory” principles (pillars)

1. Model-agnostic
   • Optimize for speed, efficiency, and cost
   • Target the “Pareto frontier”
2. Sovereign deployment model
   • Choose workspace/deployment approach: managed, custom servers, or isolated networks
3. Integrated into the full SDLC
   • Keep documentation changes synchronized with code changes
   • Planning goals define test criteria across strategy/product/dev/test communication layers

---

Agent readiness framework (8 pillars)

Factory assesses whether agents can safely and effectively operate using eight categories:

• Validation
  • Linters/formatters and safety safeguards
• Build system
  • CI build commands documented
  • Secure unit/integration tests
• Feedback loops
• Documentation for agents
  • e.g., readmes, agents.md
• Reproducible dev environments
• Modularity / boundaries
  • Reduce “chaos” via clear rules
• Observation
  • Time to diagnose failures; debugging visibility
• Proactive security scanning
  • Prevent secrets leakage and vulnerabilities

---

Metrics: measure outcomes, not token/lines produced

• Token/LOC metrics can be gamed (invokes Goodhart’s Law).
• Emphasize measuring results, such as:
  • Signal-to-production time
  • Number of human interventions
  • Median recovery time
  • Code expiration date
  • Cost per PR

Agents and humans should follow the same high-quality paths, with auditability (e.g., RBAC/least privilege) and ongoing monitoring.

---

“Interface-agnostic” access

• Agents should work across team tools (remote machines, CLI, Slack).
• Teams should not have to change their processes/tools to fit an AI provider’s terms.
• The factory approach is also framed as eliminating tribal knowledge, manual processes, and outdated information.

---

Main speaker / source

• Davis Palmi — Technical Specialist at Factory
  • Background mentioned: co-founded Lumetric; previously Technical Manager at Slalom innovation lab.