Vibe Coding For Creators (Make Games, Apps & VFX with LLMs)

Vibe Coding For Creators (Make Games, Apps & VFX with LLMs)

High-level takeaway

• Modern multimodal LLMs (Claude 3.7 “Sonnet”, Google Gemini / AI Studio, Grok/xAI, ChatGPT variants) are being used as creative engineering copilots — not just text tools.
• Creators can prototype interactive 3D scenes, VFX assets, video analysis pipelines, and small web apps entirely from prompts and iterative feedback, often without touching a code editor or 3D package.
• Core argument (speaker): pair product/systems thinking (PRD, iteration, role-play prompts) with LLM “chain-of-thought” reasoning to drive fast, reliable creative builds and throwaway tools.

Pair a clear product mindset (PRD + iteration) with LLM chain-of-thought reasoning to build fast, reliable creative prototypes and throwaway tools.

Key demos, features, and workflows (technical focus)

1) 3D city block multi-agent simulation (Claude 3.7 Sonnet)

• Built a 15–25s downtown intersection simulation with multiple camera views (orbit/cinematic/top-down) entirely from prompts — no 3D assets or Blender used.
• Planning-first approach: write a PRD, have the model adopt PM/tech-lead roles, and break implementation into iterative chunks (environment → traffic FSM → vehicles → pedestrians → cameras).
• Iterative refinement via screenshots and annotations to fix z-fighting, crosswalk placement, lighting, and camera moves.
• Traffic modeled as a finite-state machine (red/yellow/green cycles); agents perceive signals and coordinate.

2) “Claw/Claude artifacts” and throwaway creator apps

• Generate small interactive web tools inside the LLM environment (previewable in-browser).
• Example tools: shot-list manager (drag/reorder, drop-down camera angle, checkboxes), quick visualizers for social assets (e.g., hypercube/tesseract visualizer), 360 camera overlap visualizer.

3) Multimodal video/audio analysis (Google AI Studio / Gemini)

• Capabilities: upload videos/audio/images, 2M token context, multimodal reasoning (frame sampling + full audio), Google Search grounding, safety controls, Drive integration.
• Outputs: timestamped key moments (JSON), semantic detections, 2D/3D bounding boxes with FOV sliders; object/event JSON suitable for downstream tooling (e.g., After Effects).
• Real examples: create HUD overlays from synchronized POV videos — detect shots, hits/misses, timestamps, target color; export JSON and paste into After Effects as expressions/text layers to drive on-screen stats.
• Limitation: current video reasoning is approximate (frame sampling ~1 FPS) — requires manual spot-checking and small corrections.

4) Workflow examples and productivity uses

• Take selection & editing guidance: upload multiple takes, transcribe + analyze audio and visual energy to recommend the most engaging take and explain why.
• Speaker coaching: upload slides (PDF) + dry-run audio → slide-by-slide feedback, filler-word detection, timing/emphasis suggestions.
• Research & summarization: use Gemini/Deep Research/Perplexity to crawl sources, generate citation-backed reports, export to Google Docs; use NotebookLM to distill long papers into a 15-minute audio summary.
• Rapid prototyping + reskinning: export LLM-generated video to Runway (video-to-video) to prototype and reskin quickly instead of opening Blender/C4D/Maya.

Model- and tool-specific notes / analysis

• Claude 3.7 (Sonnet): strong “extended thinking” / chain-of-thought capability; suited for planning and stepwise engineering prompts.
• Google Gemini / AI Studio: best-in-class multimodal video reasoning, very large context window, Google search grounding, helpful safety/response dials; good for timestamped video analysis, clip selection, and object detection. “Gemini Flash” powers real-time clip tools (e.g., Opus Clips).
• Grok / xAI: fast-moving, excels at real-time/timely information and personalized product-recommendation style queries.
• ChatGPT / Deep Research: good for structured long-form research; paid tiers offer deep-research capabilities with citation exports.
• Recommendation: use multiple models as different “team members” according to strengths; adjust temperature/creativity dials depending on whether you want creativity or factual precision.

Practical how-to / best-practice tips

• Start with a high-level goal and a PRD; have the model adopt PM/tech-lead roles before asking for code.
• Break features into iterative chunks; prioritize foundation features first.
• Use annotated screenshots for visual fixes — faster and clearer than long textual descriptions.
• Export LLM outputs (JSON/timestamps) into existing tools (After Effects, Runway, Notion, Google Docs).
• Use voice dictation / voice notes for rapid idea capture and prompt building.
• Expect imperfect outputs for video/object detection (1 FPS sampling, bounding-box tuning needed); spot-check and iterate.
• For privacy/training concerns: check provider terms — some free modes may use uploaded data for training unless you opt out via paid API keys or settings.

Tools, platforms, and libraries mentioned

• Models / platforms: Claude 3.7 Sonnet (Anthropic); Google Gemini / AI Studio (Gemini Flash / 1.5 / 2.0 Pro); Grok / xAI; ChatGPT (4o/4.5 / Deep Research); Gemini Live.
• Creator tools & integrations: Runway (video-to-video), After Effects, Premiere, Blender / Cinema 4D / Maya, p5.js / three.js, Cursor (in-browser dev preview), Opus Clips, Notion, Google Drive/Docs.
• Research / audio: Perplexity, NotebookLM, 11/12 Labs, other “video intelligence” startups.
• Other mentions: Agisoft Metashape, RealityCapture, Puget Systems benchmarks.

Use cases emphasized

• Rapid prototyping of 3D scenes and multi-agent simulations without deep 3D expertise.
• Generating throwaway web apps to speed production workflows (shot lists, checklists, visualizers).
• Automated video analysis: chaptering, highlight extraction, object/event detection, HUD generation.
• Pre-production & planning: shot lists, storyboards, camera rig/360 overlap visualization.
• Speaker rehearsal and targeted feedback.
• Research condensing and discovery.

Limitations & cautions

• Video understanding is imperfect (frame sampling, occasional mis-detections); manual verification is required.
• Privacy/training caveats vary by provider and mode — read provider documentation if concerned about model training on uploads.
• Not a replacement for deep 3D expertise when production-quality scenes are required — excellent for prototyping and rapid iteration, not final VFX-grade outputs.

Main speakers / sources (mentioned)

• Speaker: the video’s creator (signed off as “Bavo”).
• Models/platforms referenced: Anthropic Claude (3.7 Sonnet), Google Gemini / AI Studio, OpenAI ChatGPT (4o/4.5), Grok (xAI).
• Other people/sources: Logan (Google contact), Peter Levels (example developer), Andre Karthy (term “vibe coding”), Elon Musk / xAI, Hugging Face (CSO interview referenced), startups like Opus Clips, 12 Labs, Perplexity, NotebookLM.