YuE2 - новая крутая модель для создания музыки с вокалом. Локально, бесплатно, хорошо. Большой обзор

Summary of the video (tech + product/workflow focus)

What the video covers

• An overview and tutorial-style workflow for “E2 / UV E2”: a local, open-source, multimodal music generation model (≈ 3B parameters) designed to generate full songs with vocals and instrumentals.
• The model supports generating music from:
  1. Text prompts (text → song)
  2. Audio reference (audio → song, i.e., remixes / style transfer with reference melody/chords)

Key motivation / context

• Mentions that some earlier “local models” were replaced with a more restricted/censored version and added download limits (framed as a contrast to this local model).
• Emphasizes that this workflow is local and (for them) free, with explicit control.

---

Hardware / performance details

• “UI2/E2” feels comfortable on GPUs with ~24GB VRAM (example benchmark mentioned on RTX 4090).
• Claims it can still run on 8GB cards (“large eight-gigabyte buckets” mentioned).
• Rough speed estimates:
  • On BF16: about ~30 seconds of generated song per minute of runtime (average; variable).
  • For a long example: generation time scales so that 5 minutes of audio may take several minutes in practice.
• Cloud alternative (sponsored/advertising):
  • Rent GPUs via Imers Cloud (clusters with multiple 4090s mentioned).
  • Pricing described as per server time used (seconds), not per tokens.
  • Workflow: pre-build prompts → launch server with ComfyUI → queue/wait → collect outputs → shut down.
  • Includes a permanent referral link for a 20% bonus on the first top-up.

---

Model architecture / capabilities (main analysis)

• E2/EUV E2 is described as planning the song structure first, then rendering audio.
  • First: plans composition / score (described like composition planning such as Row ABC).
  • Then: renders audio based on that score.
• This enables a workflow feature:
  • You can extract/edit the intermediate “score” (conceptually exported to a MIDI-like/diff file, compatible with MIDI/diff tooling).
  • Presented as giving “explicit control over structure” versus typical “black box” generation.

---

Installation & setup (ComfyUI-focused)

• Installation is described as straightforward:
  • Update ComfyUI to v0.36.0+
  • Download the workflow and place models in the correct folders:
    • In the ComfyUI models directory
    • With an audio encoder folder for reference-audio mode
• The workflow is “native nodes heavy” and defaults aim to be minimal/simple.

---

Checkpoints & model variants

Two main components:

1. Checkpoint models - One checkpoint model with two precision/quantization variants:

   • BF16: fuller quality; higher VRAM requirement (e.g., 16GB+)
   • INT8 / quantized (“Guf” models): lower VRAM usage
   • Guidance:
   • Use BF16 if you have 16GB+
   • Use quantized versions if VRAM is limited

2. Audio Encoder - Used for recognizing music from reference audio (melody/chords depending on the selected mode).

---

Text-to-song workflow (ComfyUI nodes & controls)

The workflow is described as two major stages:

Stage 1: Score creation

• A node generates ABC/score representation.
• Options:
  • Full vs Melody mode:
    • Full: plans melody + chords
    • Melody: only melody (no chords)
• Parameters:
  • “Randomness” / seed behavior (some nodes default to random; optionally can be fixed for reproducibility)
  • Max ABC tokens (maximum score length)

Stage 2: Audio generation

• A sampling node generates music using the planned score.
• Parameters:
  • Max duration (developer-stated comfortable max around 300 seconds / 5 minutes)
  • Sampling steps (e.g., “32”)
  • Batch size / number of samples / number of variants
• Decoding & saving:
  • Uses decoder nodes; if VRAM is insufficient decoding can fail (OOM risk).
  • Audio saving node outputs the final files.

---

Audio reference / remix workflow (audio-to-song)

Core difference vs text workflow:

• Swap/extend the workflow using:
  • the Audio Encoder
  • an audio loading node
• How it works:
  • The score is created from the reference audio
  • Then style prompt is applied to generate the final song

Lyrics handling:

• Lyrics can remain available, but behavior depends on settings:
  • Lyrics can be disabled (keeping reference words)
  • Or replaced (using prompt lyrics)

Operating modes demonstrated

1. Melody-to-melody remix - Keep melody from reference; apply new style - Prompt lyrics often disabled to preserve reference structure

2. Full remix (melody + chords) - Uses more “full” score extraction from reference to preserve harmony

3. Replace lyrics - If you provide lyrics in the prompt/lyrics field:

   • lyrics may be replaced
   • but results can degrade if lyrics length, syllable rhythm, and verse/chorus structure don’t match what the reference expects

Pitfall / troubleshooting note (important)

• If you change the reference audio but the workflow “hangs” or reuses cached results:
  • the score may not refresh, so the old melody/score persists
• Fix suggestions:
  • reset cache / toggle ComfyUI
  • refresh page / restart node execution
  • ensure the node generating the score from reference audio actually re-runs

---

Review / results: what the generated songs sounded like

The video includes listening examples (via a “Musical Manure Truck” block). Key qualitative takeaways:

Instrumental tracks

• Sometimes loop/monotony issues (e.g., blues-like loops).
• A glitchphonk-like example also looped similarly.
• Still considered usable for sampling.

Vocal tracks (text-to-audio)

• Russian pronunciation and handling of longer/complex text is evaluated as often “pretty good.”
• Hardcore rap attempt:
  • Can follow the beat, but prompt correctness matters
  • Hardcore achieved only after multiple attempts (approx. “10 tries”)
• “New Metal” described as the heaviest result:
  • includes growling/screams and heavy delivery as strengths

Remix tracks (audio reference mode)

• Melody preservation works (e.g., blues → style remix; fly-song → new metal/pop style).
• Remix with lyrics replacement:
  • can be “unpredictable” when new lyrics differ in meter/structure from the reference.

---

Prompting guide: how to write style + lyrics prompts

Style prompt (text field)

• Must be short narrative language (no special tags).
• Should include:
  • genre/subgenre
  • instruments (piano/guitar/synths, etc.)
  • mood/atmosphere
  • vocals style (male/female, warm lead, bright, etc.)
  • optional tempo/rhythm
• Suggests 1–3 sentences is usually enough.

Lyrics prompt (lyrics field)

• Uses section tags in square brackets, e.g.:
  • [verse], [chorus], [bridge], [outro]
• Rules emphasized:
  • tags should be lowercase
  • tags must be on separate lines
  • blank line between sections is important; otherwise the model may treat sections as continuous
• Instrumental mode:
  • If generating instrumental, lyrics can be left empty
  • Still recommended to leave only instrumental settings because blank lyrics may otherwise cause the model to invent text/melody
• Intro caution:
  • Documentation says intro with text is unreliable for vocals; better to keep intro as instrumental-only
• Meter/pacing constraints:
  • Avoid long lines / overloaded verses (worsens meter and melody)
  • Keep line lengths consistent within a section
  • The model implicitly aligns syllables to beats; too short/too long lines can drift

---

Intermediate “score” editing advantage (big review point)

• One of the strongest claims in the video:
  • Score planning provides predictability and structure control
  • You can edit the score (manually or via MIDI-like workflows) and feed it back
  • Compared to “proprietary services,” this structured controllability is presented as a major advantage.

---

Additional workflow ecosystem recommendations

• Mentions custom node packs for ComfyUI:
  • Coffee UI custom nodes
    • includes a “midi editor”-type node (described)
    • and a node for training LoRA from within the UI
  • Another named pack (“Alex”):
    • nodes include structured generation helpers
    • possibly auto-downloads models
    • has a Russian version with explanations
• Recommends checking those nodes via links in the description.

---

Main speakers / sources

• Zhenya (Женя) — main reviewer/presenter (hosts the review, explains model/prompting/workflow, shows examples).
• Evgen (Евгений) — listens through generated tracks in the “Musical Manure Truck” block and demonstrates remix/text examples.
• E2/EUV E2 creators referenced as the Multimodal Art / Projection team (open-source model source).
• ComfyUI — workflow platform used.
• Imers Cloud — cloud GPU sponsor (advertised service).