MCP with Synfire

Yes, this idea is not new. I like it. The LLM chatbots are indeed great at talking about music. But they are bad at generating symbolic music. The output is often trivial, i.e. you can more easily just play some stuff on your keyboard and it will sound more interesting. 

OpenAI and Anthropic can't really help here. This type of architecture requires a custom model specifically trained on the symbolic music domain. A general purpose LLM can't compose.

I'm pretty sure we could make an impressive MCP demo that's magically tinkering with an arrangement in front of your eyes. Apart from the "Wow" effect however, how useful and reliable will that be in practice? Other than written language or code, a LLM can't assess the correctness or quality of music. It also can't hear and choose sounds (remember how "device extraction" only works for a few synthesizer plug-ins). 

An agent is effectively deaf. It could generate some random phrases/clips and then it must wait for the user to judge the output before it can continue. That's actually the Factory workflow.

And then there's the general question of how much automation of the entire creative process is even desirable. 

An automated composition is no longer your music. You didn't make it, you can't claim ownership, you didn't learn anything new. You just waved a hand and the "intelligent random surprise radio" played music that has nothing to do with yourself. The satisfaction of achievement, however small, is gone. The pride of something you accomplished yourself - gone.

I think generating raw material as building blocks is the maximum involvement Synfire should have with generative AI. You will edit the stuff anyway, so it will be more or less yours. You will combine it in ways nobody else would - also great.

Synfire should help you find your way. Not walk the way for you.

I’m actually curious about this.

Absolutely, me too. That would be a great gadget to play around with.

You can already try for yourself:

1. Copy a progression to the clipboard.
2. Paste it into a LLM chat window.
3. Instruct the LLM to compose 16 bars of music based on that. 
4. Ask for the result in JSON format (let it suggest a format first)
   1. positions and durations: 1920 ticks per 4/4
   2. pitch as MIDI note numbers

Hey, I actually did this. Implemented a simple JSON import and asked Claude to compose like "a professional media composer for games or film". 

I mean, it's impressive what Claude pulled off, if you consider that it had to somehow build an internal representation of a score, think about it with respect to motif development, get the pitches and durations correct, export the stuff in JSON format, and all that. 

Really, I tried to make it sound as nice as possible (VSL Strings, VSL Brass, some articulations). For this initial suggestion it burned through 20% of my daily token limit.

If you look at the Figures extracted from the import, you see how simple and easy to sketch these are by hand or with basic phrases from a library. 

VSL can do that with their own instruments because they have full control over them. This detailed level of automation does not work with most third party instruments (NKS might help). Most plug-ins don't communicate with the outside. Only the user can load sounds and set them up manually.

Having AI configure a mixing console, load plug-ins, do the routing, CC automation, etc, is great. I don't know if Synfire's very simple mixer and usually low number of instruments would justify MCP. You can just drop a preset and be done.

And composing Claude can't ;-)

if you look at the boring stuff it generates, language models would make synfire look really bad

I wouldn't dismiss it outright. MCP is interesting for intelligent scripting. A potentially nice assistant for beginners who can't yet figure out how to setup sounds. In order for an agent to do that, it needs to learn which sounds are available. It needs device descriptions for all sounds. A beginner doesn't have these. This is a household task the user needs to complete upfront.

How useful is typing a prompt like "Copy the dynamics of this phrase to the phrase in container Verse", when it merely replaces a single drag & drop action? 

Or: "Soften all piano phrases that come before the chorus", when lowering velocity is impossible do right without hearing and judging the result?

Or: "Generate a variation to the build-up", when each attempt costs a huge chunk of your token limit? Composing music based on words is ridiculously expensive (its a "language" model, after all).

I see the most value in an interactive user manual. Ask the AI for step-by-step instructions on how to complete a task that you describe in natural language. That way, you can learn something useful instead of sitting back and watching the AI mess up your arrangement.