Whose message is it then? Is this a message with no one sending it?
It is the message of the customer.
Yet, you said "One would think there is a big difference between on well-recognized symbol (a swastika) that sends a clear message and a pink cake with blue icing that no one would know what it meant unless they are told.". But then again, the above is irrelevant because going by your theory.
1. A swastika can be used to send very different messages.
2. What message the customer chooses to send does not matter, since the baker is sending no message.
3. The baker is forced to send the message anyway, or just not make any such cakes.
So, the "One would think..." was not relevant. Or if it was, how was it relevant?