spumux capabilities/limitations
says the error message in the title is usually due to using too many colors. For me it was caused by the use of a background color for a button font in DVDStyler. Once I removed the background color and erased the temp directory, the menu generating went smoothly. In order to make the font stand out from its background, I darkened the background image from underneath it.
Addition: For a text button with lots of words, this error seems to pop up even though the color limitation mentioned above is met. I followed an advice given on DVDStyler forums and manually enlarged the width and height of a problematic button and that seems to have worked. What an annoying glitch…