Seems like a good idea, I'd probably use it. Not sure if you want additional, 'mineral-ish' adjectives here, so I'll throw in a few extras.
White: Pearl, Moonstone
Yellow: Citrine, Zircon
Orange: Amber
Green: Emerald, Jade, Peridot, Tourmaline
Blue: Sapphire, Aquamarine, Turquoise
Red: Ruby...