You don't lose the yellow stars. The Red are just temporary so you can do the quests "earn 10 red stars." It actually makes it easier, so you can get your 10 red stars on the first few levels.