Should I try pulling...?

Is there no downside to skipping the anniversary...?

It depends on how you want to play going forward... Ibar comes around every two and a half months, so if you want to boost your magic power, pulling on the anniversary banner is an option. Personally, I recommend the Marionette from the upcoming Jester banner; she's a bit tricky but has strong abilities that can be applied in various ways. (Laurion from the Golden Land banner is truly top-tier, so waiting for that might also be a good move.)