For the Time Quest

I thought I'd start it! But I can't!! And you know how once you accept a Time Quest, you can't cancel it until you clear it, right? Well, I accepted one... When I tried to accept another one, it said something like 'You can accept this after clearing other Time Quests.' So, can I just give up on the Time Quest??