Is it okay to save for something fun?

I really want to travel, but every time I put money toward a trip I hear that little voice saying I should be saving for something more serious. How do you decide when a fun goal is worth funding?