You are the only person who can justify driving that far to swim. If it is fun, and you enjoy doing it, and it keeps you in shape, then I would say it is justified. It is difficult for anyone else to tell you "if it's worth it", because it has to be worth it to YOU! You have to think of the reasons you are doing it, and evaluate it on your own to tell yourself if it is worth it. Try and think about what else you would be doing during the 3ish hours you are gone to swim. If it is at a point during your day where you would otherwise be sitting at home watching television, then I would say it is absolutely worth it. However, if you have been at work all day and this is the only time you get to spend with your family, then it might not be so much. But, if you absolutely love swimming, need (and want) it to help you stay in shape, then that will make your evaluation more complicated. Like I said, it is up to you to determine if it is worth it or not.

As for improving your times, there are things outside of the water that can help you out, since your water time is limited. Make the most out of your water time, mixing aerobic training and some speed work (depending on which distances you mainly swim), along with mixing up your strokes (once again depending on your prefered events.) Then, at home, try to develop some sort of dryland (or weight lifting) program, to help build up extra strength. To go along with the strength, I would recommend using some sort of Pilates excercise (if availible) to increase your flexibility. Flexibility combined with your body weight training out of the water will compliment your swimming very well. Even increased stretching daily will help tremendously. Abdominal work, body weight squats and lunges, pushups, etc, can all be used to your advantage and will add to your performance in the water.