Your biggest challenge will be the other people you work with. So choose wisely. Don’t allow yourself to be overly dazzled by musical ability. Yes, you want to work with good musicians. But you also want to work with like-minded people. Coping with contrary prima donnas can drain all the joy out of music. When looking for musicians to work with, always ask yourself: “Can I get along with this person?” And know yourself. Like all of us, you likely have things that tick you off. Know your ability to put up with crap. If it’s easy to push your buttons, then know that going in. Don’t saddle yourself with people who constantly rub you the wrong way.
Bear in mind that music is a psychologically intimate endeavor. To some, disagreeing with their musical taste is tantamount to questioning their worth as a person. So gathering a group of touchy musicians to work together is a hotbed for discord. In fact, based on your first post, I can start an argument right off the bat: Why does a folk music group need a drummer? (See how easy that was?)
Go for it, but tread carefully.