I've written a wind quintet and a string quintet. I enjoyed doing both, but I have problems with both too. In the case of the wind one, it's that old thing about the godammed flute - it just can't seem to blend properly unless you take bizarre precautions in the scoring. It's the attack sound: all the other instruments can make a 100% clean attack, but the flute (however good the player) ends up going 'pth'.
With string 5tets I'd have much preferred to have two 'cellos rather than the two violas I had to use (for depth and umpf). Better still, a sextet with two of everything. But even with a sextet it's starting to feel like a different, less chamber-like ensemble, without the intimacy of a string quartet...
I always thought it was the horn that was the main problem with wind quintets. I did think of two I like though:
profile by Hans-Joachim Hespos and Stockhausen's
Adieu (
Zeitmasze is also nice but it isn't a "proper" wind quintet, replacing the horn with an English horn).
I think one of the difficulties of establishing a repertoire for string quintet must be that the instrumentation isn't standardised. The one I wrote, like Alistair's, is for string quartet and double bass (though mine adds electronics and his adds soprano). A combination I've used a number of times within larger ensembles is a sextet of 2 violins, viola, 2 cellos and bass, which I think would make an interesting ensemble on its own too. I've also used the quintet with two violas within a larger work (but often appearing on its own), with the second viola doubling on third violin, thus duplicating the lineup of Morton Feldman's
Violin and String Quartet.
I did quite like the massed bassoons. To me they almost had about them an "archaic" sound as of some Renaissance consort of curtals/dulcians, more so than a modern brass ensemble would have anyway.