I felt rather discouraged after joining several bug trackers and forums though

Bingo! I think everyone feels this pain. And we can go on and on about need for community input in this process, but this just kills people's interest - even developer interest very often. But for design, I agree with everyone here who don't think these processes work

There are no FOSS design applications.

Agree with all the points you are making around this, but echoing Ryan, there's GIMP and Inkscape, apart from others. These are actually very capable tools. Usable: may not.

For some applications, I would have liked to spend some time on redesigning features, but it was totally unclear who was running the project...

I think this is the basically the heart of the problem. I am filled with optimism on the supply side (designers) but this demand side is still bothering me. We need a little more insight here. If, hypothetically, we do gather a bunch of talented designers who want to contribute, I am not sure they will have autonomy and support from the core maintainers of these projects.

Even in well-organized product teams designers sometimes struggle to convince others their designs should be implemented

YES! But just wish to add that the best design teams have become a whole lot more mature about this overtime and there are some principles around this stuff that didn't exist before.

Community-driven FOSS projects typically have a development process that is optimized for engineers..

Again, bullseye. I think a design pathway would need to bypass these existing processes.

Conclusion

Are you saying we are missing the point here and that what we need to do instead is educate open-source design leaders about the value and techniques of design and product leadership?

Sorry I am forcing words in your mouth because this seems like a very interesting perspective and I am suddenly blown by this and it is changing everything I thought about how to solve this problem before reading your bit.

...An existing community may not like the sudden changes in the design...

Very accurate observations.

FOSS is badly designed because it's badly designed

I agree with a lot of your thinking, Koos, but you know, I fear that we are making assumptions on the desires of the open source software maintainers. May be they do value design too, but don't have the skills or have too much inertia. We know from experience that the great commercially-backed FOSS efforts have all placed critical emphasis on design. This almost always means that there is design ambition on the part of the their leaders. Have any ideas on how to test this hypothesis? I was just going to be really raw and just message them.

Alternatively, there may be projects that are not of the scratch-your-own-itch kind that still have passionate engineers...

Fascinating. Aligns with the kind of work Henry shared about.

As you mentioned, there needs to be coordination, which, I think, means it takes the power of decision making (about product vision, feature prioritization, etc.)...

Again, you hit the heart of the problem. I know a lot of open-source product leaders struggle with burnout and frustration. May be we are beginning to question the organization of FOSS contribution, which is a very complex matter in itself, and the focus of people's entire PhDs (I kid you not, I know people like that).

But hey, I walked into this with optimism, and so we gotta believe there is a way..