Prior SOF experience isn't required to be a PA in one of the Special Forces groups. It is very common for most of those positions to be filled by former 18 series guys that then went on to PA school, however, it is not disqualifying if you didn't earn your tab and beret before switching career fields.
I will say, being a PA in group may not fulfill the medical fantasy that you have in your head. You might have some deployments that require you to operate independently and with little support where you get to practice medicine. You might not. You will not be "in charge" of the 18Ds. Getting them all in one room at the same time is nearly impossible. If you are a solid PA and have beneficial medical training that you can provide, you will find an audience with some of the medics (notice I said some).
Since you are looking at going to the National Guard I can't speak to the opportunities that they have for PAs. On the active side there are other SOF jobs that allow you to practice medicine and lead as a PA that are more fulfilling, in my opinion, than being a PA in one of the SF groups.