hillbilly
if everything in your soil is in spec. as I figure you do then it can only be sun or moisture. Moisture is also a big thing in growing Ginseng. Light is easy to check with low cost meters, but do the tests over a few different weeks for best seasonal levels. Moisture is tougher to get right.
It is not uncommon for some plots to be slower than others and your plants are nothing to worry about, but it is also not uncommon for some two year old plots to have 20% produce seed.
To find the best sites plant 20 different sites with fifty seeds and return every year to check. We still go back and check on sites we did twenty years ago and are now close to someones herbijauna gardens, cottage, hunt camp or bear barrels.
Recheck your soil tests and light meter readings and take note of moisture levels, I don't know of any inexpensive moisture testing equipment out there, maybe others do.
guy :hope this helped