I see yellow spots on my wild simulated seng quite often, and also on wild ginseng in my area.
From my experience it starts as little yellow spots, and later on they may turn to a bullseye looking spot (brown in center, yellow on edges) and in advanced stage the leaves will look like they are melting, tops will die.
It does not always advance to the leaf melt or tops die off stage. Sometimes it stays in the yellow or bullseye spot stages. Just depends on how much rain you get, and how much the fungal issue advances.
Alternaria - I assume is the cause.
Fungicides - would be the solution that many turn to.
I choose not to spray mine though - I am just not going to do that. Over time they will thin out to a level that the land can support even though it is prone to fungal issues.
Where I used to plant large beds, at 4-5 seeds per sq ft...
I now plant much smaller areas and space them out, and only plant 1-2 seeds per sq ft.
That has helped here on my place.
Good Luck !
PS - below is a pic of a couple of 2 prongs (wild sim) with a few spots on them. I took this pic on June 14.
I am trying to upload a picture but I am not having much luck. Thanks a lot for the info guys. If the upload of the picture worked then there should be picture below... If not how can I get the picture to show?