It sounds like \"Root Rot\". I have the same problem due to the very wet weather this year. We have been treating the \"soil\" with Subdue GR. We also started using Aliette WDG as a folier treatment. The plant takes the Aliette in through the leaves and into the root to kill the disease. The treatments seem to be working as we are finding less dead plants.
If you dig a root and it seems waterlogged and soft, it most likley is root rot.
One other thing that can cause the tops to turn red and yellow is too much sun. Do you have enough shade?
As hillbilly said, buy the book from Scott Persons, it is a great guide to farming Ginseng.
MMG