A level 80 warrior creature has 9729 armor. C=15232.5. So, the cap is (9729+15232.5)/3=8320.5. Let’s say a player has 30% armor penetration from armor penetration rating and no other modifiers that complicate the calculation (talents, Battle Stance, Sunder Armor, etc.). The
game chooses the minimum of 8320.5 and 9729, so 8320.5. That is multiplied by 30% = 2496.15, and so that much armor is ignored. The effective armor on the target is 7232.85 (9729-2496.15). From a player point of view, the armor penetration rating didn’t ignore the full 30%, but instead ignored 25.66%. (85.5% as effective as expected).