In the implementation of equation H3-6, for the term Mr, the value of MZ is curently being used. The ambiguity is arising from the fact that in H3-6, the code has not explicitly stated what to use for Mr.
Since MZ is 0.0 for member 1 in your case, the value of equation H3-6 is turning out to be 0.1128.
The ratio per equation H1-1b is coming out to be 8.463/8.82 = 0.959. Hence, H1-1b governs.
If you assign a beta angle of 90 degrees to member 1, MZ will become 8.463 kip-inch and the ratio will become 1.072 as per H3-6.
We will change the manner in which this equation is checked so that the resultant of MZ and MY will be used as Mr in equation H3-6.