If the coefficient of x7 in [ax2+(1bx)]11 equals the coefficient of x-7 in [ax-(1bx2)]11, then a and b satisfy the relation
(3)
Tr+1 of (ax2+1bx)11=Cr11(ax2)11-r(1bx)r=Cr11a11-rbr(x)22-3r
Tr+1 of (ax-1bx2)11=Cr11(ax)11-r(-1bx2)r
=Cr11a11-rbrx11-3r(-1)r
∴ Coefficient of x7 in
(ax2+1bx)11=C511a6b5
and coefficient of x-7 in
(ax-1bx2)11=C611a5b6
Now, C511a6b5=C611a5b6
∴ ab=1