If the coefficient of x8 in (ax2+1bx)13 is equal to the coefficient of x-8 in (ax-1bx2)13, then a and b will satisfy the relation
(1)
∵ Tr+1 in (ax2+1bx)13
=Cr13(ax2)13-r(1bx)r=Cr13a13-rbrx26-3r
For coefficient of x8, we have 26-3r=8 ⇒ r=6
⇒ Coefficient of x8=C613a7b6
∵ Tr+1 in (ax-1bx2)13=Cr13(ax)13-r(-1bx2)r
=Cr13a13-rbrx13-3r(-1)r
For coefficient of x-8, we have 13-3r=-8 ⇒ r=7
∴ Coefficient of x-8=-C713a6b7
Now, C613a7b6=-C713a6b7 ⇒ ab=-1 ⇒ ab+1=0