$${1 \over {\left( {1 - ax} \right)\left( {1 - bx} \right)}}$$

= $${\left( {1 - ax} \right)^{ - 1}}{\left( {1 - bx} \right)^{ - 1}}$$

= $$\left[ {1 + \left( { - 1} \right)\left( { - ax} \right) + {{\left( { - 1} \right)\left( { - 2} \right)} \over {1.2}}{{\left( { - ax} \right)}^2} + ...} \right]$$ -
               $$\left[ {1 + \left( { - 1} \right)\left( { - bx} \right) + {{\left( { - 1} \right)\left( { - 2} \right)} \over {1.2}}{{\left( { - bx} \right)}^2} + ...} \right]$$

= $$\left[ {1 + ax + {a^2}{x^2} + ... + {a^{n - 1}}{x^{n - 1}} + {a^n}{x^n}}+.... \right]$$ -
               $$\left[ {1 + bx + {b^2}{x^2} + ... + {b^{n - 1}}{x^{n - 1}} + {b^n}{x^n}}+.... \right]$$

Coefficient of xⁿ =

$${a^n} + {a^{n - 1}}b + {a^{n - 2}}{b^2} + .... + {b^n}$$

= $${a^n}\left[ {1 + {b \over a} + {{{b^2}} \over {{a^2}}} + ..... + {{{b^n}} \over {{a^n}}}} \right]$$

= $${a^n}\left[ {{{{{\left( {{b \over a}} \right)}^{n + 1}} - 1} \over {{b \over a} - 1}}} \right]$$

= $${a^n}\left[ {{{{b^{n + 1}} - {a^{n + 1}}} \over {{a^{n + 1}}\left( {{{b - a} \over a}} \right)}}} \right]$$

= $${{{{b^{n + 1}} - {a^{n + 1}}} \over {b - a}}}$$

Let r = 2

$$\therefore$$ 2nd, 3rd and 4th terms are in AP.

2nd term = T₂ = $${}^m{C_1}.y$$

Coefficient of T₂ = $${}^m{C_1}$$

3rd term = T₃ = $${}^m{C_2}.{y^2}$$

Coefficient of T₃ = $${}^m{C_2}$$

4th term = T₄ = $${}^m{C_3}.{y^3}$$

Coefficient of T₂ = $${}^m{C_3}$$

$$\therefore$$ 2.$${}^m{C_2}$$ = $${}^m{C_1}$$ + $${}^m{C_3}$$

$$ \Rightarrow $$ $$2.{{m\left( {m - 1} \right)} \over {1.2}}$$ = $${m \over 1}$$ + $${{m\left( {m - 1} \right)\left( {m - 2} \right)} \over {1.2.3}}$$

$$ \Rightarrow $$ 6m² - 6m = 6m +m(m² - 3m + 2)

$$ \Rightarrow $$ 6m² - 6m = 6m + m³ - 3m² + 2m

$$ \Rightarrow $$ 6m - 6 = 6 + m² - 3m + 2

$$ \Rightarrow $$ m² - 9m + 14 = 0

Now put r = 2 at each option and find answer.

In option C, $${m^2} - m(4r + 1) + 4\,{r^2} - 2 = 0$$ putting r = 2 we get

m² - 9m + 14 = 0. So Option C is correct.

$${\left( {1 + x} \right)^{{3 \over 2}}}$$ = 1 + $${3 \over 2}x + {{{3 \over 2}.{1 \over 2}} \over {1.2}}{x^2} + ...$$

= 1 + $${3 \over 2}x + {3 \over 8}{x^2}$$ (As $$x$$ is so small, so $${x^3}$$ and higher powers of $$x$$ neglected)

$${{{{\left( {1 + x} \right)}^{{3 \over 2}}} - {{\left( {1 + {1 \over 2}x} \right)}^3}} \over {{{\left( {1 - x} \right)}^{{1 \over 2}}}}}$$

= $${{\left( {1 + {3 \over 2}x + {3 \over 8}{x^2}} \right) - \left( {1 + {}^3{C_0}.{x \over 2} + {}^3{C_1}.{{\left( {{x \over 2}} \right)}^2}} \right)} \over {{{\left( {1 - x} \right)}^2}}}$$

= $${{{3 \over 8}{x^2} - {3 \over 4}{x^2}} \over {{{\left( {1 - x} \right)}^2}}}$$

= $${{x^2}\left( {{3 \over 8} - {3 \over 4}} \right){{\left( {1 - x} \right)}^{ - 2}}}$$

= $${ - {3 \over 8}{x^2}\left( {1 - {1 \over 2}\left( { - x} \right) + ....} \right)}$$

= $$ - {3 \over 8}{x^2} - {3 \over {16}}{x^3}$$

[ As x³ is so small we can ignore $$-{3 \over {16}}{x^3}$$]

= $$ - {3 \over 8}{x^2}$$

General term of $${\left[ {a{x^2} + \left( {{1 \over {bx}}} \right)} \right]^{11}}$$ is T_r+1.

T_r+1 = $${}^{11}{C_r}{\left( {a{x^2}} \right)^{11 - r}}{\left( {{1 \over {bx}}} \right)^r}$$

= $${}^{11}{C_r}{\left( a \right)^{11 - r}}{\left( b \right)^{ - r}}{\left( x \right)^{22 - 3r}}$$

For the coefficient of x⁷,

$$ \Rightarrow $$ 22 - 3r = 7

$$ \Rightarrow $$ r = 5

So coefficient of x⁷ = $${}^{11}{C_5}{\left( a \right)^6}{\left( b \right)^{ - 5}}$$ ......(1)

Now General term of $${\left[ {ax - \left( {{1 \over {b{x^2}}}} \right)} \right]^{11}}$$ is T_r+1.

T_r+1 = $${}^{11}{C_r}{\left( {a{x}} \right)^{11 - r}}{\left( { - {1 \over {bx}}} \right)^r}$$

= $${}^{11}{C_r}{\left( a \right)^{11 - r}}{\left( { - 1} \right)^r}{\left( b \right)^{ - r}}{\left( x \right)^{11 - r}}{\left( x \right)^{ - 2r}}$$

For the coefficient of x^-7,

11 - 3r = -7

$$ \Rightarrow $$ r = 6

$$\therefore$$ Coefficient of x^-7 = $${}^{11}{C_6}{\left( a \right)^5}{\left( { - 1} \right)^6}{\left( b \right)^{ - 6}}$$

According to question,

Coefficient of x⁷ = Coefficient of x^-7

$$ \Rightarrow $$ $${}^{11}{C_5}{\left( a \right)^6}{\left( b \right)^{ - 5}}$$ = $${}^{11}{C_6}{\left( a \right)^5}{\left( { - 1} \right)^6}{\left( b \right)^{ - 6}}$$

$$ \Rightarrow $$ $$ab = 1$$