Btukfyl

I really fail to understand why the answer cannot be option (1).

Here's my method:

For $a = -1$, the coefficient determinant determinant is 0. And if we create $Delta_z$, the determinant in which coefficients of $z$ are replaced by $3,6,b$, (b= 9), that determinant is non zero.

My book says that if coefficient determinant is $0$ and at least one of $Delta_y, Delta_z, Delta_x$ is non $0$ then the system has no solution.

But the answer given is $(2)$. What's wrong with my method? What's the best way to tackle such problems?

You miscalculated $Delta_z$. The book solution is correct.
In order to have no solution, it is necessary that the determinant of the matrix of coefficients be zero: this indeed yields $a=-1$. The rank of the matrix is $2$ in this case, and up to a constant multiple, the only nontrivial "dependence" of the equations can be $1cdot (1)-2cdot (2) +1cdot (3) = 0$.
So under the $a=-1$ assumption, there are infinitely many solutions iff this generalizes to the right hand sides, that is, iff $3-2cdot 6 + b =0$, i.e., $b=9$.
Hence, the answer is $a=-1, bneq 9$.

I suppose the best way to tackle is to row-reduce the matrix and see what happens. You have to eliminate the third and the first columns:
$$
begin{pmatrix}
1 & a & 1 & 3\
1 & 2 & 2 & 6 \
1 & 5 & 3 & b
end{pmatrix}
to
begin{pmatrix}
1 & a & 1 & 3\
0 & 2-a & 1 & 3 \
0 & 5-a & 2 & b-3
end{pmatrix}
to
begin{pmatrix}
1 & 2a-2 & 0 & 0\
0 & 2-a & 1 & 3 \
0 & 1+a & 0 & b-9
end{pmatrix}
$$
so if $a=-1$ the last equation reads $0=b-9$, thus having $b=9$ you have $0=0$, which eliminates the equation, and now you have 2 equations in 3 unknowns, hence an infinite number of solutions. Thus (i) cannot be the answer.

The condition for no solutions would require that $a=-1$ (forcing the last equation to read $0=b-9$) and $b ne 9$ (forcing it to be false).

UPDATE

In retrospect, as you say, if $a=-1$ then $Delta = 0$, so there will not be a unique solution. However, if $b = 9$, the last column of $Delta_z$ is $(3,6,9) = 3 times (1,2,3)$, so exactly $3$ times the last column of $Delta$, hence, $Delta_z = 3Delta = 0$, and the idea in your book is correct.

I would do it this way, using row reduction. The criterion for a non-homogeneous linear system to have solutions is that the matrix of the linear system (l.h.s.) and the augmented matrix have the same rank.

From this criterion we deduce readily that if this system has no solution, the matrix of the l.h.s. cannot have rank $3$, hence its determinant, which is equal to $-(a+1)$ is zero, whence $a=-1$. Computing the determinant by row reduction shows this matrix has rank $2$.

Now, if $a=-1$, the augmented matrix reduces as follows:
$$begin{bmatrix}1&-1&1&3\1&2&2&6\1&5&3&bend{bmatrix}rightsquigarrow
begin{bmatrix}1&-1&1&3\0&3&1&3\0&3&1&b-6end{bmatrix}rightsquigarrow
begin{bmatrix}1&-1&1&3\0&3&1&3\0&0&0&b-9end{bmatrix} $$
Thus, for $a=-1$, the augmented matrix has rank $3$ if and only if $b-9ne 0$.

Rewrite the system as:
$$
AX=B
$$
where $X=(x,y,z)^t$, $B=(3,6,b)^t$ and
$$A = begin{bmatrix}
1 & a & 1 \
1 & 2 & 2 \
1 & 5 & 3
end{bmatrix}$$

First you must have $det(A)=0$ which after calculations give $a=-1$.
So the system becomes:

$$A = begin{bmatrix}
1 & -1 & 1 \
1 & 2 & 2 \
1 & 5 & 3
end{bmatrix}$$

Now, row 3 becomes all zero when $r_3rightarrow r_3-2r_2+r_1$. Applying this to the augmented matrix $[A|B]$ gives the $(3,4)$ entry to be:
$$
b-9
$$

It is required that this entry is nonzero in order to have an inconsistent matrix.
Thus, $a=-1, bneq 9$.