Systems of Linear Equations

A system is two or more equations with the same variables, true simultaneously. Solving a system means finding the variable values that satisfy all equations at once. Geometrically: finding where the lines (or planes, or hyperplanes) intersect.

Two methods, one outcome

The two algebraic methods you’ll use most are substitution and elimination. Both arrive at the same answer; elimination is faster when both equations are in standard form ($ax + by = c$). Substitution is faster when one equation is already solved for a variable.

For the example below, we’ll use elimination because the equations are in standard form.

Worked example

Step 1 — choose a variable to eliminate. We’ll eliminate $y$. The $y$-coefficients are $4$ and $-3$. Their least common multiple is $12$.

Step 2 — scale each equation so the $y$-coefficients become $\pm 12$.

• Multiply equation 1 by $3$: $9x+12y=30$
• Multiply equation 2 by $4$: $8x-12y=4$

Critical detail: when you multiply an equation by a constant, you must multiply every term — including the right-hand-side constant. Forgetting to scale the constant is the most common mistake at this step.

Step 3 — add the equations. The $y$-terms cancel:

Step 4 — substitute back into either original equation. Using equation 1:

Solution: $(x,y)=(2,1)$.

Three possible outcomes

Every linear system has exactly one of three outcomes:

1. Unique solution. Lines intersect at one point. Most systems are like this.
2. No solution. Lines are parallel — they never intersect. Algebraically, you’ll arrive at a contradiction like $0=5$.
3. Infinitely many solutions. Lines are the same — every point on the line satisfies both equations. Algebraically, you’ll arrive at $0=0$.

Determinant of a 2×2 matrix

For a 2×2 matrix $\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)$, the determinant is

A 2×2 system has a unique solution exactly when this determinant is nonzero. If $ad-bc=0$, the two equations are either parallel (no solution) or identical (infinitely many solutions) — the same three-outcome trichotomy you just saw, read off in one number.

Connection to linear algebra

Solving a system of $n$ equations in $n$ variables is exactly the operation that matrix inversion performs. The system

has solution $\mathbf{x}=A^{-1}\mathbf{b}$ (when $A$ is invertible). Gaussian elimination generalizes the elimination method you just used to systems of any size.

Connection to machine learning

The closed-form solution to linear regression is a system of linear equations solved with matrices:

The matrix $X^{T}X$ encodes the system; $X^{T}y$ encodes the right-hand side; $\beta$ is the vector of coefficients you’re solving for. A numerical computing library’s linear-solve routine does this in one line — but conceptually, it’s the same operation as eliminating variables one at a time.

When you see a “singular matrix” error from such a routine, it means your system has no unique solution — the design matrix is rank-deficient, equivalent to the “no solution” or “infinitely many solutions” cases above.