Btukfyl

In frequentist statistics, a 95% confidence interval is an interval-producing procedure that, if repeated an infinite number of times, would contain the true parameter 95% of the time. Why is this useful?

Confidence intervals are often misunderstood. They are not an interval that we can be 95% certain the parameter is in (unless you are using the similar Bayesian credibility interval). Confidence intervals feel like a bait-and-switch to me.

The one use case I can think of is to provide the range of values for which we could not reject the null hypothesis that the parameter is that value. Wouldn't p-values provide this information, but better? Without being so misleading?

In short: Why do we need confidence intervals? How are they, when correctly interpreted, useful?

So long as the confidence interval is treated as random (i.e., looked at from the perspective of treating the data as a set of random variables that we have not seen yet) then we can indeed make useful probability statements about it. Specifically, suppose you have a confidence interval at level $1-alpha$ for the parameter $theta$, and the interval has bounds $L(mathbf{x}) leqslant U(mathbf{x})$. Then we can say that:

$$mathbb{P}(L(mathbf{X}) leqslant theta leqslant U(mathbf{X}) | theta) = 1-alpha
quad quad quad text{for all } theta in Theta.$$

Moving outside the frequentist paradigm and marginalising over $theta$ for any prior distribution gives the corresponding marginal probability result:

$$mathbb{P}(L(mathbf{X}) leqslant theta leqslant U(mathbf{X})) = 1-alpha.$$

Once we fix the bounds of the confidence interval by fixing the data to $mathbf{X} = mathbb{x}$, we no longer appeal to this probability statement, because we now have fixed the data. However, if the confidence interval is treated as a random interval then we can indeed make this probability statement --- i.e., with probability $1-alpha$ the parameter $theta$ will fall within the (random) interval.

Within frequentist statistics, probability statements are statements about relative frequencies over infinitely repeated trials. But that is true of every probability statement in the frequentist paradigm, so if your objection is to relative frequency statements, that is not an objection that is specific to confidence intervals. If we move outside the frequentist paradigm then we can legitimately say that a confidence interval contains its target parameter with the desired probability, so long as we make this probability statement marginally (i.e., not conditional on the data) and we thus treat the confidence interval in its random sense.

I don't know about others, but that seems to me to be a pretty powerful probability result, and a reasonable justification for this form of interval. I am more partial to Bayesian methods myself, but the probability results backing confidence intervals (in their random sense) are powerful results that are not to be sniffed at.