Why Constants Are Ignored in Log-Likelihood and the Role of Argmax

You might want to read this  before you read this article. 

In statistical modeling, particularly in Maximum Likelihood Estimation (MLE), the log-likelihood function is frequently used due to its mathematical convenience. One common step in simplifying the log-likelihood expression is ignoring additive constants that do not depend on the parameters being optimized. But why is this acceptable? And what does the notation argmax really signify? This article clarifies these important concepts.

The Log-Likelihood Function

Consider a binomial distribution modeling 7 heads in 10 coin tosses. The log-likelihood function for the probability of heads \( \theta \) is given by:

\[ \log L(\theta) = \log \binom{10}{7} + 7 \log \theta + 3 \log (1 - \theta) \]

The term \( \log \binom{10}{7} \) is a constant with respect to \( \theta \). Since it does not change as \( \theta \) changes, it has no influence on the location of the maximum. In optimization, we are interested in the parameter value that maximizes the function, not the absolute value of the function itself.

Thus, we simplify:

\[ \log L(\theta) \propto 7 \log \theta + 3 \log (1 - \theta) \]

where the symbol “\( \propto \)” means “proportional to” — indicating that the omitted constant has no bearing on optimization.

The Meaning of Argmax

In MLE, we often see the expression:

\[ \hat{\theta} = \arg\max_{\theta} \log L(\theta) \]

This is read as: "Find the value of \( \theta \) that maximizes the log-likelihood." The argmax operator returns the argument (in this case, the value of \( \theta \)) at which the function reaches its maximum.

This is conceptually different from simply writing:

\[ \max_{\theta} \log L(\theta) \]

which gives the maximum value of the function. In contrast, argmax tells us where that maximum occurs — which is exactly what we need to estimate parameters.

Conclusion

Dropping constants in the log-likelihood is not a shortcut — it is a mathematically valid simplification when optimizing with respect to parameters. The argmax operator formalizes this optimization by pointing to the value that best fits the data. Together, these tools form the backbone of likelihood-based inference in statistics and machine learning.