Nonsmoothness of Classifier Error Confidence Intervals

Definitions and mathematical statement

For a conventional interpretation of classification test error, let $P$ be the distribution of a test observation $(X,Y)$, and let $h$ be a classifier taking feature values to labels. Whenever the misclassification event is measurable, define \[ R_P(h)=P\{h(X)\ne Y\}. \] Let $D$ denote training data and let $\widehat h_D=A(D)$ be the classifier produced by a learning rule $A$. Under this interpretation, the test error of the learned classifier is the random quantity $R_P(\widehat h_D)$, evaluated on a fresh test observation with law $P$. For a resampling-based interval $C_{1-\alpha}(D)$, with nominal confidence level $1-\alpha$, where $0<\alpha<1$, its coverage for this target is \[ \Pr\!\left\{R_P(\widehat h_D)\in C_{1-\alpha}(D)\right\}, \] where probability includes the randomness of training and any randomized interval construction. This notation supplies a conventional interpretation, not hypotheses specified by the source.

Source conjecture. The poor coverage performance of resampling-based confidence intervals for a learned classifier's test error is partly caused by test error being a nonsmooth functional of the learned classifier.

The conjecture asks whether nonsmooth dependence in $h\mapsto R_P(h)$ contributes to the failure of such intervals to achieve their nominal coverage. The supplied record does not specify the learning rule, data-generating assumptions, resampling procedure, topology or notion of smoothness, finite-sample or asymptotic regime, or a quantitative meaning of "partly caused." It also does not explicitly distinguish the conditional error $R_P(\widehat h_D)$ from an error averaged over training data. Consequently, the source supports this qualitative conjecture, but does not determine a uniquely quantified theorem or a universal assertion about all classifiers.

Short English statement

The conjecture asks whether nonsmooth changes in a learned classifier's test error help explain poor coverage of resampling-based confidence intervals. The source leaves the statistical setting and the proposed causal mechanism insufficiently specified for a single precise theorem. Its supplied status is partially solved; this is a dated curation assessment requiring independent review.

Sources

• Supplied local UnsolvedMath registry record, permanent ID $30001324$, source problem code catalogue entry, title "Nonsmoothness of Classifier Error Confidence Intervals," in the Oberwolfach Reports Open Problems collection. No individual proposer is supplied; the proposed year is recorded as 2009.
• Supplied report citation: Challenges in Statistical Theory: Complex Data Structures and Algorithmic Optimization (2010), Oberwolfach Reports, pp. 2179--2234, DOI 10.4171/owr/2009/39. https://doi.org/10.4171/owr/2009/39. The literature-source label is "original Oberwolfach report 4082."
• Supplied literature source: "Laber--Murphy adaptive intervals." https://pmc.ncbi.nlm.nih.gov/articles/PMC3285493/.
• Literature assessment checked at 2026-08-21: supplied dated curation, not independently verified here. It labels the problem partially solved and reports that the nonsmoothness mechanism has been rigorously analyzed and adaptive confidence intervals constructed for test error, while the broad explanatory conjecture is not a complete theorem for all classifiers. The record explicitly requests independent verification of time-sensitive status and cited claims.
• The registry labels the statement corrected_verified and reports that its mathematical content was checked against the official Oberwolfach report, with notation, context, and question form normalized without changing the source-level claim. This is supplied verification provenance, not an independent verification performed for this draft.