A challenge-based benchmarking study is a structured research competition in which multiple teams or algorithms are evaluated on a common dataset using predefined tasks and metrics. The aim is to objectively compare performance across different methods under the same conditions.

Key Features: Public dataset: All participants use the same training and/or testing data.

Shared task: A specific problem is defined (e.g., classification, segmentation).

Standard metrics: All methods are assessed using the same evaluation criteria.

Leaderboard or ranking: Results are often ranked to determine which approach performs best.

Organized by a consortium or conference: Often linked to a scientific event or journal.

Example: In medical imaging, a challenge might involve segmenting brain tumors on MRI. Different teams submit their algorithms, which are then automatically scored against a gold standard. The study reports results and insights into what methods worked best.