Meet BigCodeBench by BigCode: The New Gold Standard for Evaluating Large Language Models on Real-World Coding Tasks

BigCodeBench evaluates LLMs using the calibrated Pass@1 metric, which assesses the percentage of tasks correctly solved on the first attempt2. This metric is refined using an Elo rating system to rank models based on their performance across various tasks2.

BigCodeBench addresses limitations of existing benchmarks by providing a more comprehensive and challenging set of 1,140 function-level tasks that require complex reasoning and problem-solving skills. It covers diverse libraries and function calls, and includes two components: BigCodeBench-Complete for code completion and BigCodeBench-Instruct for instruction-tuned LLMs. This benchmark aims to better evaluate LLMs' programming capabilities in real-world scenarios.