CrucibleDatasets.Dataset (CrucibleDatasets v0.5.4)

View Source

Unified dataset representation across all benchmark types.

All datasets follow this schema regardless of source (MMLU, HumanEval, GSM8K, custom).

Summary

Types

expected_type()
input_type()
item()
t()

Functions

filter(dataset, predicate)

Filter dataset items by predicate.

new(name, version, items, metadata \\ %{})

Create a new dataset with validation.

shuffle_choices(dataset, opts \\ [])

Shuffle multiple-choice options while preserving correct answer mapping.

slice(dataset, range)

Slice dataset by index range or start/count.

slice(dataset, start, count)
sort(dataset, key_or_fn, order \\ :asc)

Sort dataset items by key function or atom.

validate(dataset)

Validate dataset schema.

Types

expected_type()

@type expected_type() ::
  String.t() | integer() | %{answer: String.t(), reasoning: String.t()}

input_type()

@type input_type() ::
  String.t()
  | %{question: String.t(), choices: [String.t()]}
  | %{signature: String.t(), tests: [String.t()]}

item()

@type item() :: %{
  :id => String.t(),
  :input => input_type(),
  :expected => expected_type(),
  optional(:metadata) => map()
}

t()

@type t() :: %CrucibleDatasets.Dataset{
  items: [item()],
  metadata: map(),
  name: String.t(),
  version: String.t()
}

Functions

filter(dataset, predicate)

@spec filter(t(), (item() -> boolean())) :: t()

Filter dataset items by predicate.

Examples

iex> dataset |> Dataset.filter(fn item ->
...>   item.metadata.difficulty == "hard"
...> end)

new(name, version, items, metadata \\ %{})

Create a new dataset with validation.

shuffle_choices(dataset, opts \\ [])

@spec shuffle_choices(
  t(),
  keyword()
) :: t()

Shuffle multiple-choice options while preserving correct answer mapping.

Options

  • :seed - Random seed for reproducible shuffling

Examples

iex> dataset |> Dataset.shuffle_choices(seed: 42)

slice(dataset, range)

@spec slice(t(), Range.t()) :: t()

Slice dataset by index range or start/count.

Examples

iex> dataset |> Dataset.slice(0..9)   # First 10 items
iex> dataset |> Dataset.slice(10, 5)  # 5 items starting at index 10

slice(dataset, start, count)

@spec slice(t(), non_neg_integer(), non_neg_integer()) :: t()

sort(dataset, key_or_fn, order \\ :asc)

@spec sort(t(), (item() -> term()) | atom(), :asc | :desc) :: t()

Sort dataset items by key function or atom.

Examples

iex> dataset |> Dataset.sort(fn item -> item.id end)
iex> dataset |> Dataset.sort(:id, :desc)

validate(dataset)

Validate dataset schema.