CrucibleDatasets.Loader (CrucibleDatasets v0.5.4)

View Source

Unified dataset loading with automatic source detection and caching.

Supports loading from:

  • HuggingFace datasets
  • GitHub repositories
  • Local files
  • HTTP URLs
  • CrucibleIR.DatasetRef structs

Telemetry Events

The loader emits the following telemetry events:

  • [:crucible_datasets, :load, :start] - Emitted when loading begins

    • Measurements: %{system_time: integer}

    • Metadata: %{dataset: atom | String.t(), source: atom}

  • [:crucible_datasets, :load, :stop] - Emitted when loading completes

    • Measurements: %{duration: integer} (native time units)

    • Metadata: %{dataset: atom | String.t(), source: atom, item_count: integer}

  • [:crucible_datasets, :load, :exception] - Emitted when loading fails

    • Measurements: %{duration: integer} (native time units)

    • Metadata: %{dataset: atom | String.t(), source: atom, kind: atom, reason: term}

  • [:crucible_datasets, :cache, :hit] - Emitted on cache hit

    • Measurements: %{}

    • Metadata: %{dataset: atom | String.t()}

  • [:crucible_datasets, :cache, :miss] - Emitted on cache miss

    • Measurements: %{}

    • Metadata: %{dataset: atom | String.t()}

Summary

Functions

invalidate_cache(dataset_name)

Invalidate cache for a dataset.

load(dataset_or_ref, opts \\ [])

Load a dataset by name with automatic caching.

Functions

invalidate_cache(dataset_name)

@spec invalidate_cache(atom() | String.t()) :: :ok

Invalidate cache for a dataset.

load(dataset_or_ref, opts \\ [])

@spec load(
  atom() | String.t() | CrucibleIR.DatasetRef.t(),
  keyword()
) :: {:ok, CrucibleDatasets.Dataset.t()} | {:error, term()}

Load a dataset by name with automatic caching.

Options

  • :version - Specific version (default: "1.0")
  • :subset - Subset name for multi-config datasets
  • :cache - Use cache (default: true)
  • :sample_size - Limit items (default: all)
  • :source - Custom source path for local datasets

Examples

iex> CrucibleDatasets.Loader.load(:mmlu_stem)
{:ok, %Dataset{name: "mmlu_stem", items: [...], ...}}

iex> CrucibleDatasets.Loader.load(:humaneval, sample_size: 50)
{:ok, %Dataset{name: "humaneval", items: [50 items], ...}}

iex> CrucibleDatasets.Loader.load("custom", source: "path/to/data.jsonl")
{:ok, %Dataset{name: "custom", ...}}

iex> ref = %CrucibleIR.DatasetRef{name: :mmlu_stem, split: :train, options: [sample_size: 100]}
iex> CrucibleDatasets.Loader.load(ref)
{:ok, %Dataset{name: "mmlu_stem", items: [...], ...}}