@spec load(
  String.t(),
  keyword()
) :: {:ok, CrucibleDatasets.Dataset.t()} | {:error, term()}

Load a dataset from a file with field mapping.

Options

• :name - Dataset name (default: filename without extension)
• :version - Dataset version (default: "1.0.0")
• :format - File format (:jsonl, :json, :csv, or auto-detect)
• :fields - FieldMapping specification (default: FieldMapping.new())
• :auto_id - Auto-generate IDs for items (default: true)
• :limit - Maximum number of items to load
• :shuffle - Shuffle items after loading (default: false)
• :seed - Random seed for shuffling

Examples

iex> mapping = FieldMapping.new(input: "question", expected: "answer")
iex> {:ok, dataset} = Loader.Generic.load("data.jsonl", fields: mapping)

iex> {:ok, dataset} = Loader.Generic.load("data.csv",
...>   name: "my_dataset",
...>   format: :csv,
...>   fields: FieldMapping.new(
...>     input: "question",
...>     expected: "answer",
...>     metadata: ["difficulty"]
...>   ),
...>   limit: 100
...> )