Type Alias Criteria

Criteria: {
    evaluatorType: "criteria";
    feedbackKey: undefined | string;
    formatEvaluatorInputs: EvaluatorInputFormatter;
    agentTools?: undefined | StructuredToolInterface[];
    chainOptions?: undefined | Partial<Omit<LLMEvalChainInput<EvalOutputType, BaseLanguageModelInterface>, "llm">>;
    criteria?: undefined | (CriteriaLike & (Record<string, string> | Criteria));
    distanceMetric?: undefined | EmbeddingDistanceType;
    embedding?: any;
    llm?: any;
}

Configuration to load a "CriteriaEvalChain" evaluator, which prompts an LLM to determine whether the model's prediction complies with the provided criteria.

Type declaration

  • evaluatorType: "criteria"
  • feedbackKey: undefined | string
  • formatEvaluatorInputs: EvaluatorInputFormatter
  • OptionalagentTools?: undefined | StructuredToolInterface[]

    A list of tools available to the agent, for TrajectoryEvalChain.

  • OptionalchainOptions?: undefined | Partial<Omit<LLMEvalChainInput<EvalOutputType, BaseLanguageModelInterface>, "llm">>
  • Optionalcriteria?: undefined | (CriteriaLike & (Record<string, string> | Criteria))

    The criteria to use for the evaluator.

  • OptionaldistanceMetric?: undefined | EmbeddingDistanceType

    The distance metric to use for comparing the embeddings.

  • Optionalembedding?: any

    The embedding objects to vectorize the outputs.

  • Optionalllm?: any

Param: criteria

The criteria to use for the evaluator.

Param: llm

The language model to use for the evaluator.

Returns

The configuration for the evaluator.

Example

const evalConfig = {
  evaluators: [Criteria("helpfulness")],
};
@example
```ts
const evalConfig = {
  evaluators: [
    Criteria({
      "isCompliant": "Does the submission comply with the requirements of XYZ"
    })
  ],

Example

const evalConfig = {
  evaluators: [{
    evaluatorType: "criteria",
    criteria: "helpfulness"
    formatEvaluatorInputs: ...
  }]
};

Example

const evalConfig = {
  evaluators: [{
    evaluatorType: "criteria",
    criteria: { "isCompliant": "Does the submission comply with the requirements of XYZ" },
    formatEvaluatorInputs: ...
  }]

Settings

Member Visibility