Class HuggingFaceInference

Class implementing the Large Language Model (LLM) interface using the Hugging Face Inference API for text generation.

Example

const model = new HuggingFaceInference({
  model: "gpt2",
  temperature: 0.7,
  maxTokens: 50,
});

const res = await model.call(
  "Question: What would be a good company name for a company that makes colorful socks?\nAnswer:"
);
console.log({ res });

Hierarchy

LLM
- HuggingFaceInference

Implements

HFInput

Index

Constructors

constructor

new HuggingFaceInference(fields?): HuggingFaceInference
Parameters
- Optional fields: Partial<HFInput> & BaseLLMParams
Returns HuggingFaceInference
Overrides LLM.constructor
- Defined in docs/api_refs/langchain/src/llms/hf.ts:77

Properties

CallOptions

CallOptions: BaseLLMCallOptions

ParsedCallOptions

ParsedCallOptions: Omit<BaseLLMCallOptions, never>

apiKey

apiKey: undefined | string = undefined

API key to use.

caller

caller: AsyncCaller

The async caller should be used by subclasses to make any async calls, which will thus benefit from the concurrency and retry logic.

endpointUrl

endpointUrl: undefined | string = undefined

Custom inference endpoint URL to use

frequencyPenalty

frequencyPenalty: undefined | number = undefined

Penalizes repeated tokens according to frequency

maxTokens

maxTokens: undefined | number = undefined

Maximum number of tokens to generate in the completion.

model

model: string = "gpt2"

Model to use

temperature

temperature: undefined | number = undefined

Sampling temperature to use

topK

topK: undefined | number = undefined

Integer to define the top tokens considered within the sample operation to create new text.

topP

topP: undefined | number = undefined

Total probability mass of tokens to consider at each step

verbose

verbose: boolean

Whether to print out response text.

`Optional` cache

cache?: BaseCache<Generation[]>

`Optional` callbacks

callbacks?: Callbacks

`Optional` metadata

metadata?: Record<string, unknown>

`Optional` tags

tags?: string[]

Accessors

callKeys

get callKeys(): string[]
Keys that the language model accepts as call options.

Returns string[]
Inherited from LLM.callKeys
- Defined in docs/api_refs/langchain/src/base_language/index.ts:143

Methods

batch

batch(inputs, options?, batchOptions?): Promise<string[]>
Default implementation of batch, which calls invoke N times. Subclasses should override this method if they can batch more efficiently.
Parameters
- inputs: BaseLanguageModelInput[]
  
  Array of inputs to each batch call.
- Optional options: Partial<BaseLLMCallOptions> | Partial<BaseLLMCallOptions>[]
  
  Either a single call options object to apply to each batch call or an array for each call.
- Optional batchOptions: RunnableBatchOptions & {
  returnExceptions?: false;
  }
Returns Promise<string[]>
An array of RunOutputs, or mixed RunOutputs and errors if batchOptions.returnExceptions is set
Inherited from LLM.batch
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:159
batch(inputs, options?, batchOptions?): Promise<(string | Error)[]>
Parameters
- inputs: BaseLanguageModelInput[]
- Optional options: Partial<BaseLLMCallOptions> | Partial<BaseLLMCallOptions>[]
- Optional batchOptions: RunnableBatchOptions & {
  returnExceptions: true;
  }
Returns Promise<(string | Error)[]>
Inherited from LLM.batch
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:165
batch(inputs, options?, batchOptions?): Promise<(string | Error)[]>
Parameters
- inputs: BaseLanguageModelInput[]
- Optional options: Partial<BaseLLMCallOptions> | Partial<BaseLLMCallOptions>[]
- Optional batchOptions: RunnableBatchOptions
Returns Promise<(string | Error)[]>
Inherited from LLM.batch
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:171

bind

bind(kwargs): Runnable<BaseLanguageModelInput, string, BaseLLMCallOptions>
Bind arguments to a Runnable, returning a new Runnable.
Parameters
- kwargs: Partial<BaseLLMCallOptions>
Returns Runnable<BaseLanguageModelInput, string, BaseLLMCallOptions>
A new RunnableBinding that, when invoked, will apply the bound args.
Inherited from LLM.bind
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:67

call

call(prompt, options?, callbacks?): Promise<string>
Convenience wrapper for generate that takes in a single string prompt and returns a single string output.
Parameters
- prompt: string
- Optional options: string[] | BaseLLMCallOptions
- Optional callbacks: Callbacks
Returns Promise<string>
Inherited from LLM.call
- Defined in docs/api_refs/langchain/src/llms/base.ts:359

generate

generate(prompts, options?, callbacks?): Promise<LLMResult>
Run the LLM on the given prompts and input, handling caching.
Parameters
- prompts: string[]
- Optional options: string[] | BaseLLMCallOptions
- Optional callbacks: Callbacks
Returns Promise<LLMResult>
Inherited from LLM.generate
- Defined in docs/api_refs/langchain/src/llms/base.ts:298

generatePrompt

generatePrompt(promptValues, options?, callbacks?): Promise<LLMResult>
This method takes prompt values, options, and callbacks, and generates a result based on the prompts.
Parameters
- promptValues: BasePromptValue[]
  
  Prompt values for the LLM.
- Optional options: string[] | BaseLLMCallOptions
  
  Options for the LLM call.
- Optional callbacks: Callbacks
  
  Callbacks for the LLM call.
Returns Promise<LLMResult>
An LLMResult based on the prompts.
Inherited from LLM.generatePrompt
- Defined in docs/api_refs/langchain/src/llms/base.ts:182

getNumTokens

getNumTokens(content): Promise<number>
Parameters
- content: MessageContent
Returns Promise<number>
Inherited from LLM.getNumTokens
- Defined in docs/api_refs/langchain/src/base_language/index.ts:201

invocationParams

invocationParams(_options?): any
Get the parameters used to invoke the model
Parameters
- Optional _options: Omit<BaseLLMCallOptions, never>
Returns any
Inherited from LLM.invocationParams
- Defined in docs/api_refs/langchain/src/llms/base.ts:207

invoke

invoke(input, options?): Promise<string>
This method takes an input and options, and returns a string. It converts the input to a prompt value and generates a result based on the prompt.
Parameters
- input: BaseLanguageModelInput
  
  Input for the LLM.
- Optional options: BaseLLMCallOptions
  
  Options for the LLM call.
Returns Promise<string>
A string result based on the prompt.
Inherited from LLM.invoke
- Defined in docs/api_refs/langchain/src/llms/base.ts:66

map

map(): Runnable<BaseLanguageModelInput[], string[], BaseLLMCallOptions>
Return a new Runnable that maps a list of inputs to a list of outputs, by calling invoke() with each input.

Returns Runnable<BaseLanguageModelInput[], string[], BaseLLMCallOptions>
Inherited from LLM.map
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:78

pipe

pipe<NewRunOutput>(coerceable): RunnableSequence<BaseLanguageModelInput, Exclude<NewRunOutput, Error>>
Create a new runnable sequence that runs each individual runnable in series, piping the output of one runnable into another runnable or runnable-like.
Type Parameters
- NewRunOutput
Parameters
- coerceable: RunnableLike<string, NewRunOutput>
  
  A runnable, function, or object whose values are functions or runnables.
Returns RunnableSequence<BaseLanguageModelInput, Exclude<NewRunOutput, Error>>
A new runnable sequence.
Inherited from LLM.pipe
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:461

predict

predict(text, options?, callbacks?): Promise<string>
This method is similar to call, but it's used for making predictions based on the input text.
Parameters
- text: string
  
  Input text for the prediction.
- Optional options: string[] | BaseLLMCallOptions
  
  Options for the LLM call.
- Optional callbacks: Callbacks
  
  Callbacks for the LLM call.
Returns Promise<string>
A prediction based on the input text.
Inherited from LLM.predict
- Defined in docs/api_refs/langchain/src/llms/base.ts:376

predictMessages

predictMessages(messages, options?, callbacks?): Promise<BaseMessage>
This method takes a list of messages, options, and callbacks, and returns a predicted message.
Parameters
- messages: BaseMessage[]
  
  A list of messages for the prediction.
- Optional options: string[] | BaseLLMCallOptions
  
  Options for the LLM call.
- Optional callbacks: Callbacks
  
  Callbacks for the LLM call.
Returns Promise<BaseMessage>
A predicted message based on the list of messages.
Inherited from LLM.predictMessages
- Defined in docs/api_refs/langchain/src/llms/base.ts:392

serialize

serialize(): SerializedLLM
Returns SerializedLLM

Deprecated
Return a json-like object representing this LLM.
Inherited from LLM.serialize
- Defined in docs/api_refs/langchain/src/llms/base.ts:421

stream

stream(input, options?): Promise<IterableReadableStream<string>>
Stream output in chunks.
Parameters
- input: BaseLanguageModelInput
- Optional options: Partial<BaseLLMCallOptions>
Returns Promise<IterableReadableStream<string>>
A readable stream that is also an iterable.
Inherited from LLM.stream
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:225

streamLog

streamLog(input, options?, streamOptions?): AsyncGenerator<RunLogPatch, any, unknown>
Stream all output from a runnable, as reported to the callback system. This includes all inner runs of LLMs, Retrievers, Tools, etc. Output is streamed as Log objects, which include a list of jsonpatch ops that describe how the state of the run has changed in each step, and the final state of the run. The jsonpatch ops can be applied in order to construct state.
Parameters
- input: BaseLanguageModelInput
- Optional options: Partial<BaseLLMCallOptions>
- Optional streamOptions: Omit<LogStreamCallbackHandlerInput, "autoClose">
Returns AsyncGenerator<RunLogPatch, any, unknown>
Inherited from LLM.streamLog
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:507

toJSON

toJSON(): Serialized
Returns Serialized
Inherited from LLM.toJSON
- Defined in docs/api_refs/langchain/src/load/serializable.ts:148

toJSONNotImplemented

toJSONNotImplemented(): SerializedNotImplemented
Returns SerializedNotImplemented
Inherited from LLM.toJSONNotImplemented
- Defined in docs/api_refs/langchain/src/load/serializable.ts:222

transform

transform(generator, options): AsyncGenerator<string, any, unknown>
Default implementation of transform, which buffers input and then calls stream. Subclasses should override this method if they can start producing output while input is still being generated.
Parameters
- generator: AsyncGenerator<BaseLanguageModelInput, any, unknown>
- options: Partial<BaseLLMCallOptions>
Returns AsyncGenerator<string, any, unknown>
Inherited from LLM.transform
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:478

withConfig

withConfig(config): RunnableBinding<BaseLanguageModelInput, string, BaseLLMCallOptions>
Bind config to a Runnable, returning a new Runnable.
Parameters
- config: BaseCallbackConfig
  
  New configuration parameters to attach to the new runnable.
Returns RunnableBinding<BaseLanguageModelInput, string, BaseLLMCallOptions>
A new RunnableBinding with a config matching what's passed.
Inherited from LLM.withConfig
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:107

withFallbacks

withFallbacks(fields): RunnableWithFallbacks<BaseLanguageModelInput, string>
Create a new runnable from the current one that will try invoking other passed fallback runnables if the initial invocation fails.
Parameters
- fields: {
  fallbacks: Runnable<BaseLanguageModelInput, string, BaseCallbackConfig>[];
  }
  - fallbacks: Runnable<BaseLanguageModelInput, string, BaseCallbackConfig>[]
    
    Other runnables to call if the runnable errors.
Returns RunnableWithFallbacks<BaseLanguageModelInput, string>
A new RunnableWithFallbacks.
Inherited from LLM.withFallbacks
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:124

withRetry

withRetry(fields?): RunnableRetry<BaseLanguageModelInput, string, BaseLLMCallOptions>
Add retry logic to an existing runnable.
Parameters
- Optional fields: {
  onFailedAttempt?: RunnableRetryFailedAttemptHandler;
  stopAfterAttempt?: number;
  }
  - Optional onFailedAttempt?: RunnableRetryFailedAttemptHandler
  - Optional stopAfterAttempt?: number
Returns RunnableRetry<BaseLanguageModelInput, string, BaseLLMCallOptions>
A new RunnableRetry that, when invoked, will retry according to the parameters.
Inherited from LLM.withRetry
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:88

`Static` deserialize

deserialize(data): Promise<BaseLLM<BaseLLMCallOptions>>
Parameters
- data: SerializedLLM
Returns Promise<BaseLLM<BaseLLMCallOptions>>

Deprecated
Load an LLM from a json-like object describing it.
Inherited from LLM.deserialize
- Defined in docs/api_refs/langchain/src/llms/base.ts:438

`Static` isRunnable

isRunnable(thing): thing is Runnable<any, any, BaseCallbackConfig>
Parameters
- thing: any
Returns thing is Runnable<any, any, BaseCallbackConfig>
Inherited from LLM.isRunnable
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:557

Class HuggingFaceInference

Example

Hierarchy

Implements

Index

Constructors

Properties

Accessors

Methods

Constructors

constructor

Parameters

Optional fields: Partial<HFInput> & BaseLLMParams

Returns HuggingFaceInference

Properties

CallOptions

ParsedCallOptions

apiKey

caller

endpointUrl

frequencyPenalty

maxTokens

model

temperature

topK

topP

verbose

Optional cache

Optional callbacks

Optional metadata

Optional tags

Accessors

callKeys

Returns string[]

Methods

batch

Parameters

inputs: BaseLanguageModelInput[]

Optional options: Partial<BaseLLMCallOptions> | Partial<BaseLLMCallOptions>[]

Optional batchOptions: RunnableBatchOptions & { returnExceptions?: false; }

Returns Promise<string[]>

Parameters

inputs: BaseLanguageModelInput[]

Optional options: Partial<BaseLLMCallOptions> | Partial<BaseLLMCallOptions>[]

Optional batchOptions: RunnableBatchOptions & { returnExceptions: true; }

Returns Promise<(string | Error)[]>

Parameters

inputs: BaseLanguageModelInput[]

Optional options: Partial<BaseLLMCallOptions> | Partial<BaseLLMCallOptions>[]

Optional batchOptions: RunnableBatchOptions

Returns Promise<(string | Error)[]>

bind

Parameters

kwargs: Partial<BaseLLMCallOptions>

Returns Runnable<BaseLanguageModelInput, string, BaseLLMCallOptions>

call

Parameters

prompt: string

Optional options: string[] | BaseLLMCallOptions

Optional callbacks: Callbacks

Returns Promise<string>

generate

Parameters

prompts: string[]

Optional options: string[] | BaseLLMCallOptions

Optional callbacks: Callbacks

Returns Promise<LLMResult>

generatePrompt

Parameters

promptValues: BasePromptValue[]

Optional options: string[] | BaseLLMCallOptions

Optional callbacks: Callbacks

Returns Promise<LLMResult>

getNumTokens

Parameters

content: MessageContent

Returns Promise<number>

invocationParams

Parameters

Optional _options: Omit<BaseLLMCallOptions, never>

`Optional` fields: Partial<HFInput> & BaseLLMParams

`Optional` cache

`Optional` callbacks

`Optional` metadata

`Optional` tags

`Optional` options: Partial<BaseLLMCallOptions> | Partial<BaseLLMCallOptions>[]

`Optional` batchOptions: RunnableBatchOptions & {
returnExceptions?: false;
}

`Optional` options: Partial<BaseLLMCallOptions> | Partial<BaseLLMCallOptions>[]

`Optional` batchOptions: RunnableBatchOptions & {
returnExceptions: true;
}

`Optional` options: Partial<BaseLLMCallOptions> | Partial<BaseLLMCallOptions>[]

`Optional` batchOptions: RunnableBatchOptions

`Optional` options: string[] | BaseLLMCallOptions

`Optional` callbacks: Callbacks

`Optional` options: string[] | BaseLLMCallOptions

`Optional` callbacks: Callbacks

`Optional` options: string[] | BaseLLMCallOptions

`Optional` callbacks: Callbacks

`Optional` _options: Omit<BaseLLMCallOptions, never>

`Optional` options: BaseLLMCallOptions

`Optional` options: string[] | BaseLLMCallOptions

`Optional` callbacks: Callbacks

`Optional` options: string[] | BaseLLMCallOptions

`Optional` callbacks: Callbacks

`Optional` options: Partial<BaseLLMCallOptions>

`Optional` options: Partial<BaseLLMCallOptions>

`Optional` streamOptions: Omit<LogStreamCallbackHandlerInput, "autoClose">

fields: {
fallbacks: Runnable<BaseLanguageModelInput, string, BaseCallbackConfig>[];
}

`Optional` fields: {
onFailedAttempt?: RunnableRetryFailedAttemptHandler;
stopAfterAttempt?: number;
}

`Optional` onFailedAttempt?: RunnableRetryFailedAttemptHandler

`Optional` stopAfterAttempt?: number

`Static` deserialize

`Static` isRunnable