Class ChatLlamaCpp

To use this model you need to have the node-llama-cpp module installed. This can be installed using npm install -S node-llama-cpp and the minimum version supported in version 2.0.0. This also requires that have a locally built version of Llama2 installed.

Example

// Initialize the ChatLlamaCpp model with the path to the model binary file.
const model = new ChatLlamaCpp({
  modelPath: "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin",
  temperature: 0.5,
});

// Call the model with a message and await the response.
const response = await model.call([
  new HumanMessage({ content: "My name is John." }),
]);

// Log the response to the console.
console.log({ response });

Hierarchy

SimpleChatModel<LlamaCppCallOptions>
- ChatLlamaCpp

Index

Constructors

constructor

new ChatLlamaCpp(inputs): ChatLlamaCpp
Parameters
- inputs: LlamaCppInputs
Returns ChatLlamaCpp
Overrides SimpleChatModel.constructor
- Defined in docs/api_refs/langchain/src/chat_models/llama_cpp.ts:83

Properties

CallOptions

CallOptions: LlamaCppCallOptions

ParsedCallOptions

ParsedCallOptions: Omit<LlamaCppCallOptions, never>

caller

caller: AsyncCaller

The async caller should be used by subclasses to make any async calls, which will thus benefit from the concurrency and retry logic.

verbose

verbose: boolean

Whether to print out response text.

`Optional` cache

cache?: BaseCache<Generation[]>

`Optional` callbacks

callbacks?: Callbacks

`Optional` maxTokens

maxTokens?: number

`Optional` metadata

metadata?: Record<string, unknown>

`Optional` tags

tags?: string[]

`Optional` temperature

temperature?: number

`Optional` topK

topK?: number

`Optional` topP

topP?: number

`Optional` trimWhitespaceSuffix

trimWhitespaceSuffix?: boolean

`Static` inputs

inputs: LlamaCppInputs

Accessors

callKeys

get callKeys(): string[]
Keys that the language model accepts as call options.

Returns string[]
Inherited from SimpleChatModel.callKeys
- Defined in docs/api_refs/langchain/src/base_language/index.ts:143

Methods

batch

batch(inputs, options?, batchOptions?): Promise<BaseMessageChunk[]>
Default implementation of batch, which calls invoke N times. Subclasses should override this method if they can batch more efficiently.
Parameters
- inputs: BaseLanguageModelInput[]
  
  Array of inputs to each batch call.
- Optional options: Partial<LlamaCppCallOptions> | Partial<LlamaCppCallOptions>[]
  
  Either a single call options object to apply to each batch call or an array for each call.
- Optional batchOptions: RunnableBatchOptions & {
  returnExceptions?: false;
  }
Returns Promise<BaseMessageChunk[]>
An array of RunOutputs, or mixed RunOutputs and errors if batchOptions.returnExceptions is set
Inherited from SimpleChatModel.batch
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:159
batch(inputs, options?, batchOptions?): Promise<(BaseMessageChunk | Error)[]>
Parameters
- inputs: BaseLanguageModelInput[]
- Optional options: Partial<LlamaCppCallOptions> | Partial<LlamaCppCallOptions>[]
- Optional batchOptions: RunnableBatchOptions & {
  returnExceptions: true;
  }
Returns Promise<(BaseMessageChunk | Error)[]>
Inherited from SimpleChatModel.batch
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:165
batch(inputs, options?, batchOptions?): Promise<(BaseMessageChunk | Error)[]>
Parameters
- inputs: BaseLanguageModelInput[]
- Optional options: Partial<LlamaCppCallOptions> | Partial<LlamaCppCallOptions>[]
- Optional batchOptions: RunnableBatchOptions
Returns Promise<(BaseMessageChunk | Error)[]>
Inherited from SimpleChatModel.batch
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:171

bind

bind(kwargs): Runnable<BaseLanguageModelInput, BaseMessageChunk, LlamaCppCallOptions>
Bind arguments to a Runnable, returning a new Runnable.
Parameters
- kwargs: Partial<LlamaCppCallOptions>
Returns Runnable<BaseLanguageModelInput, BaseMessageChunk, LlamaCppCallOptions>
A new RunnableBinding that, when invoked, will apply the bound args.
Inherited from SimpleChatModel.bind
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:67

call

call(messages, options?, callbacks?): Promise<BaseMessage>
Makes a single call to the chat model.
Parameters
- messages: BaseMessageLike[]
  
  An array of BaseMessage instances.
- Optional options: string[] | LlamaCppCallOptions
  
  The call options or an array of stop sequences.
- Optional callbacks: Callbacks
  
  The callbacks for the language model.
Returns Promise<BaseMessage>
A Promise that resolves to a BaseMessage.
Inherited from SimpleChatModel.call
- Defined in docs/api_refs/langchain/src/chat_models/base.ts:431

callPrompt

callPrompt(promptValue, options?, callbacks?): Promise<BaseMessage>
Makes a single call to the chat model with a prompt value.
Parameters
- promptValue: BasePromptValue
  
  The value of the prompt.
- Optional options: string[] | LlamaCppCallOptions
  
  The call options or an array of stop sequences.
- Optional callbacks: Callbacks
  
  The callbacks for the language model.
Returns Promise<BaseMessage>
A Promise that resolves to a BaseMessage.
Inherited from SimpleChatModel.callPrompt
- Defined in docs/api_refs/langchain/src/chat_models/base.ts:452

generate

generate(messages, options?, callbacks?): Promise<LLMResult>
Generates chat based on the input messages.
Parameters
- messages: BaseMessageLike[][]
  
  An array of arrays of BaseMessage instances.
- Optional options: string[] | LlamaCppCallOptions
  
  The call options or an array of stop sequences.
- Optional callbacks: Callbacks
  
  The callbacks for the language model.
Returns Promise<LLMResult>
A Promise that resolves to an LLMResult.
Inherited from SimpleChatModel.generate
- Defined in docs/api_refs/langchain/src/chat_models/base.ts:303

generatePrompt

generatePrompt(promptValues, options?, callbacks?): Promise<LLMResult>
Generates a prompt based on the input prompt values.
Parameters
- promptValues: BasePromptValue[]
  
  An array of BasePromptValue instances.
- Optional options: string[] | LlamaCppCallOptions
  
  The call options or an array of stop sequences.
- Optional callbacks: Callbacks
  
  The callbacks for the language model.
Returns Promise<LLMResult>
A Promise that resolves to an LLMResult.
Inherited from SimpleChatModel.generatePrompt
- Defined in docs/api_refs/langchain/src/chat_models/base.ts:406

getNumTokens

getNumTokens(content): Promise<number>
Parameters
- content: MessageContent
Returns Promise<number>
Inherited from SimpleChatModel.getNumTokens
- Defined in docs/api_refs/langchain/src/base_language/index.ts:201

invocationParams

invocationParams(): {
    maxTokens: undefined | number;
    temperature: undefined | number;
    topK: undefined | number;
    topP: undefined | number;
    trimWhitespaceSuffix: undefined | boolean;
}
Get the parameters used to invoke the model

Returns {
    maxTokens: undefined | number;
    temperature: undefined | number;
    topK: undefined | number;
    topP: undefined | number;
    trimWhitespaceSuffix: undefined | boolean;
}
- maxTokens: undefined | number
- temperature: undefined | number
- topK: undefined | number
- topP: undefined | number
- trimWhitespaceSuffix: undefined | boolean
Overrides SimpleChatModel.invocationParams
- Defined in docs/api_refs/langchain/src/chat_models/llama_cpp.ts:105

invoke

invoke(input, options?): Promise<BaseMessageChunk>
Invokes the chat model with a single input.
Parameters
- input: BaseLanguageModelInput
  
  The input for the language model.
- Optional options: LlamaCppCallOptions
  
  The call options.
Returns Promise<BaseMessageChunk>
A Promise that resolves to a BaseMessageChunk.
Inherited from SimpleChatModel.invoke
- Defined in docs/api_refs/langchain/src/chat_models/base.ts:119

map

map(): Runnable<BaseLanguageModelInput[], BaseMessageChunk[], LlamaCppCallOptions>
Return a new Runnable that maps a list of inputs to a list of outputs, by calling invoke() with each input.

Returns Runnable<BaseLanguageModelInput[], BaseMessageChunk[], LlamaCppCallOptions>
Inherited from SimpleChatModel.map
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:78

pipe

pipe<NewRunOutput>(coerceable): RunnableSequence<BaseLanguageModelInput, Exclude<NewRunOutput, Error>>
Create a new runnable sequence that runs each individual runnable in series, piping the output of one runnable into another runnable or runnable-like.
Type Parameters
- NewRunOutput
Parameters
- coerceable: RunnableLike<BaseMessageChunk, NewRunOutput>
  
  A runnable, function, or object whose values are functions or runnables.
Returns RunnableSequence<BaseLanguageModelInput, Exclude<NewRunOutput, Error>>
A new runnable sequence.
Inherited from SimpleChatModel.pipe
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:461

predict

predict(text, options?, callbacks?): Promise<string>
Predicts the next message based on a text input.
Parameters
- text: string
  
  The text input.
- Optional options: string[] | LlamaCppCallOptions
  
  The call options or an array of stop sequences.
- Optional callbacks: Callbacks
  
  The callbacks for the language model.
Returns Promise<string>
A Promise that resolves to a string.
Inherited from SimpleChatModel.predict
- Defined in docs/api_refs/langchain/src/chat_models/base.ts:483

predictMessages

predictMessages(messages, options?, callbacks?): Promise<BaseMessage>
Predicts the next message based on the input messages.
Parameters
- messages: BaseMessage[]
  
  An array of BaseMessage instances.
- Optional options: string[] | LlamaCppCallOptions
  
  The call options or an array of stop sequences.
- Optional callbacks: Callbacks
  
  The callbacks for the language model.
Returns Promise<BaseMessage>
A Promise that resolves to a BaseMessage.
Inherited from SimpleChatModel.predictMessages
- Defined in docs/api_refs/langchain/src/chat_models/base.ts:468

serialize

serialize(): SerializedLLM
Returns SerializedLLM

Deprecated
Return a json-like object representing this LLM.
Inherited from SimpleChatModel.serialize
- Defined in docs/api_refs/langchain/src/chat_models/base.ts:391

stream

stream(input, options?): Promise<IterableReadableStream<BaseMessageChunk>>
Stream output in chunks.
Parameters
- input: BaseLanguageModelInput
- Optional options: Partial<LlamaCppCallOptions>
Returns Promise<IterableReadableStream<BaseMessageChunk>>
A readable stream that is also an iterable.
Inherited from SimpleChatModel.stream
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:225

streamLog

streamLog(input, options?, streamOptions?): AsyncGenerator<RunLogPatch, any, unknown>
Stream all output from a runnable, as reported to the callback system. This includes all inner runs of LLMs, Retrievers, Tools, etc. Output is streamed as Log objects, which include a list of jsonpatch ops that describe how the state of the run has changed in each step, and the final state of the run. The jsonpatch ops can be applied in order to construct state.
Parameters
- input: BaseLanguageModelInput
- Optional options: Partial<LlamaCppCallOptions>
- Optional streamOptions: Omit<LogStreamCallbackHandlerInput, "autoClose">
Returns AsyncGenerator<RunLogPatch, any, unknown>
Inherited from SimpleChatModel.streamLog
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:507

toJSON

toJSON(): Serialized
Returns Serialized
Inherited from SimpleChatModel.toJSON
- Defined in docs/api_refs/langchain/src/load/serializable.ts:148

toJSONNotImplemented

toJSONNotImplemented(): SerializedNotImplemented
Returns SerializedNotImplemented
Inherited from SimpleChatModel.toJSONNotImplemented
- Defined in docs/api_refs/langchain/src/load/serializable.ts:222

transform

transform(generator, options): AsyncGenerator<BaseMessageChunk, any, unknown>
Default implementation of transform, which buffers input and then calls stream. Subclasses should override this method if they can start producing output while input is still being generated.
Parameters
- generator: AsyncGenerator<BaseLanguageModelInput, any, unknown>
- options: Partial<LlamaCppCallOptions>
Returns AsyncGenerator<BaseMessageChunk, any, unknown>
Inherited from SimpleChatModel.transform
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:478

withConfig

withConfig(config): RunnableBinding<BaseLanguageModelInput, BaseMessageChunk, LlamaCppCallOptions>
Bind config to a Runnable, returning a new Runnable.
Parameters
- config: BaseCallbackConfig
  
  New configuration parameters to attach to the new runnable.
Returns RunnableBinding<BaseLanguageModelInput, BaseMessageChunk, LlamaCppCallOptions>
A new RunnableBinding with a config matching what's passed.
Inherited from SimpleChatModel.withConfig
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:107

withFallbacks

withFallbacks(fields): RunnableWithFallbacks<BaseLanguageModelInput, BaseMessageChunk>
Create a new runnable from the current one that will try invoking other passed fallback runnables if the initial invocation fails.
Parameters
- fields: {
  fallbacks: Runnable<BaseLanguageModelInput, BaseMessageChunk, BaseCallbackConfig>[];
  }
  - fallbacks: Runnable<BaseLanguageModelInput, BaseMessageChunk, BaseCallbackConfig>[]
    
    Other runnables to call if the runnable errors.
Returns RunnableWithFallbacks<BaseLanguageModelInput, BaseMessageChunk>
A new RunnableWithFallbacks.
Inherited from SimpleChatModel.withFallbacks
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:124

withRetry

withRetry(fields?): RunnableRetry<BaseLanguageModelInput, BaseMessageChunk, LlamaCppCallOptions>
Add retry logic to an existing runnable.
Parameters
- Optional fields: {
  onFailedAttempt?: RunnableRetryFailedAttemptHandler;
  stopAfterAttempt?: number;
  }
  - Optional onFailedAttempt?: RunnableRetryFailedAttemptHandler
  - Optional stopAfterAttempt?: number
Returns RunnableRetry<BaseLanguageModelInput, BaseMessageChunk, LlamaCppCallOptions>
A new RunnableRetry that, when invoked, will retry according to the parameters.
Inherited from SimpleChatModel.withRetry
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:88

`Static` deserialize

deserialize(data): Promise<BaseLanguageModel<any, BaseLanguageModelCallOptions>>
Parameters
- data: SerializedLLM
Returns Promise<BaseLanguageModel<any, BaseLanguageModelCallOptions>>

Deprecated
Load an LLM from a json-like object describing it.
Inherited from SimpleChatModel.deserialize
- Defined in docs/api_refs/langchain/src/base_language/index.ts:295

`Static` isRunnable

isRunnable(thing): thing is Runnable<any, any, BaseCallbackConfig>
Parameters
- thing: any
Returns thing is Runnable<any, any, BaseCallbackConfig>
Inherited from SimpleChatModel.isRunnable
- Defined in docs/api_refs/langchain/src/schema/runnable/base.ts:557

Class ChatLlamaCpp

Example

Hierarchy

Index

Constructors

Properties

Accessors

Methods

Constructors

constructor

Parameters

inputs: LlamaCppInputs

Returns ChatLlamaCpp

Properties

CallOptions

ParsedCallOptions

caller

verbose

Optional cache

Optional callbacks

Optional maxTokens

Optional metadata

Optional tags

Optional temperature

Optional topK

Optional topP

Optional trimWhitespaceSuffix

Static inputs

Accessors

callKeys

Returns string[]

Methods

batch

Parameters

inputs: BaseLanguageModelInput[]

Optional options: Partial<LlamaCppCallOptions> | Partial<LlamaCppCallOptions>[]

Optional batchOptions: RunnableBatchOptions & { returnExceptions?: false; }

Returns Promise<BaseMessageChunk[]>

Parameters

inputs: BaseLanguageModelInput[]

Optional options: Partial<LlamaCppCallOptions> | Partial<LlamaCppCallOptions>[]

Optional batchOptions: RunnableBatchOptions & { returnExceptions: true; }

Returns Promise<(BaseMessageChunk | Error)[]>

Parameters

inputs: BaseLanguageModelInput[]

Optional options: Partial<LlamaCppCallOptions> | Partial<LlamaCppCallOptions>[]

Optional batchOptions: RunnableBatchOptions

Returns Promise<(BaseMessageChunk | Error)[]>

bind

Parameters

kwargs: Partial<LlamaCppCallOptions>

Returns Runnable<BaseLanguageModelInput, BaseMessageChunk, LlamaCppCallOptions>

call

Parameters

messages: BaseMessageLike[]

Optional options: string[] | LlamaCppCallOptions

Optional callbacks: Callbacks

Returns Promise<BaseMessage>

callPrompt

Parameters

promptValue: BasePromptValue

Optional options: string[] | LlamaCppCallOptions

Optional callbacks: Callbacks

Returns Promise<BaseMessage>

generate

Parameters

messages: BaseMessageLike[][]

Optional options: string[] | LlamaCppCallOptions

Optional callbacks: Callbacks

Returns Promise<LLMResult>

generatePrompt

Parameters

promptValues: BasePromptValue[]

Optional options: string[] | LlamaCppCallOptions

Optional callbacks: Callbacks

Returns Promise<LLMResult>

getNumTokens

Parameters

content: MessageContent

Returns Promise<number>

`Optional` cache

`Optional` callbacks

`Optional` maxTokens

`Optional` metadata

`Optional` tags

`Optional` temperature

`Optional` topK

`Optional` topP

`Optional` trimWhitespaceSuffix

`Static` inputs

`Optional` options: Partial<LlamaCppCallOptions> | Partial<LlamaCppCallOptions>[]

`Optional` batchOptions: RunnableBatchOptions & {
returnExceptions?: false;
}

`Optional` options: Partial<LlamaCppCallOptions> | Partial<LlamaCppCallOptions>[]

`Optional` batchOptions: RunnableBatchOptions & {
returnExceptions: true;
}

`Optional` options: Partial<LlamaCppCallOptions> | Partial<LlamaCppCallOptions>[]

`Optional` batchOptions: RunnableBatchOptions

`Optional` options: string[] | LlamaCppCallOptions

`Optional` callbacks: Callbacks

`Optional` options: string[] | LlamaCppCallOptions

`Optional` callbacks: Callbacks

`Optional` options: string[] | LlamaCppCallOptions

`Optional` callbacks: Callbacks

`Optional` options: string[] | LlamaCppCallOptions

`Optional` callbacks: Callbacks

Returns {
maxTokens: undefined | number;
temperature: undefined | number;
topK: undefined | number;
topP: undefined | number;
trimWhitespaceSuffix: undefined | boolean;
}

`Optional` options: LlamaCppCallOptions

`Optional` options: string[] | LlamaCppCallOptions

`Optional` callbacks: Callbacks

`Optional` options: string[] | LlamaCppCallOptions

`Optional` callbacks: Callbacks

`Optional` options: Partial<LlamaCppCallOptions>

`Optional` options: Partial<LlamaCppCallOptions>

`Optional` streamOptions: Omit<LogStreamCallbackHandlerInput, "autoClose">

fields: {
fallbacks: Runnable<BaseLanguageModelInput, BaseMessageChunk, BaseCallbackConfig>[];
}

`Optional` fields: {
onFailedAttempt?: RunnableRetryFailedAttemptHandler;
stopAfterAttempt?: number;
}

`Optional` onFailedAttempt?: RunnableRetryFailedAttemptHandler

`Optional` stopAfterAttempt?: number