dspy.LM¶
dspy.LM(model: str, model_type: Literal['auto', 'chat', 'text', 'responses'] = 'auto', temperature: float | None = None, max_tokens: int | None = None, cache: bool = True, callbacks: list[BaseCallback] | None = None, num_retries: int = 3, provider: Provider | None = None, finetuning_model: str | None = None, launch_kwargs: dict[str, Any] | None = None, train_kwargs: dict[str, Any] | None = None, use_developer_role: bool = False, engine: Engine | Literal['auto', 'lm15', 'litellm'] | None = 'auto', async_engine: AsyncEngine | None = None, prompt_cache: CacheConfig | None = None, **kwargs) ¶
Bases: BaseLM
A language model supporting chat or text completion requests for use with DSPy modules.
lm(request) runs one dspy.lm15.Request and returns a dspy.lm15.Response; adapters call it once per candidate. lm("hello") is a convenience that renders one user message with this LM’s generation defaults and returns a list of outputs. Custom backends are engines passed with engine=; see https://dspy.ai/community/normalized-lm-api-migration/.
Create a new language model instance for use with DSPy modules and programs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model | str | The model to use. This should be a string of the form | required |
model_type | Literal['auto', 'chat', 'text', 'responses'] | ‘auto’ (default) uses lm15’s native provider API family, independently of engine selection. OpenAI and Azure use Responses; other providers keep their native dialect. ‘chat’, ‘responses’, and ‘text’ retain explicit API selection. Unresolved compatibility routes use Chat Completions. Custom engines own their protocol. | 'auto' |
temperature | float | None | The sampling temperature to use when generating responses. | None |
max_tokens | int | None | The maximum number of tokens to generate per response. | None |
cache | bool | Whether to cache the model responses for reuse to improve performance and reduce costs. | True |
callbacks | list[BaseCallback] | None | A list of callback functions to run before and after each request. | None |
num_retries | int | The number of times to retry a request if it fails transiently due to network error, rate limiting, etc. Requests are retried with exponential backoff. | 3 |
engine | Engine | Literal['auto', 'lm15', 'litellm'] | None | ‘auto’ prefers lm15 for representable requests; ‘litellm’ preserves the compatibility backend; ‘lm15’ refuses unsupported mappings rather than selecting LiteLLM. A custom engine implements complete(Request) -> Response and optionally stream(Request). Engines are borrowed and own their connection: api_key, api_base, timeout and the other client settings are refused with a custom engine, at construction and on every call. | 'auto' |
async_engine | AsyncEngine | None | Async counterpart when supplying a custom engine object. The pair is one unit: copy(engine=…) replaces both unless async_engine= is given too. Pass engine=None for an async-only custom LM. Capability hints come from the sync engine when present, otherwise the async engine; each call mode requires its own engine. | None |
prompt_cache | CacheConfig | None | Optional lm15 CacheConfig for provider-side prompt caching on ordinary calls. Separate from DSPy’s response cache. Requires native lm15 or a canonical custom engine; may incur cache-write/storage charges. A call-time value overrides this default, and None removes it. Explicit Request calls use only Request.config.cache. No cache resource is created. | None |
provider | Provider | None | The training/launch provider. This does not select the inference engine. | None |
finetuning_model | str | None | The model to finetune. In some providers, the models available for finetuning is different from the models available for inference. | None |
rollout_id | Optional integer used to differentiate cache entries for otherwise identical requests. Different values bypass DSPy’s caches while still caching future calls with the same inputs and rollout ID. Note that | required |
Source code in dspy/clients/lm.py
157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 | |
Methods:¶
__call__(prompt: Request | str | None = None, *, cache: bool | None = None, rollout_id: Any = None, **options: Any) -> Response | list[str | dict[str, Any] | None] ¶
Run one model call.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompt | Request | str | None | A | None |
cache | bool | None | Override this LM’s response-cache setting for the call. | None |
rollout_id | Any | Distinguish otherwise identical cached requests. | None |
**options | Any | Generation options for a string prompt, in the OpenAI vocabulary ( | {} |
Returns:
| Type | Description |
|---|---|
Response | list[str | dict[str, Any] | None] | The |
Source code in dspy/clients/base_lm.py
acall(prompt: Request | str | None = None, *, cache: bool | None = None, rollout_id: Any = None, **options: Any) -> Response | list[str | dict[str, Any] | None] async ¶
Async equivalent of __call__, with the same execution ownership.
Source code in dspy/clients/base_lm.py
copy(**kwargs) ¶
dump_state() ¶
Return a sanitized reconstruction state for this LM.
A custom engine is recorded as its class path and its own dump_state(); the class must be importable by that path in the process that loads the state. A class defined in __main__ (a script or notebook) loads only where __main__ defines it again; for durable state define the engine in an importable module.
Returns:
| Type | Description |
|---|---|
| A dictionary that can be passed to | |
| reconstruct this |
Source code in dspy/clients/lm.py
finetune(train_data: list[dict[str, Any]], train_data_format: TrainDataFormat | None, train_kwargs: dict[str, Any] | None = None) -> TrainingJob ¶
Source code in dspy/clients/lm.py
infer_provider() -> Provider ¶
inspect_history(n: int = 1, file: TextIO | None = None) -> None ¶
kill(launch_kwargs: dict[str, Any] | None = None) ¶
launch(launch_kwargs: dict[str, Any] | None = None) ¶
load_state(state: dict[str, Any], *, allow_custom_lm_class: bool = False) classmethod ¶
Source code in dspy/clients/lm.py
reinforce(train_kwargs) -> ReinforceJob ¶
Source code in dspy/clients/lm.py
Choosing an engine¶
engine="auto" prefers native lm15 and may select the optional LiteLLM compatibility engine for unsupported routes, settings or representations before provider I/O. engine="lm15" requires native; engine="litellm" explicitly selects compatibility. Provider failures never trigger a backend switch. Native imports and calls do not require LiteLLM. Compatibility use requires pip install "dspy[litellm]"; without it, an actionable error names that command. model_type="text" uses compatibility, not native lm15. All engines retain the canonical Request/Response contract. See the migration guide for limitations.
Application-owned exception: text-only Bedrock Nova¶
Native Bedrock Nova/Converse is not supported by this DSPy release. The following recipe is a bounded, application-owned exception, not restored native support or an accepted migration-gate exception. A custom engine is not the only integration mechanism: optional LiteLLM compatibility also remains available. This recipe does not verify built-in Nova execution through that engine. It uses the public complete(Request) -> Response engine contract and LiteLLM installed through DSPy’s optional extra, pinned to the version tested here:
This example targets amazon.nova-lite-v1:0 in us-east-1, using AWS credentials configured for your application. It forwards a string system prompt and all user/assistant text blocks, including demonstrations. Only max_tokens and temperature (0–1) are supported. Tools, media, developer/tool messages, continuation state, structured outputs, reasoning, provider-side caching and other non-default generation settings are refused before the SDK call.
import asyncio
from dataclasses import replace
import dspy
import litellm
from openai import APIError # LiteLLM's SDK errors inherit this public type.
from dspy import lm15
MODEL = "bedrock/amazon.nova-lite-v1:0"
REGION = "us-east-1"
litellm.num_retries = 0 # Application-owned LiteLLM setting; DSPy owns retries.
class NovaText:
supported_params = {"max_tokens", "temperature"}
def complete(self, request: lm15.Request) -> lm15.Response:
config = request.config
if request.model != MODEL or request.tools or not isinstance(request.system, (str, type(None))):
raise lm15.UnsupportedFeatureError("Nova recipe requires its fixed model, no tools, and a string system prompt")
if replace(config, max_tokens=None, temperature=None) != lm15.Config():
raise lm15.UnsupportedFeatureError("Nova recipe supports only max_tokens and temperature")
if config.temperature is not None and config.temperature > 1:
raise lm15.UnsupportedFeatureError("Nova recipe requires temperature between 0 and 1")
messages = []
if request.system is not None:
messages.append({"role": "system", "content": request.system})
for message in request.messages:
if (message.role not in {"user", "assistant"} or message.text is None
or message.continuation or any(part.continuation for part in message.parts)):
raise lm15.UnsupportedFeatureError("Nova recipe accepts only text messages without continuation state")
messages.append({"role": message.role, "content": [
{"type": "text", "text": part.text} for part in message.parts
]})
options = {key: getattr(config, key) for key in self.supported_params if getattr(config, key) is not None}
try:
raw = litellm.completion(
model=MODEL, messages=messages, aws_region_name=REGION,
timeout=30, num_retries=0, caching=False, drop_params=False, **options,
)
except APIError as exc:
status = getattr(exc, "status_code", None)
error = {400: lm15.InvalidRequestError, 401: lm15.AuthError,
403: lm15.AuthError, 429: lm15.RateLimitError}.get(status, lm15.ProviderError)
if isinstance(exc, litellm.Timeout):
error = lm15.TimeoutError
elif isinstance(exc, litellm.APIConnectionError):
# LiteLLM also uses this class for unclassified HTTP failures.
error, status = lm15.ProviderError, None
elif status is not None and status >= 500:
error = lm15.ServerError
raise error(str(exc), provider="bedrock", status=status) from exc
response = lm15.response_from_openai_chat(raw.model_dump(), model=request.model)
if response.message.text is None:
raise lm15.UnsupportedFeatureError("Nova recipe received a non-text response")
return response
class AsyncNovaText(NovaText):
async def complete(self, request: lm15.Request) -> lm15.Response:
return await asyncio.to_thread(super().complete, request)
Use the pair for ordinary synchronous and asynchronous prediction:
lm = dspy.LM(MODEL, engine=NovaText(), async_engine=AsyncNovaText(),
temperature=0, max_tokens=128, cache=True, num_retries=2)
dspy.configure(lm=lm, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False))
program = dspy.Predict("question -> answer")
print(program(question="What is the capital of France?").answer)
print(asyncio.run(program.acall(question="What is the capital of Germany?")).answer)
In an existing event loop, use await program.acall(...) instead of asyncio.run(...). The async engine runs the same blocking SDK call in a worker thread; cancelling the await does not stop an in-flight HTTP request. The SDK timeout is 30 seconds. There is no streaming or engine serialization in this recipe; reattach the engine after loading a program saved without its LM.
DSPy owns response caching and retry limits; LiteLLM caching and retries are disabled here. Keep this application’s LiteLLM retry setting at zero and do not add SDK/router fallback or retry policies. Use an isolated DSPy cache for this deployment; changing AWS account, region or endpoint does not itself partition DSPy’s response cache. The public lm15 response reader preserves reported usage, finish reasons and provider data; this recipe makes no cost-accounting promise.
Retry classification is limited to what LiteLLM identifies: rate limits, timeouts and server failures are retryable. Its generic APIConnectionError can also represent an unclassified HTTP failure (observed with a local 403), so this recipe reports it as a non-retryable LMProviderError, without claiming an HTTP status. The SDK exception remains in the cause chain; connection failures are deliberately not retried automatically either.
Validation boundary: the copied recipe was executed through dspy.Predict with LiteLLM 1.96.0 against a disposable loopback Converse endpoint, with fake AWS credentials and locally checked SigV4 signatures. Sync/async text output, request model/body, local refusals, SDK failures, DSPy retries and cache hits were checked. This is local protocol validation with canned responses, not live AWS validation: AWS model availability, IAM permissions, regional support and real Nova output quality remain unverified. It does not establish acceptance of this exception by the migration gate.