Modalities
Traditionally, LLMs map text input to text output. Newer multimodal foundation models can accept additional modalities such as images and, in some cases, audio.
There are also many scenarios where you generate non‑text outputs from text prompts (images or speech) using diffusion models or Text-to-Speech (TTS) models. Below is an illustration of the input/output permutations across the three common modalities. This is by no means a comprehensive coverage on all of the different pathways possible but should serve to illustrate the overall pattern.
flowchart LR
%% Inputs
T_IN[Text]
I_IN["Image"]
A_IN["Audio"]
ADD((+))
%% Outputs
T_OUT>"Text"]
T_OUT2>"Text"]
I_OUT>"Image"]
I_OUT2>"Image"]
A_OUT>"Audio"]
A_OUT2>"Audio"]
%% Paths / Models
LLM{{"LLM"}}
TTS{{"Text-to-Audio"}}
TTS2{{"Text-to-Audio"}}
MLLM{{"Multimodal LLM"}}
DIFF{{"Diffusion Models"}}
I2I{{"Diffusion Models/GANs/VAEs"}}
%% Connections
T_IN --> LLM --> T_OUT
T_IN --> TTS --> A_OUT
T_IN --> ADD
I_IN --> ADD
A_IN --> ADD
A_IN --> T_OUT2
ADD --> MLLM
MLLM --> DIFF --> I_OUT
MLLM --> TTS2 --> A_OUT2
MLLM --> T_OUT2
I_IN --> I2I --> I_OUT2
%% === COLOR THEMING ===
%% (Separate comments — not inline)
classDef text fill:#60A5FA,fill-opacity:0.3
classDef image fill:#34D399,fill-opacity:0.3
classDef audio fill:#FBBF24,fill-opacity:0.3
classDef model fill:#FECACA,fill-opacity:0.3
classDef add fill:#BFDBFE,fill-opacity:0.3
%% Apply consistent color classes
class T_IN,T_OUT,T_OUT2 text;
class I_IN,I_OUT,I_OUT2 image;
class A_IN,A_OUT,A_OUT2 audio;
class ADD add;
class LLM,VLM,T2I,TTS,TTS2,A2T,MLLM_TI,MLLM_TA,MLLM,DIFF,I2I model;
As you can imagine, covering all of these different paths while maintaining a relatively flexbile and developer friendly public API for any framework would be challening. Therefore, currently for the agents built using our framework we support text, vision, and PDF documents for input and only text as output. This won't prevent users from having an image generation model wrapped inside a Tool and giving the agent access to this.
If you would like to see other modalities supported on either input or output side of the equation, we'd welcome your contributions or discussions for such feature requests.
Text
Text is the default modality supported in Railtracks. No special operations are needed and you can simply follow the rest of the docs.
Image
Given the LLM powering your agent, you can pass image inputs to your multimodal agent. The only required step is to construct a UserMessage and pass in the parameter attachment
message_history = rt.llm.MessageHistory(
[
rt.llm.UserMessage(
content="What is the image below showing?",
attachment="https://cdn.britannica.com/39/226539-050-D21D7721/Portrait-of-a-cat-with-whiskers-visible.jpg",
)
]
)
attachment parameter can be a single str or a list[str]. We currently support the following:
- Both file and web URLs of the following types:
jpeg,png,gif,webp, andavif(OpenAI, Anthropic and Gemini do not acceptavif, but some OpenAI-compatible servers do) byte64encoded string of the image
A web URL without a file extension (for example a CDN link) is passed to the provider as an image. A URL whose extension names an audio, video, text or Office document file (such as .mp3, .mp4 or .docx) raises a ValueError, since those formats are not supported as attachments yet. With trust_urls=True, such a URL is HEAD-probed first and accepted when the server reports an image or PDF, for example a thumbnail service that renders a frame from a .mp4 path.
PDF documents
PDF input works through the same attachment parameter. The provider must natively support PDF input (currently OpenAI's gpt-5.x family and Anthropic's Claude models via the file content block); other providers will reject the request.
message_history = rt.llm.MessageHistory(
[
rt.llm.UserMessage(
content="Summarize the key findings in this paper.",
attachment="path/to/local/paper.pdf",
)
]
)
Supported PDF sources:
- Local file paths ending in
.pdf - Base64-encoded PDF payloads or full
data:application/pdf;base64,...URIs - URLs ending in
.pdf, only withtrust_urls=True
PDFs reach the provider inline as base64, so Railtracks downloads a PDF URL in your process before the call. That download is opt-in, and only for URLs you trust (see Attachment URLs in production):
# URL PDFs are fetched in-process, so they're opt-in via trust_urls=True.
# Only set this when the URL is developer-controlled (hardcoded, internal,
# trusted CDN) — never for URLs that originated from end-user input.
message_history = rt.llm.MessageHistory(
[
rt.llm.UserMessage(
content="Summarize the key findings in this paper.",
attachment="https://arxiv.org/pdf/1706.03762v7.pdf",
trust_urls=True,
)
]
)
Tuning the fetch timeout and size limit
The HEAD probe and PDF download share a per-request timeout that defaults to 10 seconds. Raise it via attachment_timeout for large PDFs over slow links. The download stops once it passes attachment_max_mb (32 MB by default) and raises a ValueError, so a very large or never-ending response cannot exhaust memory:
# Per-request timeout (seconds) for the HEAD probe and PDF download.
# Defaults to 10; raise it for large PDFs over slow links. The download
# stops at attachment_max_mb (defaults to 32). Both only apply when
# trust_urls=True.
message_history = rt.llm.MessageHistory(
[
rt.llm.UserMessage(
content="Summarize this paper.",
attachment="https://internal.example.com/large-report.pdf",
trust_urls=True,
attachment_timeout=60,
attachment_max_mb=64,
)
]
)
Both settings only apply when trust_urls=True. With trust_urls=False, Railtracks itself makes no network requests. With trust_urls=True, it makes none for local files, data URIs and image URLs with a recognized extension.
Attachment URLs in production
Railtracks doesn't check where an attachment URL points. Who downloads it depends on the attachment:
| Attachment | Downloaded by |
|---|---|
| Local file, base64 or data URI | Nobody. Railtracks reads the file or uses the bytes as they are. |
| Image URL | The provider, in most cases. When the provider can't take the URL, litellm (which Railtracks uses to call providers) downloads it in your process first, whatever trust_urls is set to: every image URL for Gemini, and http:// URLs for Anthropic. litellm refuses URLs that point at a non-public address. |
| PDF URL, or the HEAD probe of a URL without an image or PDF extension | Railtracks, in your process, only with trust_urls=True. There is no address check, so internal and localhost URLs work. |
A typical path from development to production:
- While building, pass local file paths or bytes.
- For URLs you trust, such as your own storage, an internal server or test fixtures, set
trust_urls=True. - When URLs come from end users, download them in your app, apply your own URL rules and content moderation, then pass the bytes. This is also how to attach an image from an internal host that litellm would refuse:
import httpx
def download_attachment(url: str) -> str:
"""Download an end-user URL under your app's rules and return a data URI."""
# Check the URL against your own allow/deny rules before fetching it.
response = httpx.get(url, timeout=10)
response.raise_for_status()
# Moderate response.content here, before it reaches the model.
mime_type = response.headers["Content-Type"].split(";")[0]
payload = base64.b64encode(response.content).decode()
return f"data:{mime_type};base64,{payload}"
user_url = "https://cdn.britannica.com/39/226539-050-D21D7721/Portrait-of-a-cat-with-whiskers-visible.jpg"
message_history = rt.llm.MessageHistory(
[
rt.llm.UserMessage(
content="What is the image below showing?",
attachment=download_attachment(user_url),
)
]
)
litellm's check can be changed with litellm.user_url_allowed_hosts (hosts it should allow) and litellm.user_url_validation (on by default). Both apply to the whole process.
When the PDF download fails, the error shows the URL without its credentials or query string, so signed URLs don't end up in logs. Errors raised by litellm or the provider can still include the full URL.