Conductr Agent Observability

Retrieving Agent Runs

If you have used Conductr to store your agent runs, you can directly utilize them and perform evaluations by following the simple steps in the snippet below:

import railtownai
from railtracks import evaluations as evals

# Select your agent run ids:
AGENT_RUN_IDS: list[str] = [
"ID_1",
"ID_2",
]

# Retrieve the agent runs
payload = railtownai.get_agent_runs(AGENT_RUN_IDS, skip_errors=True)

# extract `AgentDataPoint`s
data = evals.extract_agent_data_points(payload)

# Continue with evaluations as before

Sending Evaluations

Similar to sending your agent runs to Conductr, you can upload your agent evaluations by passing the upload_agent_evaluation to the evaluation function.

from railtownai import upload_agent_evaluation

results = evals.evaluate(
    data=data,
    evaluators=evaluators,
    payload_callback=upload_agent_evaluation, # Your evals will be sent to Conductr automatically
)

Required API Keys

You will need the following keys set up on your project for the above

Retreiving Agent Runs

CONDUCTR_PROJECT_PAT="..."
CONDUCTR_PROJECT_ID="..."
RAILTOWN_API_KEY="..."

Sending Agent Evaluations

RAILTOWN_API_KEY="..."
EVALUATIONS_API_TOKEN=".."

Please refer to your Conductr project page at https://cndr.railtown.ai/ for more info.