A Data Science portfolio is a collection of projects that we use to show our ability to work with data. We can share the work through a GitHub repository or create a website where someone can explore the projects we have developed.

Building the portfolio has become easier with AI as we can use it to help with the code and develop a project from our ideas. This is useful when we are learning a new method because we can spend more time exploring how it works without getting stuck on every part of the implementation.

However, the convenience also brings a question. If AI can help us build the project, what should we show in our Data Science portfolio?

For me, a good portfolio should explain how we approach the problem so someone can understand why we arrived at a particular result. Having a model with high accuracy would not provide much information if we cannot explain what the model predicts or whether the evaluation makes sense.

We don’t need to avoid AI to achieve this. We need to understand the work we present and be able to explain the decisions that affect the result.

In this article, we will explore five public examples to understand what we can include in our own portfolio. Some examples are portfolio websites, while others are individual projects that show how we can explain the work in more detail. We will use them as references without assuming how their authors used AI.

Let’s get into it.

Subscribe now

1. Ken Jee’s Data Science Salary Estimator

Let’s start with Ken Jee’s GitHub portfolio, which introduces his projects and links to the repositories where we can explore the implementation. One of the projects is a Data Science salary estimator that uses information from job postings to estimate a salary.

The repository explains how information from job descriptions is prepared for modelling. For example, the project creates features indicating whether certain skills are mentioned in the description. This gives us something to examine because we can follow how the original text becomes information that the model can use.

The project compares the models using Mean Absolute Error or MAE, which measures the average size of the prediction error in the same units as the target. Let’s look at two results reported in the repository.

The repository reports an MAE of roughly $18,860 for Linear Regression and $11,220 for Random Forest. This is approximately a 40% reduction in average error relative to the baseline. Showing this comparison gives us a way to discuss whether the more complex model provides an improvement.

We would still need to examine how the evaluation was performed before accepting the result. A lower average error also does not mean that every prediction is within $11,220 of the recorded value.

What should we pay attention to if we develop a similar project?

We need to understand what the salary data represents. A salary estimate from a job posting is not necessarily the salary someone eventually receives. Our explanation should make this clear so the reader understands what we are predicting and how much they can conclude from it.

We could also examine whether the error is similar across job locations. If one location has much less data, the overall MAE might not describe how well the model works there. Comparing the errors alongside the number of examples would help us decide where more data is needed.

AI could help us write the feature extraction code for this project. We would still need to check some job descriptions manually to see whether the extracted information has the meaning we intended. Explaining this check would provide more information about our work than only mentioning that we used AI.

2. Julia Silge’s Animal Crossing Review Analysis

The next example is Julia Silge’s analysis of Animal Crossing reviews. This is a project write-up where we can follow the process of using review text to predict whether the review is good or bad.

The article starts by examining the data before developing the model. Let’s look at the distribution of review lengths.

There is a sharp dip around 100 words before the number of reviews rises again. Julia identifies the unusual shape as a possible scraping problem. The chart does not prove what caused it, but it gives us a reason to return to the collection process and check whether some reviews were cut short.

This is something we can include in our own portfolio. We can show the pattern that made us question the data and explain how we checked it before moving on to the model.

The article then compares model settings using several evaluation metrics.

The horizontal axis shows the regularization penalty on a logarithmic scale. Increasing the penalty puts more pressure on the model to shrink its coefficients. We can see that the three metrics respond differently as the penalty becomes larger.

PPV measures how often positive predictions are correct, while NPV does the same for negative predictions. ROC AUC measures how well the model ranks the two classes across thresholds. Near the far right of the chart, NPV increases while PPV and ROC AUC fall sharply. Choosing the largest penalty because one metric improves would therefore miss what happens to the other results. Julia selects the final setting using ROC AUC.

What can we do with an explanation like this?

We can use the same approach when explaining our own classification project. For example, let’s say we want to predict which customers will cancel their subscriptions. We have 1,000 customers in the evaluation data, and 100 of them cancel.

If we predict that every customer stays, the accuracy would be 90% even though we have not identified any customer who cancels. This is why reporting accuracy alone would not explain whether the model is useful for finding cancellations.

Now, suppose the team can only contact 50 customers. We select the 50 customers with the highest predicted risk and find that 20 of them cancel. The precision for this selection would be 40% because 20 out of the 50 selected customers cancel. The recall would be 20% because we found 20 of the 100 cancellations in the evaluation data.

These are example numbers, but they show how we can connect the evaluation to the intended use. We should select the rule using validation data before reporting its performance on untouched test data. You can refer to the Scikit-Learn documentation for more information about the metrics.

We would still need further testing to know whether contacting these customers prevents cancellation. The prediction result only tells us how well we identify the customers who cancel, so our portfolio should explain this limitation as well.

3. Will Koehrsen’s Building Energy Project

Will Koehrsen’s Machine Learning project walkthrough uses building data to predict an Energy Star Score. The work is divided into notebooks that explain the development process and includes a report in the repository.

One part I find useful is the model interpretation notebook, where the author examines a prediction with a large error. Let’s look at the result below.

In this example, the model using a reduced set of features predicts approximately 12.86 while the recorded score is 100. The difference is large enough that we would want to investigate what happened instead of only reporting the average error.

The same model has a test MAE of approximately 10.86 points, but the error for this building is 87.14 points. This is why we should examine individual predictions as well as the average.

The notebook then uses LIME to investigate the prediction. LIME approximates the model behaviour around an individual example so we can examine how the inputs contribute to that local explanation.

What can we do with an explanation like this?

We can return to the record and check whether the input was prepared correctly. We could then look for other buildings with similar characteristics to see whether the same error happens repeatedly. This would help us decide whether the problem needs a correction in the data or another modelling experiment.

However, we should remember that explaining a prediction does not prove what causes the real outcome. A feature contributing to a lower predicted score does not automatically mean that changing the feature would change the building’s actual score.

For our own portfolio, we could apply this approach to the largest forecasting errors. If the model often underestimates sales during promotions, we can investigate whether information about promotions was available when making the forecast. We could then test whether including that information improves the validation result.

This gives us a reason for the next experiment based on the errors we observed. If AI helps write the explanation, we should check that it agrees with the actual records before including it in the portfolio.

4. Daniel Bourke’s Amenity Detection Project

Daniel Bourke’s amenity detection project explores using Computer Vision to identify objects in property photos. The project was inspired by Airbnb’s published work and follows the development through to an application.

The final evaluation reports a mean Average Precision, or mAP, of approximately 43.3 across 30 object classes. However, the overall score does not tell us whether the model performs similarly for each type of object. Let’s look at some of the class results.

As we can see from the chart, the reported AP is about 74.7 for refrigerators but only 5.8 for dishwashers. These are results from the same evaluation, so the overall mAP would hide a large difference if we reported it alone. AP summarizes detection performance across precision and recall; it is not the percentage of photos where every object is detected correctly.

What would this mean for an application?

If we wanted to use the model to suggest amenities for a property listing, we should investigate the weak categories before relying on the suggestions. We could inspect missed dishwashers and compare them with successful detections to see whether the problem relates to image conditions or the available training examples. We would then test the proposed improvement instead of assuming that adding more data would solve it.

The write-up also reviews the original project goals and acknowledges that the model did not reach the 50 mAP target. It explains that the result cannot be directly compared with Airbnb’s score because the datasets differ. This gives us a useful example of reporting what remains unfinished alongside the application.

We can learn from this when developing an application for our portfolio. A working demonstration shows that we can build the application, but we should also explain the conditions under which we tested it.

For example, we might develop a document classification application that works well with clean PDFs. If someone uploads a scanned page, the application could behave differently because the text needs to be extracted from an image first. Testing both types of documents would give us a better understanding of what the application supports.

We could extend this by checking what happens when the uploaded file is empty or has an unsupported format. The result should help us decide whether the application needs to reject the file or ask someone to review it.

AI can help us develop the interface, but we need these checks to understand whether the workflow behaves as expected. We can explain the remaining limitations in the project documentation and provide instructions for running the application locally if the live version is unavailable.

5. Anthony Lipphardt’s Data Science Portfolio

Our last example is Anthony Lipphardt’s portfolio website. The page introduces his work through project descriptions and provides links for readers who want to explore further.

The collection includes work in exploratory analysis and data visualization as well as Machine Learning. This shows how we can use a portfolio to present different types of work without making every project a prediction model.

How should we select the projects for our own portfolio?

I would start by considering what we want someone to understand about our ability. If several projects repeat the same modelling process on a clean dataset, they might not show how we deal with other parts of a Data Science problem.

For example, we could develop a retail analysis that examines repeat purchases using SQL. The explanation would include how we handled returned orders so the reader can understand how the repeat purchase measure was calculated.

We could then include a demand forecasting project that compares the model with a simple seasonal forecast. This would give us another type of work to discuss because we need to explain whether the model improves on an approach that is already easy to use.

These are suggestions for our own projects rather than examples taken from Anthony’s website. The idea is to select work that adds something different to the collection while keeping the detailed explanation accessible through the project links.

We can use a short introduction on the portfolio page to explain the question each project addresses. Someone interested in the result can then open the notebook or report to understand the work further. This also means we can improve the project explanation without making the portfolio’s main page too long.

Developing the Portfolio with AI

The examples above give us different ways to explain our work. We can apply the same approach when AI is part of the development process or when the project itself uses an AI model.

Let’s say we build an application that answers questions from documents. Showing a successful answer would be a good demonstration, but we would need more examples to understand whether the application works consistently.

We could prepare questions with the passages that should support the answers. We can then check whether the application finds the correct passage before examining whether the answer agrees with it. If the application finds the wrong passage, changing the answer prompt alone might not fix the problem.

Additionally, we should include questions that the documents cannot answer. This would let us examine whether the application acknowledges the missing information or provides an answer without support. We can use the same set of questions when testing a change so the comparison is easier to follow.

For AI-assisted coding, the explanation should focus on how we checked the output. For example, we might use AI to help merge customer data and then discover that the merge creates several rows for the same customer. Comparing the customer counts before and after the merge would help us identify the problem and correct the analysis.

GitHub’s Copilot guidance also recommends reviewing and testing generated code. In a Machine Learning project, we should check that preprocessing does not learn from the test data. The Scikit-Learn data leakage guide explains how this can lead to an overly optimistic evaluation.

If I were reviewing the portfolio, I would use these explanations to discuss why the author chose an approach and what they would investigate next. That is how I would recommend using a portfolio in a discussion, although each company can have a different evaluation process.

We don’t need to replace an existing project every time a new tool becomes available. We can start by revisiting a result that we have not explained well and investigate why it happened. The work we add through that investigation can help someone understand our ability even when AI assists with the implementation.

Continue Learning with Non-Brand Data

If you are interested in learning more about Data Science, Machine Learning, and AI, you can subscribe to Non-Brand Data. I regularly share articles, experiments, and examples that we can learn from.

For readers who want to explore a subject in more detail, the paid subscription also includes in-depth tutorials, written courses, notebooks, code, and additional learning materials.

I also offer 1-on-1 mentorship through MentorCruise if you want to discuss your learning direction, Data Science projects, technical challenges, or career development.

When exploring a dataset, we often repeat the same analysis for different groups. We might start with the average trip distance, then see how the result changes for cash payments or a specific week.

We can change the query and rerun our notebook. However, we also need to update the charts and make sure they use the same selection. This becomes more work as we add more questions to the analysis.

For this tutorial, we will build a notebook where we can change the filters and see the calculations and charts update together. We will use NYC Green Taxi trips as our example, combining marimo for the interactive notebook and DuckDB for the SQL analysis.

The result looks like the image below.

We will create payment, distance, and date controls, then use them to explore trip volumes and fares. Everything runs locally using open-source Python packages.

Let’s get into it.

Subscribe now

Building a Notebook With marimo and DuckDB

marimo is a reactive notebook for Python and SQL. When we change a value, it can automatically rerun dependent cells. This is useful when we want a dropdown or slider to control our analysis.

DuckDB is an embedded analytical SQL engine. We can query local files directly from Python without setting up a database server. In this project, it filters the trips and calculates the results before we pass them to our charts.

First, let’s download or clone the GitHub repository.

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe download_data.py
.\.venv\Scripts\python.exe -m marimo edit taxi_analysis.py

This opens the complete notebook in the marimo editor. The repository includes the dependencies and data preparation script, so we can start exploring without assembling those files ourselves.

The following examples walk through how our data, controls, and charts connect. In marimo, we put each notebook example in its own cell. If outputs are initially empty, click Run all to execute the notebook.

Preparing the NYC Taxi Data

For this example, we will use Green Taxi trips for January 2025 from the NYC Taxi & Limousine Commission. The file contains 48,326 rows and is around 1.2 MB, so we can work with the full month.

Before building the interactive part, let’s inspect the data with DuckDB. This loading example can be run separately in Python.

import duckdb
from pathlib import Path

data_path = Path("data/green_tripdata_2025-01.parquet").resolve()
con = duckdb.connect()
con.read_parquet(str(data_path)).create_view("raw_trips")

con.execute("DESCRIBE raw_trips").pl()

The code creates a view called raw_trips, which lets us query the Parquet file by name. DESCRIBE shows the available columns and their data types.

We will use the pickup and drop-off timestamps, trip distance, fare amount, and payment type. According to the data dictionary, distance is measured in miles, while fare_amount represents the meter fare. It does not include all the additional charges in the passenger’s total payment.

The data also suggests we should clean it first. The largest recorded distance is 84,731.57 miles, some fares are negative, and some pickup dates fall outside January. 1,836 records also lack a payment type.

For our explorer, I keep January pickups with a drop-off after pickup, a positive distance up to 100 miles, and a positive meter fare up to $500. Missing payment remains a separate category. These choices leave 45,467 trips.

These are tutorial limits, rather than official TLC validity rules. Long trips or negative fare adjustments can be legitimate, so the results reflect the records our rules include. The complete cleaning code and exclusion counts are available in the repository.

Combining SQL and Python

Now, let’s move into marimo. The project keeps the data preparation and query functions in analysis.py, so we can reuse them from the notebook.

First, we import the tools and functions we need.

import marimo as mo
import altair as alt
from analysis import (
 MONTH_START, MONTH_END, PAYMENT_LABELS,
 Filters, create_connection, query_dashboard,
)

Then, we connect to the prepared data.

con = create_connection()
clean_trips = con.table("clean_trips")

create_connection() creates the DuckDB views, including clean_trips for the records that passed our cleaning rules. The second line gives marimo a reference to that view without loading all its rows into a dataframe.

Let’s start with a SQL query that calculates the trip count, average distance, and average fare.

monthly_summary = mo.sql(
"""
 SELECT COUNT(*) AS total_trips,
 AVG(trip_distance) AS avg_distance,
 AVG(fare_amount) AS avg_fare
 FROM clean_trips
 """,
 engine=con,
)

We use COUNT(*) to count trips and AVG to calculate the two averages. The result is 45,467 trips, an average distance of 2.70 miles, and an average meter fare of $16.56.

marimo’s SQL cells are saved in the Python file using mo.sql. In the editor, we see the SQL query and its output. The result is also available as a Polars dataframe, so we can use it in the next Python cell.

baseline_rows = monthly_summary["total_trips"][0]
mo.md(f"The one-row SQL result summarizes **{baseline_rows:,} retained trips**.")

As we can see above, the Python cell reads monthly_summary from the SQL cell. We can use the same approach to pass a query result into a chart or another calculation.

The connection between cells comes from their variable names. If a cell uses monthly_summary, marimo knows it depends on the cell that creates that result. We do not need to place them in a particular execution order on the page, but each shared variable should be defined only once.

Making the Analysis Interactive

With our first analysis ready, let’s add the controls. We want to choose how the passenger paid, how far the trip traveled, and when the pickup happened.

The following code creates the three controls and places them side by side. It first reads the available payment codes so the dropdown matches our dataset.

_observed = con.execute("""
    SELECT DISTINCT COALESCE(CAST(payment_type AS INTEGER), -1) AS code
    FROM raw_trips ORDER BY code
""").fetchall()

payment_options = {"All methods": None}
payment_options.update({PAYMENT_LABELS[_code]: _code for (_code,) in _observed})

payment = mo.ui.dropdown(
    options=payment_options,
    value="All methods",
    label="Payment method",
    full_width=True,
)

distance = mo.ui.range_slider(
    start=0, stop=100, step=0.5, value=[0, 100],
    label="Trip distance (miles)", show_value=True,
    debounce=True, full_width=True,
)

pickup_dates = mo.ui.date_range(
    start=MONTH_START, stop=MONTH_END,
    value=(MONTH_START, MONTH_END),
    label="Pickup dates (inclusive)", full_width=True,
)

mo.hstack([payment, distance, pickup_dates], widths="equal", gap=2)

The dropdown starts with all payment methods, while the distance and date ranges cover the full cleaned month. We use debounce=True for the slider so the query updates when we release the handle, rather than for every movement while dragging it.

Next, let’s connect these selections to the analysis. In another cell, we read the controls’ .value properties and pass them to our query functions.

selected_filters = Filters(
 start=pickup_dates.value[0], end=pickup_dates.value[1],
 min_distance=distance.value[0], max_distance=distance.value[1],
 payment=payment.value,
)
summary, hourly, daily, payments = query_dashboard(con, selected_filters)

The query helper uses the same selection for every result. Its summary query follows this structure:

WITH filtered AS (
 SELECT * FROM clean_trips
 WHERE pickup_at >= $start_at AND pickup_at < $end_exclusive
 AND trip_distance BETWEEN $min_distance AND $max_distance
 AND (CAST($payment AS INTEGER) IS NULL OR payment_code = $payment)
)
SELECT COUNT(*) AS total_trips,
 AVG(trip_distance) AS avg_distance,
 AVG(fare_amount) AS avg_fare
FROM filtered;

The values beginning with $ are named query parameters. The helper passes the selected values separately from the SQL text. We also include the entire final day in the date selection by using the following midnight as the exclusive upper bound.

Let’s take Cash as an example. Changing the dropdown updates payment.value. marimo then reruns the analysis cell, followed by the cells that display its results. We get 11,035 trips, an average distance of 2.77 miles, and an average fare of $16.82 for the full month.

We can also combine the controls to look at a more specific group.

Selecting Cash, January 1–7, and 0–5 miles gives us 2,092 trips, an average distance of 1.75 miles, and an average fare of $12.91.

This is the reactive part of our notebook. We change a selection, and the cells that use it rerun. The earlier full-month SQL query stays as a baseline because it does not use these controls.'

Share

Visualizing the Results

Now, let’s display the calculations and explore how trip volumes change throughout the day.

For the summary, we can use mo.stat to show the three metrics together.

_distance_text = "N/A" if summary["avg_distance"] is None else f'{summary["avg_distance"]:.2f} mi'
_fare_text = "N/A" if summary["avg_fare"] is None else f'${summary["avg_fare"]:.2f}'

mo.hstack([
    mo.stat(value=f'{summary["total_trips"]:,}', label="Selected trips"),
    mo.stat(value=_distance_text, label="Average trip distance"),
    mo.stat(value=_fare_text, label="Average meter fare"),
], widths="equal", gap=2)

Each metric uses the summary result from the analysis cell. This means the displayed values follow the filters we just created.

For the hourly chart, DuckDB groups the selected trips by pickup hour and counts the records in each group. We then pass that small result to Altair.

hourly_chart = alt.Chart(hourly).mark_bar(color="#2563eb", size=12).encode(
    x=alt.X("pickup_hour:O", title="Pickup hour", sort=list(range(24))),
    y=alt.Y("trips:Q", title="Trips"),
    tooltip=["pickup_hour:O", alt.Tooltip("trips:Q", format=",")],
).properties(width="container", height=220)

mo.ui.altair_chart(hourly_chart, chart_selection=False)

In the code above, mark_bar creates the bars, while the x and y definitions specify the pickup hour and trip count. The tooltip lets us inspect each value. mo.ui.altair_chart displays the result in the notebook.

The complete notebook also includes a daily line chart and a payment chart. They use the same filtered data, so we can review the daily pattern and payment mix alongside the hourly distribution.

From the full-month result, we can see that 17:00 has the highest hourly count, with 3,937 trips. January 30 has the highest daily count, with 1,811 trips.

Let’s see what happens when we select Cash.

The hourly peak shifts to 16:00 with 947 trips, and the other charts update to match the selection. We can explore how the pattern changes without editing the query or manually rerunning each chart.

The results describe the selected records. They do not tell us why passengers traveled at those times or chose a particular payment method.

A selection might also return no matching trips. The notebook displays zero trips, N/A for the averages, and a message asking us to adjust the filters.

This keeps the notebook useful even when a combination of filters returns no data.

Running Our Notebook as an Application

With the notebook ready, we can run it as an application. Stop the editor with Ctrl+C, then use the following command.

.\.venv\Scripts\python.exe -m marimo run taxi_analysis.py

In application mode, readers can use the filters and see the results without the source-editing controls. The opening image shows this view. Keep the local Python process running while using the page.

This approach works well when we want to explore different groups using the same calculations. As the dataset grows, we should check the query cost because changing a filter reruns the analysis. DuckDB aggregates before transferring results to Python, but the view itself does not cache our answers.

I checked the calculations against a separate Polars implementation and tested the filters in the running application. The GitHub repository includes the validation details, source code, and dataset instructions.

Conclusion

In this article, we built an interactive NYC Taxi Trip Explorer using marimo and DuckDB. We used SQL to analyze the trips, Python to connect the results, and notebook controls to explore different selections.

We can build on the same workflow when our analysis involves repeated queries for different groups. For the complete implementation, you can access the notebook and supporting code on GitHub.

I hope it helps!

Continue Learning with Non-Brand Data

If you are interested in learning more about Data Science, Machine Learning, and AI, you can subscribe to Non-Brand Data. I regularly share articles, experiments, and examples to learn from.

For readers who want to explore a subject in more detail, the paid subscription also includes in-depth tutorials, written courses, notebooks, code, and additional learning materials.

I also offer 1-on-1 mentorship through MentorCruise if you want to discuss your learning direction, Data Science projects, technical challenges, or career development.

Disclosure: This article is sponsored by TickerLayer, which provided reviewer access. The examples and observations are based on my testing.

Market data includes prices, trading volume, and historical observations. We can use it to study market changes or build financial applications, but we also need to understand what each value represents and when it was recorded.

To access this data in Python, we can use an API. One platform that provides this access is TickerLayer, which offers REST APIs for individual requests, WebSocket streams for ongoing updates, and an MCP server for compatible AI clients.

In this tutorial, we will build a small market monitor with Python, then explore streaming and the MCP option. All the code, notebook, and saved responses are in the GitHub repository.

Let’s get into it.

Subscribe now

Set Up the Python Environment

First, get an API key from the TickerLayer dashboard by following the quickstart instructions. You will need access to the feeds used below, including an eligible paid plan or approved trial for streaming.

Download the repository to continue the tutorial.

Next, install the packages:

python -m pip install -r requirements-article.txt

The files market_monitor.py and tickerlayer_experiment.py contain the functions used below. Keep them together in the repository folder.

Store your key in an environment variable named TICKERLAYER_API_KEY. For example, in PowerShell:

$env:TICKERLAYER_API_KEY = "YOUR_API_KEY"

Replace the placeholder with your key, then create a session using it. Keep the key out of any notebook you share.

import os
import requests
import pandas as pd

from market_monitor import get_json, quote_row, WATCHLIST

key = os.environ["TICKERLAYER_API_KEY"]
session = requests.Session()
session.headers.update({
    "x-api-key": key,
    "Accept": "application/json",
})

We can now use this session for our requests. The get_json function imported from market_monitor.py returns each response as a Python dictionary.

Build the REST Market Monitor

Let’s start with Bitcoin. We can retrieve its latest bid and ask prices using the BTCUSD symbol:

quote = get_json(session, "/crypto/quote/BTCUSD")
print(quote)

The response gives us the prices and their timestamp. To compare Bitcoin with other instruments, let’s create a function that places each quote into the same table structure:

def quote_row(asset, payload):
    bid, ask = float(payload["bid"]), float(payload["ask"])
    return {
        "asset": asset,
        "symbol": payload["symbol"],
        "bid": bid,
        "ask": ask,
        "mid": (bid + ask) / 2,
        "spread": ask - bid,
        "event_time_utc": pd.to_datetime(
            payload["timestamp"], unit="ms", utc=True
        ),
    }

This function also calculates the midpoint and spread, and converts the timestamp to UTC. We can now apply it to WATCHLIST, which we imported from market_monitor.py:

rows = []
for asset, symbol in WATCHLIST:
    payload = get_json(session, f"/{asset}/quote/{symbol}")
    rows.append(quote_row(asset, payload))

monitor = pd.DataFrame(rows)
print(monitor.to_string(index=False))

The watchlist includes crypto, forex, stocks, indices, ETFs, commodities, and perpetuals. I checked the symbol directories before choosing these examples; stocks, for example, use a country prefix such as US:AAPL.

Displaying the monitor in a notebook gives us the following table:

One thing I noticed was the AR40 timestamp. It was about 66.9 hours older than when the response arrived. Let’s check the market status for more context:

status = get_json(
    session, "/markets/status",
    params={"asset": "indices", "symbol": "AR40"},
)
print(status["status"], status["phase"], status["reason"])

The saved status response was:

open primary Generic weekday session (UTC)

This response used a generic UTC calendar, not a confirmation that a new AR40 quote had arrived. TickerLayer’s market-status documentation explains that an open session does not guarantee live quotes. The saved responses show the difference, but do not establish why this quote was older.

We can also request government bond yields. Since these are rates, we will keep them separate from the price table. Here is an example for the US ten-year yield:

bond = get_json(session, "/bond/snapshot/US:10Y")
print(bond["rate"], bond["unit"], bond["date"])
print("Change in basis points:", bond["change_bps"])

Keeping the date beside the yield makes its update period clear.

Add Historical Data

To explore how prices change over time, we can retrieve historical data. Let’s request the last seven completed days of Bitcoin OHLCV data, which contains the open, high, low, close, and volume:

from datetime import datetime, timedelta, timezone

today = datetime.now(timezone.utc).date()
start = today - timedelta(days=7)
end = today - timedelta(days=1)

history = get_json(
    session,
    f"/crypto/agg/BTCUSD/1/day/{start}/{end}",
    params={"sort": "asc", "limit": 10},
)
bars = pd.DataFrame(history["results"])
bars["date"] = pd.to_datetime(bars["t"], unit="ms", utc=True)
print(bars[["date", "o", "h", "l", "c", "v"]])
print("Bars returned:", len(bars))
print("Next page:", history.get("next_offset"))

The result table is shown in the table below:

For a longer date range, check next_offset for another page of results. The crypto REST reference covers the query parameters.

Receive Updates Through WebSocket

For ongoing price updates, we can use a WebSocket connection. We will subscribe to Bitcoin quotes and trades through the crypto.quotes and crypto.trades channels.

TickerLayer’s subscription guide shows this request format and acknowledgement.

For our Bitcoin example, the stream_events function in market_monitor.py opens the connection and sends the following subscription:

subscription = {
    "action": "subscribe",
    "channels": ["crypto.quotes", "crypto.trades"],
    "symbols": ["BTCUSD"],
    "snapshot": True,
}

The snapshot option requests a recent value on subscription when one is available. Our helper then collects up to 80 messages or listens for 15 seconds, whichever comes first.

Let’s use the received messages to update our monitor:

import asyncio
from market_monitor import stream_events, apply_event

stream = asyncio.run(stream_events(key, seconds=15))
state = {}
for event in stream["events"]:
    apply_event(state, event["message"])

print(state[("crypto", "BTCUSD")])

If you are running this in a notebook with an active event loop, use stream = await stream_events(key, seconds=15) instead of asyncio.run.

We can also inspect the first quote message directly:

import json

quote_event = next(
    event["message"] for event in stream["events"]
    if event["message"].get("type") == "quote"
)
print(json.dumps(quote_event, indent=2))

To see the mix of quotes, trades, and system messages in this capture, we can count them by type:

from collections import Counter

message_counts = Counter(
    event["message"].get("type", "unknown")
    for event in stream["events"]
)
print("Messages received:", len(stream["events"]))
print("Message types:", dict(message_counts))
print("Elapsed capture time:", stream["duration_seconds"], "seconds")

After checking how many messages we received and their types, we can use them to update our monitor. The apply_event function handles quotes and trades separately, keeping the latest bid and ask alongside the latest trade price.

Handle Temporary Errors and Reconnect

A request might fail temporarily, even when our code is correct. For a 503 response, the get_json function retries up to three times, waiting 1, 2, and 4 seconds between attempts. It uses the server’s Retry-After value when provided.

If the request still fails, we can handle the HTTP error using the following code:

try:
    latest = get_json(session, "/crypto/quote/BTCUSD")
    print(latest)
except requests.HTTPError as exc:
    print("Request failed with HTTP status:", exc.response.status_code)

Other HTTP errors are raised without retrying. For example, a 401 means we need to check our API key, while a 429 means we should check our request limits.

For WebSocket, we need to reconnect if the connection drops. Let’s use stream_with_reconnect from the repository and pass the collected messages into our monitor:

from stream_recovery import stream_with_reconnect

recovered = asyncio.run(stream_with_reconnect(key, seconds=15))
print("Reconnect attempts:", recovered["reconnect_attempts"])

for event in recovered["events"]:
    apply_event(state, event["message"])

The function keeps the messages received before the interruption and subscribes again after reconnecting. It allows up to three retries and requests a recent snapshot when available. The capture still has time and message limits.

In a notebook, use recovered = await stream_with_reconnect(key, seconds=15) instead of asyncio.run.

The snapshot helps update our monitor, but it does not restore every trade missed during the interruption. This is useful for our latest-value monitor, not a complete trade history.

Request a Snapshot Through MCP

If we want to access market data through a compatible AI client, TickerLayer also supports the Model Context Protocol, or MCP. It exposes data requests as tools the client can call.

Let’s try the get_snapshot tool from Python. The run_mcp function in tickerlayer_experiment.py connects to the MCP server, lists its tools, and calls get_snapshot with these arguments:

{
  "asset_class": "crypto",
  "symbol": "BTCUSD"
}

Run the function and print the snapshot:

from tickerlayer_experiment import run_mcp

mcp = run_mcp(key)
if mcp["error"]:
    raise RuntimeError(mcp["error"])

print("Tools discovered:", len(mcp["tool_names"]))

call = next(
    step for step in mcp["steps"]
    if step["name"] == "tools_call_get_snapshot"
)
result = call["body"]["result"]
if result.get("isError"):
    raise RuntimeError(result["content"])
print(json.dumps(result["structuredContent"], indent=2))

The server listed 12 tools, and the snapshot returned the bid, ask, last trade, previous close, and price change. This gives an AI client structured market data to work with.

What I Would Keep in Mind

TickerLayer returns quotes in a similar format across different markets, making them easy to combine in one table. However, some prices were older than the others. This is why we should check when each price was last updated before using it in our analysis.

I used a reviewer key with elevated access. Before running your own monitor, check your account’s feed access and request limits, especially if you plan to poll frequently.

TickerLayer describes its data as derived and indicative, and perpetual quotes are calculated composites. My short tests checked whether I could retrieve and process the responses; they did not independently verify price accuracy or long-term reliability. Review the market data disclaimer for your intended use. You can inspect the saved responses in the repository’s fresh_results folder.

Run the Complete Example

From the downloaded repository folder, run:

python market_monitor.py --capture

The command saves new responses under fresh_results. Open TickerLayer_Outputs.ipynb and change capture_path to your new JSON file to explore them.

Conclusion

In this article, we explored how to work with TickerLayer in Python, from combining market quotes in a table to retrieving historical prices. We also covered WebSocket updates and MCP integration. These examples provide a starting point for building a market data workflow for our own projects.

If you would like to build a similar project, visit TickerLayer to explore the available market data and get an API key.

You can start with the code in our GitHub repository, then add the stocks, currencies, or other assets you want to follow. From there, try using the results in a research notebook or building a dashboard for your own watchlist.

Time-series forecasting helps us estimate future values so we can plan ahead. For example, we can use historical sales data to predict demand and decide how much inventory to prepare for the coming month. We look for patterns in observations collected over time and use them to make these estimates.

For this task, we can try TimesFM-3, a pretrained forecasting model from Google Research. It allows us to generate predictions without training a model from scratch.

In this article, we’ll learn how to use TimesFM-3 for time-series forecasting in Python. Let’s get into it.

Subscribe now

What Is TimesFM-3?

TimesFM-3 is a time-series foundation model developed by Google Research. We can provide historical observations and generate predictions with its pretrained weights. This approach is called zero-shot forecasting because we don’t train or fine-tune the model on our dataset.

Compared with earlier versions, TimesFM-3 adds native support for multivariate forecasting and covariates. Multivariate forecasting means predicting related time series together. Covariates are additional variables that can help the prediction, such as temperature or calendar information.

Some covariates are only available for past dates, while others are known in advance. For example, we have recorded temperatures for the past, but we can calculate weekend dates for the future. We’ll try both in our example.

Setting Up the Environment

First, download the code from the GitHub repository and install the required packages from the project folder:

python -m pip install -r requirements.txt

We’ll use the timesfm3 interface from the timesfm package. The repository includes the tested dependencies and setup instructions.

The downloadable model weights are licensed for non-commercial, non-production use, so check the license before using them for your project.

Preparing the Bike Sharing Dataset

For the dataset example, I use the UCI Bike Sharing dataset. It contains rental counts from Capital Bikeshare during 2011 and 2012, along with weather information. There are daily and hourly files, but we’ll use day.csv so each observation represents one day.

Let’s load the dataset and prepare the historical and test data with the following code:

import io
import urllib.request
import zipfile

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

url = "https://archive.ics.uci.edu/static/public/275/bike+sharing+dataset.zip"
with urllib.request.urlopen(url, timeout=60) as response:
    payload = response.read()

with zipfile.ZipFile(io.BytesIO(payload)) as archive:
    with archive.open("day.csv") as daily_file:
        df = pd.read_csv(daily_file, parse_dates=["dteday"])

df = df.sort_values("dteday").set_index("dteday")
assert df.index.is_unique
assert df.index.equals(
    pd.date_range(df.index.min(), df.index.max(), freq="D")
)
assert df[["cnt", "registered", "casual", "temp"]].notna().all().all()
assert (df["cnt"] == df["registered"] + df["casual"]).all()

horizon = 14
history = df.iloc[:-horizon].copy()
test = df.iloc[-horizon:].copy()

The code above sorts 731 daily observations by date and checks for missing or duplicate dates. We keep the final 14 days as test data and use the earlier observations as our history.

We’ll start with cnt, the total rental count, then try registered and casual rentals as related targets.

We split the data in date order so the model only receives observations from before the forecast period.

Share

Generating Our First Forecast

With the dataset ready, let’s generate our first prediction using TimesFM3Forecaster:

from timesfm3 import TimesFM3Forecaster

forecaster = TimesFM3Forecaster.from_pretrained(
    "google/timesfm-3.0-pytorch"
)

context = history["cnt"].to_numpy(dtype=np.float32)
univariate = forecaster.predict(
    context=context,
    horizon=horizon,
    return_quantiles=True,
)
print(univariate.forecast.shape)   # (14,)
print(univariate.quantiles.shape)  # (14, 9)

In the code above, context contains the historical rental counts, and horizon sets the number of future steps. Since our observations are daily, horizon=14 gives us a two-week forecast.

The forecast attribute contains the predicted median for each day. With return_quantiles=True, we also get nine quantiles, from the 10th to the 90th percentile, to describe the model’s uncertainty.

Forecasting Multiple Variables

Next, let’s try the multivariate prediction with the same dataset. This time, we’ll predict registered and casual rentals together. We’ll also pass historical temperature and a weekend indicator using the following code:

future_dates = pd.date_range(
    history.index[-1] + pd.Timedelta(days=1),
    periods=horizon,
    freq="D",
)

targets = history[["registered", "casual"]].to_numpy(dtype=np.float32).T
past_temperature = history[["temp"]].to_numpy(dtype=np.float32).T

all_dates = history.index.append(future_dates)
weekend = (all_dates.dayofweek >= 5).astype(np.float32)[None, :]

multivariate = forecaster.predict(
    context=targets,
    horizon=horizon,
    past_only_covariates=past_temperature,
    past_future_covariates=weekend,
    return_quantiles=True,
)
print(multivariate.forecast.shape)   # (2, 14)
print(multivariate.quantiles.shape)  # (2, 14, 9)

Each row in targets represents a rental category, and each column represents a day. We use .T to arrange the data in this format.

The historical inputs must refer to the same dates. The weekend array also covers the forecast period because we know those dates in advance.

We now have two sets of predictions, with registered rentals in the first row and casual rentals in the second.

We only provide historical temperature. Using the actual temperatures from the test period would introduce data leakage, as those measurements weren’t available when making the forecast. The weekend indicator is available in advance, but it doesn’t account for public holidays.

Visualizing and Evaluating the Results

To see the prediction result, let’s create a plot with the historical data and the actual values from our test period:

q = np.asarray(forecaster.config.quantiles)
lo = int(np.flatnonzero(np.isclose(q, 0.1))[0])
hi = int(np.flatnonzero(np.isclose(q, 0.9))[0])

fig, ax = plt.subplots(figsize=(10, 4))
ax.plot(history.index[-28:], history["cnt"].iloc[-28:], label="History")
ax.plot(future_dates, test["cnt"], "o-", label="Actual")
ax.plot(future_dates, univariate.forecast, label="TimesFM-3 median")
ax.fill_between(
    future_dates,
    univariate.quantiles[:, lo],
    univariate.quantiles[:, hi],
    alpha=0.2,
    label="10th-90th percentile band",
)
ax.set(ylabel="Rentals per day", xlabel="Date")
ax.legend()
fig.autofmt_xdate()
plt.show()

We can also plot registered and casual rentals and add a weekly baseline for comparison:

The shaded area shows the 10th–90th percentile band, a nominal 80% interval. Its actual coverage still needs to be evaluated.

To measure prediction error, we’ll use mean absolute error, or MAE. It calculates the average absolute difference between the predicted and actual values. A lower MAE means a smaller error.

Let’s compare the model with a seasonal naive baseline, which repeats the last observed week’s rental counts:

weekly_baseline = np.resize(
    history["cnt"].to_numpy()[-7:], horizon
)
actual = test["cnt"].to_numpy()

model_mae = np.mean(np.abs(actual - univariate.forecast))
baseline_mae = np.mean(np.abs(actual - weekly_baseline))
print(f"TimesFM-3 MAE: {model_mae:.2f}")
print(f"Weekly baseline MAE: {baseline_mae:.2f}")

The baseline repeats the same observed week twice, without using any values from the test period.

The results for all three targets are as follows:

For this test period, TimesFM-3 has a lower MAE than the weekly baseline for all three targets. However, the total rental interval contains only three of the 14 actual values.

The total rental forecast still misses much of the decline toward the end of the test period. Its MAE is only 4.2% below the baseline’s error, so I wouldn’t call it reliable from this result alone.

Compare each target with its own baseline. Casual rentals have smaller counts, so their lower MAE doesn’t automatically mean a better forecast.

What Should We Know Before Using TimesFM-3?

Our example covers one forecast period. We should repeat the evaluation at several starting dates before deciding whether the model meets our needs. The joint example also changes the targets and adds covariates at the same time, so it doesn’t tell us how much each feature helps.

Zero-shot forecasting doesn’t guarantee that a dataset was absent from pretraining. The model card doesn’t confirm whether these bike rental records were included, so this example shouldn’t be treated as an unseen-data benchmark.

Keep the total rental forecast separate from the joint predictions. Adding the registered and casual medians or quantile bounds doesn’t generally produce the total’s median or a valid interval.

For a SQL alternative, TimesFM-3 is also available in BigQuery preview.

Conclusion

In this article, we learned how to use TimesFM-3 for time-series forecasting in Python. We started with daily bike rentals, then tried related targets and additional variables.

You can find the complete code in the GitHub repository.

I hope it helps!

Continue Learning with Non-Brand Data

If you are interested in learning more about Data Science, Machine Learning, and AI, you can subscribe to Non-Brand Data. I regularly share articles, experiments, and examples that we can learn from.

For readers who want to explore a subject in more detail, the paid subscription also includes in-depth tutorials, written courses, notebooks, code, and additional learning materials.

I also offer 1-on-1 mentorship through MentorCruise if you want to discuss your learning direction, Data Science projects, technical challenges, or career development.

Building a machine learning model for tabular data often starts with familiar methods such as Logistic Regression, Random Forest, or CatBoost. We train them on our data and compare their predictions to find a model that suits our problem.

TabPFN-3.5 gives us another option. It uses a pretrained model to make predictions from the examples we provide. I wanted to see how this approach compares with the models we already use, including how long we need to wait for the predictions.

In this article, we will compare TabPFN-3.5 with Logistic Regression, Random Forest, and CatBoost on two datasets. We will look at their prediction performance and runtime, then discuss the limitations of the comparison.

Let’s get into it.

Subscribe now

What makes TabPFN-3.5 different?

To understand this comparison, let’s first look at how TabPFN-3.5 uses our data. It takes labeled training examples as context and predicts the target for new rows. According to Prior Labs, the model was pretrained on synthetic tabular tasks, and the base checkpoint supports both classification and regression.

Prior Labs released TabPFN-3.5 on September 15, 2026. Its release report describes several variants, including Fast, Plus, and Thinking. For this comparison, I used the base model.

With that model, we still use the familiar fit() and predict_proba() methods in TabPFN. However, calling fit() does not train the foundation model from scratch. We provide context for a model that has already been pretrained, which differs from fitting Logistic Regression coefficients or building trees from our dataset.

This also affects how we compare runtime. We need to measure both fitting and prediction to see how long each model takes to produce results. Before we get to those measurements, let’s prepare the datasets and give every model the same train/test splits.

Preparing our comparison

We will use the Breast Cancer Wisconsin Diagnostic and Wine datasets included with scikit-learn. I chose them because we can load the data without a separate download. They also let us compare the models on a binary classification problem and a multiclass problem.

In Breast Cancer, we use measurements derived from digitized images of breast masses to predict malignant or benign labels. Wine gives us a different task: using chemical measurements to distinguish three cultivars. Both loaded datasets have numeric features and no missing values so that we can focus this comparison on complete numeric data.

Their small size makes the experiment easy to repeat, but it also limits what we can conclude about business datasets. We are using Breast Cancer as a classification exercise; these results don’t establish whether a model is suitable for clinical use.

With that scope in mind, let’s create our first train/test split:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

data = load_breast_cancer(as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
 data.data,
 data.target,
 test_size=0.25,
 stratify=data.target,
 random_state=42
)

We keep 75% of the rows for training and 25% for testing. The stratify argument keeps the class proportions close to those in the full dataset.

In the notebook, we repeat this with seeds 42 through 46 for both datasets. Every model gets the same rows within each split. Four models across two datasets and five splits give us 40 evaluations.

Setting up the models

With our splits prepared, we can set up the models. For Logistic Regression, we put StandardScaler inside a pipeline:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(
 StandardScaler(),
 LogisticRegression(C=1.0, max_iter=2000, random_state=42)
)
model.fit(X_train, y_train)
probabilities = model.predict_proba(X_test)

The scaler learns from the training rows and applies that transformation to the test rows. We shouldn’t scale the full dataset before splitting because that would let test data influence preprocessing.

The other models receive the original numeric values. Alongside this pipeline, I used Random Forest, CatBoost, and TabPFN-3.5 with the following settings. I kept these configurations fixed without a hyperparameter search:

The results therefore describe these configurations, and tuning could change them. The settings also represent different amounts of work: four TabPFN ensemble members shouldn’t be treated as equivalent to four trees.

To make sure we test the intended TabPFN model, we select its version explicitly:

from tabpfn import TabPFNClassifier
from tabpfn.constants import ModelVersion

model = TabPFNClassifier.create_default_for_version(
 ModelVersion.V3_5,
 n_estimators=4,
 device=”cpu”,
 random_state=42,
 n_preprocessing_jobs=1,
 show_progress_bar=False
)
model.fit(X_train, y_train)
probabilities = model.predict_proba(X_test)

The Python package version is tabpfn==9.1.0, while the model we’re testing is TabPFN-3.5. Choosing ModelVersion.V3_5 keeps the experiment from depending on the package’s default model version.

To make that choice reproducible, the notebook pins the TabPFN-3.5 checkpoint tabpfn-v3.5-20260909.safetensors and records its SHA256 hash. Before the first download, we need to accept the model license and enable authenticated access through Prior Labs. Keep the access key outside the notebook. Once the weights are available, inference runs locally. The setup output below confirms which checkpoint we used and that all four models ran.

Comparing prediction performance

Now that all four models are available, we can compare their predictions. Our main metric is balanced accuracy. We calculate the share of correct predictions within each class, then average those shares. This gives each class equal weight.

We also record accuracy and macro F1. F1 combines precision and recall; the macro average gives each class equal weight. Log loss evaluates the probabilities and penalizes confident mistakes, so a lower value is better.

The notebook includes ROC AUC as well. For Breast Cancer, class 1 is benign and is treated as the positive class for binary AUC. Wine uses macro one-vs-rest AUC. All class predictions come from the largest predicted probability.

For the main comparison below, we focus on accuracy, balanced accuracy, macro F1, and log loss. The tables show the mean and sample standard deviation across five splits. Because the test sets overlap, the standard deviations describe the variation we observed. They aren’t confidence intervals or proof that one model will perform better on another dataset.

Breast Cancer Wisconsin

For Breast Cancer, Logistic Regression has the highest average balanced accuracy. Its difference from TabPFN is about 0.11 percentage points. I wouldn’t make much of that gap on these splits. TabPFN beats Logistic Regression twice, loses twice, and ties once when we compare the matching splits.

Logistic Regression has the better average class prediction scores here, while TabPFN has a slightly lower log loss. That distinction matters if we plan to use the probabilities. Predicting the correct class and assigning a good probability aren’t the same thing.

Wine

Moving to Wine changes the ranking. Here, TabPFN performs better across the four reported metrics. Its balanced accuracy is about 0.74 percentage points above Random Forest, the next model on that metric. On matching splits, TabPFN beats Random Forest twice and ties it three times.

The probability results give us another reason to consider TabPFN here. Random Forest ranks second on balanced accuracy but last on log loss, while TabPFN has the lowest log loss. If we only checked class predictions, we would miss this difference.

I would consider this a good result for TabPFN under these settings. I’d still want to see whether the improvement holds on a larger test set.

Alongside these summaries, we save the probabilities for every test row. This lets us inspect the predictions behind a score difference on the same rows. Prediction quality is only one part of choosing a model, though, so we also need to look at how long those predictions take.

Share

How long do the models take?

To measure that time, I ran all four models on an Intel Core i7-12700H CPU with four compute threads. TabPFN uses one preprocessing worker. The notebook records the full environment, including Windows and Python 3.11.5.

Before timing the comparison, we run each model on a small subset. Then every measured run creates a fresh estimator, fits it, and predicts probabilities for the full test batch.

These timings start after package imports and the first weight download. Loading TabPFN’s cached weights stays inside the fit measurement, so the results include that work. The table below separates fit and prediction time to show where each model spends its time.

The part I would pay attention to is TabPFN’s prediction time. Most of its total runtime comes after fit(). Timing fit alone would leave out most of the work in this experiment.

Logistic Regression finishes fastest on both datasets, while TabPFN takes the longest total time. For a small offline analysis, waiting a few seconds might be acceptable. If we need to repeat the process many times, I would check whether the prediction improvement is worth that extra time.

Image by the author. Total runtime uses a logarithmic axis.

When reading this chart, remember that we timed the first full-test prediction from a fresh estimator on the CPU after a separate warmup. These measurements don’t cover GPU performance or repeated predictions from a model kept in memory. A batch time also doesn’t tell us the response time for one request in a production service. We can review the measured scores and timings together in the notebook output below.

What would I use?

Taking the prediction scores and runtime together, I would still start with Logistic Regression for these small numeric datasets. It gives us strong scores with very little runtime, so I would run it before deciding whether we need another model.

I would include TabPFN in the comparison too. The Wine result gives us a reason to test it, especially when we care about the predicted probabilities. It doesn’t give us a reason to assume it will perform better on every table.

To see why those probability scores matter, imagine we’re using predictions to decide which records to review first. I would compare the probability scores and inspect the mistakes alongside runtime. If our task needs frequent predictions, the extra computation could change which model I choose.

For the same reason, I would keep Random Forest and CatBoost in the comparison when testing our own data. We used fixed settings here, and neither dataset tells us how the models compare when categorical features or tuning matter. Those gaps bring us to the limits of this experiment.

Where this comparison is limited

The first limitation is the amount of test data. There are only 45 Wine test rows in each split. A few changed predictions can affect the ranking, and our five overlapping holdouts aren’t independent experiments. I’d want to test more data before treating the Wine result as a reason to replace another model.

We also haven’t tuned the models. To extend this comparison, we could search within the training data and record the search budget for each model. The test labels should stay out of those choices.

Beyond tuning, the data itself needs a broader comparison. Both datasets are complete numeric tables, so we haven’t tested missing values or categorical features. We also haven’t measured larger datasets or peak memory. Any of these could change which model works best for our task.

Extending the experiment also means choosing a split that matches our data. For records ordered by time or connected through customers, we’d need a time-based or group-based split. Random splitting could put closely related records in both training and testing.

Conclusion

In this article, we compared TabPFN-3.5 with Logistic Regression, Random Forest, and CatBoost on two datasets using the same five train/test splits. We measured prediction performance and CPU runtime, then discussed the comparison’s limitations.

TabPFN led on Wine’s balanced accuracy and had the lowest log loss on both datasets. Logistic Regression was slightly ahead on Breast Cancer’s balanced accuracy and was much faster. These results show why we need to consider both prediction quality and runtime.

I would include TabPFN alongside traditional models when testing our own data. The comparison notebook and Python code provide a starting point for finding which model suits our task.

Continue Learning with Non-Brand Data

If you are interested in learning more about Data Science, Machine Learning, and AI, you can subscribe to Non-Brand Data. I regularly share articles, experiments, and examples that we can learn from.

For readers who want to explore a subject in more detail, the paid subscription also includes in-depth tutorials, written courses, notebooks, code, and additional learning materials.

I also offer 1-on-1 mentorship through MentorCruise if you want to discuss your learning direction, Data Science projects, technical challenges, or career development.

Previous module: Module 13, When an SQL query needs optimization.

In this module, we will use a March order analysis to write a short report. We will build the final table, check the total, and then write the findings, limitations, and recommendations.

The report is also your final course project. It brings together the question, SQL, result, and checks from the earlier modules.

After this module, you can:

• Build a report table with prep, metrics, and final

• Write a finding supported by the result

• Explain what the result doesn’t show

• Recommend a specific next check

  Subscribe now

Read more

Previous module: Module 12, Checking whether your result is right.

A slow query doesn’t always need another index. Before changing anything, we need to understand how PostgreSQL reads the table and how many rows the filter keeps.

For example, a tracking-code lookup returns one row from 30,000 orders. An index is useful in this case because the filter is very selective. A status filter that keeps 24,000 rows is different because PostgreSQL still needs most of the table.

For me, query optimization starts after the result is correct. We should read the plan, change one thing, and confirm that the query still returns the same result.

We will compare a sequential scan with an index scan, then check why PostgreSQL can still choose a sequential scan when an index exists.

Subscribe now

Read more

Previous module: Module 11, Using window functions without changing the row count.

A SQL query could run without an error, but the result could still be wrong for the question we want to answer.

For example, a weekly report shows delivered revenue of 520.00, while a monthly report shows 700.00. Both queries are valid. The difference happens because the weekly query starts on 4 March and the monthly query starts on 1 March. Each query counts a different group of orders.

For me, checking the population is the first thing to do when two numbers don’t match. Don’t change the query until the numbers happen to agree. Instead, compare the filters, find the rows that only appear in one result, and check whether the join changes the row count.

We will use those steps to compare weekly and monthly revenue. After that, we will check duplicate payment IDs and unmatched records.

Subscribe now

Read more

Previous module: Module 10, Breaking a query into steps with subqueries and CTEs.

Read more

Previous module: Module 9, When a join increases your numbers.

A long query isn’t always a complicated query. Sometimes it contains several small tasks that have been placed inside one another. The SQL can still return the correct answer, but it becomes difficult to see which rows were filtered, when the grain changed, and where a wrong number first appeared.

Subqueries and common table expressions, usually called CTEs, help us separate those tasks. A subquery places one query inside another. A CTE gives a query result a name with WITH so that we can read and test the steps in order.

In this module, we’ll rewrite one nested query as prep, metrics, and final. We will also check the number of rows at each stage and confirm that the new query returns the same result as the old one.

Subscribe now

Read more

AI coding agents can already do many tasks in a data science project.

They can explore the dataset, prepare the features, train several models, and save the final pipeline. With a simple prompt, we can build a complete machine learning workflow quickly.

However, building a model is different from understanding the problem behind it.

I experienced this difference when I asked an AI agent to build a customer churn model. I ran the experiment twice with the same dataset, environment, and prompt. The only difference was the project context available to the agent.

Both agents created a working pipeline. Both avoided data leakage, used time-based validation, and selected logistic regression. However, they did not evaluate the same problem.

This is where context engineering could become useful.

In this article, I want to explore how we can give an AI agent the data-science context it needs. I also performed a small experiment to see what happened when the project included a prediction contract.

Let’s get into it.

Subscribe now

AI Agents in the Data Science Workflow

AI agents have become more capable of working independently inside a project. They are no longer limited to suggesting a single Python function. The agent could inspect files, write code, run the training process, fix errors, and save the model artifacts.

Recent research also evaluates this capability in a data-science setting. DataSciBench evaluates language models using realistic data-science tasks rather than only isolated code generation.

Another benchmark, AgentDS, compares AI-only and human-AI workflows across domain-specific data-science challenges. The agents produced executable pipelines and ran iterative experiments. However, they often returned to generic modeling strategies when the task required domain knowledge.

A similar situation appeared in SciAgentArena. The agent became more useful when the analysis workflow and evaluation criteria were clearly specified. The performance was less reliable when the research question was open-ended.

For me, the results show that the coding capability is only one part of the workflow.

The AI agent might know how to build a churn model, but it does not automatically understand what churn means for our project.

Share

The Data Science Problem Is More Than the Dataset

Let’s use customer churn prediction as an example.

The dataset might contain a snapshot_date, customer information, and a target named churned_next_30d. From these columns, the AI agent could identify the prediction target, encode categorical variables, and notice that the target was imbalanced.

However, several important questions were still unanswered.

When would the prediction happen? Would the customer be scored at the start or the end of the month? Was cancellation_date already available during the prediction? Should the model use random validation or future data? How many customers could the retention team contact?

Another question: Was the model predicting churn risk, or estimating the effect of sending a retention offer?

These questions would change how we prepare and evaluate the model. However, we could not answer them reliably from the column names.

To provide the missing information, I created a prediction contract for the project.

# Customer churn prediction contract

At the end of every calendar month, the retention team scores customers
with an active subscription.

The target indicates whether the customer cancels during the 30 days
following snapshot_date.

Only information known by the end of snapshot_date may be used.
Cancellation dates, the account status after 30 days, and retention-offer
outcomes are not available at prediction time.

Use records before July 2025 for development. Records from July through
December 2025 are the untouched final evaluation period.

Average precision is the primary metric. The retention team can contact
only 10% of customers, so also report recall among the 10% with the
highest predicted risk.

The evaluation measures risk ranking. It does not estimate the causal
effect of contacting a customer.

The document did not teach the agent how to create a Scikit-Learn pipeline. The agent already knew how.

Instead, the prediction contract explained decisions the agent should not make on its own.

Share

Connecting the Project Context with the AI Agent

Repository-level instructions are one way to connect the project knowledge with an AI agent.

VS Code supports several instruction files depending on the selected agent, including .github/copilot-instructions.md, AGENTS.md, CLAUDE.md, and scoped instruction files. The VS Code documentation describes this as a context-engineering workflow, where project knowledge stays available across agent interactions.

For the experiment, I used a short AGENTS.md file.

# Project instructions

Before changing data preparation, feature engineering, training, or
evaluation, read docs/prediction-contract.md.

Treat the prediction moment, target window, unavailable information,
validation design, and evaluation metrics in that document as binding
project constraints.

Do not silently replace them with generic machine-learning defaults.

The final implementation must be runnable from the command line. Save the
fitted pipeline and a JSON metrics file, then run the finished command.

The AGENTS.md file was intentionally short. Its purpose was to tell the agent where the important context was and how to use it.

The detailed information stayed inside docs/prediction-contract.md. OpenAI also described a similar pattern in its harness-engineering article, where a short AGENTS.md file points the agent toward deeper project knowledge.

For a data-science project, the deeper knowledge could include the prediction definition, data availability, validation design, metric reasoning, and operational constraints.

The flow was simple. The AI agent read AGENTS.md, while AGENTS.md directed it to the prediction contract. The contract then provided the data-science decisions used during model development.

Experiment Setup

For the experiment, I generated a synthetic customer-month dataset with 9,000 rows. The data covered January 2024 to December 2025, and the overall churn rate was around 2.38%.

The dataset also contained several risky columns:

• cancellation_date

• account_status_30d

• retention_offer_accepted

The first two columns perfectly revealed the target because they were created after the prediction moment. Meanwhile, retention_offer_accepted represented an action that happened after customer scoring.

I then prepared two separate project repositories. Both had the same dataset, package versions, README file, and prompt. However, only the second repository contained AGENTS.md and docs/prediction-contract.md.

Both experiments used OpenAI Codex CLI 0.147.0 with the gpt-5.6-sol model and medium reasoning effort. Each condition was run once in a fresh session.

The prompt was the following:

Build a complete, reproducible customer churn model using the data in this
repository. Train and evaluate the model, save the fitted pipeline and a
machine-readable metrics file, and document the important modeling decisions.
Work independently and run the finished pipeline before you stop.

The experiment was not designed as a benchmark for Codex. One run for each condition would not be enough to make a general performance claim. Instead, I wanted to observe which decisions came from the dataset and which came from the project context.

Reproducing the Experiment

To make the experiment accessible, I published the complete project in a public GitHub repository.

The repository contains the 9,000-row synthetic dataset, the data-generation script, the exact prompt used in both runs, both training pipelines, the AGENTS.md file, the prediction contract, the saved metrics, and the figures used in this article. The data is synthetic and contains no real customer information.

The README explains how to install the required packages and rerun each condition. This lets readers inspect each agent's decisions instead of relying only on the metric values shown here.

The final evaluation periods selected by both runs are shown below.

Even though both agents received the same prompt and data, they reserved different months for the final evaluation. This difference also means that the average-precision scores could not be directly compared.

Experiment Result Without Project Context

The result without project context was better than I expected.

The agent inspected the suspicious columns and their relationship with the target. It found that cancellation_date and account_status_30d perfectly revealed churn. The agent correctly identified them as outcome leakage and excluded them from the features.

It also excluded retention_offer_accepted and customer_id.

The agent noticed that the data covered 24 monthly periods. Therefore, it avoided random validation and created expanding-window validation folds. It selected average precision as the main metric, compared several models, and saved a reproducible pipeline.

These were reasonable data-science decisions.

However, the agent still had to decide on information the project did not provide.

It selected September to December 2025 as the untouched test period. The agent reported outreach performance at 5%, 10%, and 20% capacity. It selected the model using the highest mean validation average precision and created another decision threshold based on the F1 score.

None of the decisions were necessarily wrong. However, none came from the project owner either.

The final evaluation included 1,500 rows and 43 churners. The model achieved 0.0836 average precision and captured 12 churners within the monthly top 10% of the risk scores.

The agent produced good code, but it also created part of the prediction problem.

Experiment Result With Project Context

The second agent received the prediction contract through the project instructions.

It reserved July to December 2025 as the final evaluation period, resulting in 2,250 rows and 67 churners. The agent used only pre-July data for model development and maintained time order during validation.

It compared a class-weighted logistic regression baseline with a gradient-boosting model. The prediction contract also provided the selection rule: the challenger could replace the baseline only when it improved average precision without reducing recall at the 10% outreach capacity.

Gradient boosting did not pass the requirement. Therefore, the agent kept the logistic-regression baseline.

The final model achieved an average precision of 0.0748. Within the 10% outreach capacity, the model captured 18 of 67 churners, which resulted in 0.2687 recall.

The agent also saved the prediction moment, target window, excluded columns, validation period, baseline comparison, and operational metric inside the JSON metrics file. The limitation that the model did not estimate the causal effect of a retention offer was also recorded.

The image shows selected information from the actual metrics.json files. The context run did not only use the required decisions during training. It also kept them as machine-readable evidence for future work.

The overall decision comparison is shown below.

The project context did not suddenly make the AI agent better at Python or machine learning. Both runs were already technically capable.

The difference was that the second agent solved the prediction problem defined by the project rather than selecting its own version of the problem.

What Did the Project Context Change

The prediction contract did not contain any model code. It didn't explain logistic regression, tune the hyperparameters, or provide the feature-transformation pipeline.

What it changed was the number of decisions the agent needed to make.

This difference matters because many data-science problems aren't caused by Python errors. The training script might run successfully, the metric might be calculated correctly, and the model file might be saved.

However, the result could still be misleading when the target was defined at the wrong time, future information entered the features, or the evaluation did not represent how the model would be used.

For the churn project, one of the most important sentences was the following:

At the end of every calendar month, the retention team scores customers with an active subscription.

The sentence defined the prediction moment. From this information, the agent could determine what data was available, how to form the target, and why time-based evaluation was required.

Another important sentence was:

The retention team can contact only 10% of customers.

This information turned the task from a generic classification problem into a ranking problem connected with an operational decision.

The value of context engineering was not providing more information in general. It provided the specific information the AI agent could not reliably infer.

What Should Remain as Human Work

Project context could preserve human decisions, but it could not make all the decisions for us.

The data scientist still needed to decide whether churn was the right target, whether the 30-day window matched the business action, and whether the historical customers represented future conditions.

We also needed to decide whether a risk model was the correct method. A model that ranks customers by churn risk does not automatically tell us which customers would respond to a retention offer.

After we defined the decisions, the AI agent could help translate them into the data pipeline, validation process, metrics, and model artifacts.

Conclusion

AI coding agents already understand a lot about Python, Scikit-Learn, and common machine learning workflows. Repeating the same coding explanation in every prompt might not be the most valuable use of project instructions.

More valuable context was the information the code could not explain: when the prediction happened, what information was available, how to simulate the future, which metric represented the operational decision, and what the model result could not prove.

In my experiment, the agent without project context still produced a good pipeline. However, it still had to make several important decisions on its own.

The agent with project context solved the version of the prediction problem that the project had already defined.

For me, this is the role of context engineering in data science.

It does not teach the AI agent how to code the model. It teaches the agent which modeling problem we are trying to solve.

I hope it has helped!

Need More Personal Guidance?

If you are working on your Data Science or AI career and want more personalized guidance, I also offer mentorship through MentorCruise.

We can discuss your career direction, skills, portfolio, technical challenges, or what you should focus on next.

→ Check out my mentorship on MentorCruise

Previous module: Module 8, Choosing between INNER JOIN and LEFT JOIN.

An order total can be correct in the orders table but become wrong after a join. This happens when one order matches several item rows. SQL repeats the order value on every matched row, and SUM counts every copy.

In this module, we’ll learn how to check the row count after a join. We will also aggregate each child table before joining when the report needs one row per order.

Subscribe now

Read more

Previous module: Module 7, How to define a metric clearly.

An INNER JOIN and a LEFT JOIN can use the same columns but return a different number of customers. The difference comes from what happens when a customer doesn’t have a matching order.

In this module, we’ll compare both joins with a customer coverage example. After that, we’ll see why a WHERE condition can still remove customers from a left join.

The first thing we need to understand is which table defines the population. If the report should include every customer, the result still needs eight customers after the join. If the report is only about customers with an order, unmatched customers don’t belong in the result.

Subscribe now

Read more

Previous module: Module 6, WHERE and HAVING: filtering before and after GROUP BY.

Two people can calculate a conversion rate and still get different numbers. One person might count orders while the other counts customers. They might also use different dates to decide which rows belong in January.

The SQL can be correct in both cases because the problem starts with the metric definition. We still need to choose the numerator, denominator, counting unit, and time period.

In this module, we’ll use a January purchase cohort to compare two definitions. First, we’ll check what happens when the numerator and denominator use different date columns. After that, we’ll compare an order completion rate with a customer completion rate.

Subscribe now

Read more

Take a look at these two customer messages:

“How can I request a refund?”

“My refund still has not arrived.”

Both messages mention a refund, but the customers need different kinds of help. If we only search for the word “refund,” we would put them in the same group.

This is a familiar problem when working with text data. We have the information, but we still need to organize it before we can analyze it or use it for machine learning.

Jev is one model we could use for this task. Developed by TypeSafe, Jev answers questions about text by returning a category, score, or probability that we can use in our code.

Many articles already introduce Jev and how it can help our work. However, what makes it interesting for data science? For me, it is the ability to turn a question about our data into a column we can inspect and evaluate.

In this article, we will explore what Jev does, compare it with other approaches, and walk through a Python example. We will also analyze published Jev predictions to see what happens when we only keep answers with high confidence.

Before we continue, a note about the experiment: I did not have Jev API access for this article. The API code is a documentation-based example that I have not run. The results and screenshots come from an analysis I ran on publicly available predictions so that you can reproduce that part without an API key.

Subscribe now

What is Jev

Let’s return to the refund messages. We could give Jev a message and ask it to choose between “requesting a refund” and “checking a refund.”

There are two main parts to this request:

• The state, which is the content we want to analyze.

• The questions, which describe what we want to know about that content.

TypeSafe calls Jev a System One model. The idea is to make specific decisions that our program can use, rather than produce a long written answer.

It is also worth remembering that Jev is a hosted model. Installing its Python client documentation lets us call the service; it does not install the model on our computer.

Jev provides three question types. Let’s look at what each one means.

Choice and Score also return a probability distribution and a confidence value. Noul returns the probability for the statement, without a separate confidence value. See the documentation for Choice, Score, and Noul.

For our refund example, Choice would be the starting point because we want to select a category. If we wanted to rate a document's relevance, we could use Score instead.

We can include several questions in one request. However, each question is answered independently using the same state. If one result should affect what happens next, we need to write that logic ourselves.

Share

Why would we use Jev for data science?

Suppose we have a dataset containing account information and customer messages. The account age and number of purchases are already available as columns. The messages might contain useful information too, but we cannot summarize their meaning with a simple numeric calculation.

We could ask whether each message mentions a cancellation, an unresolved issue, or a competitor. The answers could become additional features to test in our model.

This is the part I find useful. We can explain what a feature is supposed to represent and inspect the original message when its value looks wrong. That does not make the model’s internal reasoning transparent, but it gives us a clear definition for the column.

Why Jev is interesting for a data science workflow

Another interesting part of Jev is how we can use the result directly in our data workflow.

Normally, text data is difficult to use as a feature because we need to transform the meaning into something our analysis or machine learning model can understand.

With Jev, we can ask a specific question about the text and receive a category, score, or probability as the answer.

For example, suppose we have customer messages and want to know whether the customer is likely to cancel.

We could ask Jev:

Does the customer want to cancel?

The returned probability could then become another column in our dataset:

cancellation_probability

From there, we can use the value like any other feature. We could analyze it with pandas, apply a threshold, or include it in a machine learning model.

This is also where Jev fits well with ordinary Python code.

We do not need Jev to handle the entire workflow. We can use Jev for the part that requires understanding the text, while Python handles calculations and other deterministic logic.

For example, our workflow could look like this:

Customer message
      ↓
Jev
      ↓
Category or probability
      ↓
DataFrame
      ↓
Python logic or ML model

TypeSafe also uses this idea in its workflow evaluations. A larger task can be divided into several smaller Choice, Score, or Noul questions, while the surrounding application controls what happens next.

For me, this is probably one of the more useful parts of Jev for data science. We can convert information hidden inside text into values that are easier to analyze and test.

I would start with three applications.

1. Labeling text data

Imagine that we want to group survey responses into pricing, usability, reliability, and other feedback. We could describe these categories in a Choice question and ask Jev to suggest a label for each response.

The advantage is that we can begin with category descriptions rather than first training our own classifier.

However, these are still predicted labels. We should not treat them as verified answers just because they have been added to our DataFrame. We need to inspect a sample and measure the mistakes.

The experiment later in this article will focus on this issue.

2. Creating features for machine learning

For a customer churn model, a message asking to cancel might provide information that is missing from account activity alone. We could use a Noul question to create a cancellation-related feature, then compare our model with and without that feature.

TypeSafe includes combining text-derived probabilities with structured data among its suggested uses. This gives us an approach to try, not a guarantee that the new features will improve our model.

As a reminder, the message must be available when we make the prediction. A message written after the customer has already canceled would give our model information it should not have.

3. Selecting documents to review

We could also use Score to help organize a collection of documents. For example, we might describe what makes a paper relevant to a research question, then use the returned scores to decide which papers to inspect first.

This could be useful when we have many documents and a clear idea of what we are looking for. I would still check some of the low-scoring documents, because an incorrect score could hide something important.

These feature and document-review examples are possible applications. They were not tested in the banking-intent experiment below.

How does Jev compare with other approaches?

We already have several ways to classify text. Does that mean we need Jev as well?

Not necessarily. The choice depends on our data, the task, and how much control we need. Let’s compare the main options.

Rules and keyword matching

For exact patterns, I would start with ordinary code. Checking whether an ID follows a required format does not need a language model.

The difficulty appears when the meaning depends on the wording. In our refund example, finding the keyword is easy, but separating the two requests needs more information. We can keep adding rules, but they become harder to maintain as we encounter new expressions.

Jev lets us describe the distinction in words. Whether it handles those expressions correctly still needs to be tested.

Traditional text classifiers

Another option is to train a model using labeled examples. For instance, we could represent the text with TF-IDF features, which use weighted word occurrences, and train a classifier on those features. Scikit-learn has a text-classification example showing this approach.

If we already have representative labels and stable categories, I would include this as a baseline. A model we run locally also avoids sending every message to an external API.

Jev is different because we can change the category descriptions in the request without training new model weights. The limitation is that TypeSafe does not offer customer-specific fine-tuning for Jev. Instead, we adapt it through instructions and context.

Also, not needing training labels does not mean we can skip labeled data altogether. We still need trusted answers to check the predictions.

General-purpose LLMs

We could also ask a general-purpose LLM to classify our messages.

It would be misleading to say that Jev is the only option for structured answers. Models such as Claude support responses constrained by a JSON schema.

What I like about Jev’s approach is that the selected answer and the probabilities are already part of the response. We can use them to compare alternatives or decide which rows to review. We do not need to ask for a written statement about how confident the model feels.

This is not a capability unique to Jev, either. Traditional classifiers can provide probability estimates, and scikit-learn includes methods to check and improve their calibration.

If the task also requires writing an explanation or summary, I would consider a general-purpose LLM. If it only needs a defined label or score, Jev is another option worth evaluating.

Here is a short comparison.

This table compares how we would use the tools. Our experiment does not rank their performance.

What are Jev’s limitations?

I'd consider a few things before adding Jev to a project.

First, it is not a general-purpose writing model. TypeSafe also documents weaknesses with counting, exact arithmetic, date comparisons, and questions that require several reasoning steps. I would leave these calculations to Python.

Second, the input still matters. Unrelated information can reduce accuracy, and adversarial instructions inside the text can influence the answer. We should test those cases rather than assume a structured response is safe or correct.

Third, the current model accepts text, not raw images or audio. TypeSafe reports its strongest language performance in English. We also need to consider API availability, rate limits, and whether we can send our data to the service.

Finally, a high confidence value can still accompany a wrong answer. We will see two examples of this in the experiment.

For me, Jev is interesting for repeated text-labeling and scoring tasks. I would not use it to replace pandas calculations or assume it will outperform a classifier trained for our particular problem.

Using Jev with Python

Let’s look at how we could add a topic and an urgency value to a small feedback dataset.

To run this example, we need a TypeSafe account, an API key, and Python 3.10 or newer. First, install the packages:

pip install typesafe-sdk pandas

The client reads the key from the TYPESAFE_API_KEY environment variable. Set it privately in your environment, and do not include it in a notebook you plan to share.

The following example uses three sample messages. I have not run this API call, so there is no claimed model output for these inputs.

import pandas as pd
from typesafe_sdk import Choice, Noul, TypeSafeClient
 
feedback = pd.DataFrame({
    "message": [
        "My refund still has not arrived.",
        "I cannot sign in after resetting my password.",
        "Please restore access today; my team cannot work.",
    ]
})
 
questions = {
    "topic": Choice(
        instructions="Select the main topic of this feedback.",
        criteria={
            "billing": "Charges, refunds, invoices, or payments.",
            "access": "Signing in, passwords, or account access.",
            "other": "A topic outside billing and account access.",
        },
    ),
    "urgency": Noul(
        instructions="Does the message explicitly express time urgency?"
    ),
}
 
records = []
with TypeSafeClient(model="jev-1.13.0") as client:
    for message in feedback["message"]:
        response = client.system_one(state=message, questions=questions)
        topic = response.answers["topic"]
        records.append({
            "jev_topic": topic.choice,
            "jev_topic_confidence": topic.confidence,
            "jev_urgency_probability": response.answers["urgency"].noul,
            "jev_model": response.model,
        })
 
enriched = feedback.join(pd.DataFrame(records, index=feedback.index))
print(enriched.to_string(index=False))

In the code above, feedback contains the messages we want to analyze. We define a Choice question for the topic and a Noul question for urgency, then send each message to Jev.

After receiving an answer, we collect the values and join them back to the original DataFrame. Keeping the message beside its predictions makes it easier to inspect the result. The request uses the documented Python client interface.

We provide descriptions for the topic categories. These descriptions help separate the options. The other category also gives the model a place to put feedback that doesn't fit billing or account access.

That price makes it worth considering for repeated labeling, but it doesn't cover the full cost of the work. We still need to account for retries and review. This example does not measure API cost or demonstrate savings over another model.

For a larger dataset, I would first test a small sample, add error handling, and save progress between batches. The loop above is intended to explain the basic usage.

Experiment: Checking Jevs predictions with pandas

Now that we understand the basic usage, let’s look at some recorded results.

AY Automate published Jev predictions for a banking-intent classification task. For this analysis, I selected the 77-category task, which contains 231 predictions: three examples per category. The publisher reports that the original model calls ran on September 19, 2026.

I have a specific question: can we use confidence to select more accurate labels, and how many rows would still need review?

This is an analysis of the published answers, not a new run of Jev.

Load the data and check the predictions

Download the published results and save the file as recorded_results.jsonl. We can then load it with pandas:

import pandas as pd
 
raw = pd.read_json("recorded_results.jsonl", lines=True)
df = raw.loc[
    raw["system"].eq("typesafe/jev-1.13")
    & raw["task"].eq("intent77")
].copy()
df["correct"] = df["ok"].eq(True) & df["pred"].eq(df["gold"])
 
print(df[["gold", "pred", "conf"]].head(3).to_string(index=False))
print(f"Accuracy: {df['correct'].mean():.1%}")

There are three columns we need to understand:

• gold contains the expected label from the dataset.

• pred contains Jev’s prediction.

• conf contains the recorded confidence value.

We compare the predicted and expected labels to create the correct column. If the request failed, we count it as incorrect.

Here is the output:

                 gold                  pred  conf
Refund not showing up Refund not showing up  1.00
Refund not showing up        request refund  0.52
Refund not showing up        request refund  0.79
Accuracy: 78.8%

In the first three rows, the expected category is the same, but the predictions are not. Two rows were classified as request refund instead of Refund, which is not showing up.

Across the full subset, 182 of the 231 predictions were correct, giving us 78.8% accuracy.

For me, this result is a reason to inspect the labels before using them. Let’s see whether confidence helps us select a more reliable subset.

Select a confidence threshold

A threshold is simply the minimum confidence we require before keeping a prediction.

We should not choose that minimum and evaluate it on the same rows. Otherwise, we could keep adjusting it until the result looks good for the data we have already seen.

For this example, I used 77 rows to choose the threshold and kept the remaining 154 rows for evaluation. The split follows the source IDs: those ending in -0 go into the selection subset, and the others go into the evaluation subset.

This gives us one selection example and two evaluation examples per category. It is a small split of public data, not a separate production dataset.

We can perform the selection with the following code:

cal = df.loc[df["id"].str.endswith("-0")].copy()
test = df.loc[~df["id"].str.endswith("-0")].copy()
 
def evaluate(data, threshold):
    accepted = data.loc[data["ok"].eq(True) & data["conf"].ge(threshold)]
    return {
        "threshold": threshold,
        "accepted": len(accepted),
        "correct": int(accepted["correct"].sum()),
        "accuracy": accepted["correct"].mean(),
        "coverage": len(accepted) / len(data),
    }
 
thresholds = [0, .5, .7, .8, .9, .95, .99]
sweep = pd.DataFrame([evaluate(cal, t) for t in thresholds])
eligible = sweep.loc[(sweep["accuracy"] >= .95) & (sweep["accepted"] >= 10)]
if eligible.empty:
    raise ValueError("No threshold met the rule; keep manual review.")
chosen = eligible.sort_values(
    ["accepted", "threshold"], ascending=[False, True]
).iloc[0]
threshold = float(chosen["threshold"])
print(sweep.round(4).to_string(index=False))

The code tries seven thresholds. From the thresholds that reach at least 95% accuracy and keep at least 10 selection rows, it chooses the one that keeps the most rows.

These requirements are settings for our example. They are not universal rules for using Jev.

The selected threshold was 0.99. On the selected subset, it kept 32 predictions, of which 31 were correct.

Next, let’s apply that same threshold to the 154 evaluation rows:

result = evaluate(test, threshold)
print(result)

As the result shows, the filter kept 67 predictions, and 65 were correct. This gives us 97.0% accuracy among the accepted predictions.

Without the filter, Jev was correct on 124 of the 154 evaluation rows, or 80.5%.

The filtered result looks better, but we need to check how much data remains. Only 43.5% of the evaluation rows passed the filter. This percentage is called coverage. The other 87 rows, or 56.5%, still need review.

In other words, we have not made Jev 97% accurate on the whole dataset. We have selected a smaller group of answers that were more accurate in this experiment.

This helps plan the labeling process because we can see both the quality of the selected predictions and the remaining work.

Inspect the incorrect answers

Two mistakes remain among the accepted predictions. Let’s take a closer look.

test["review_required"] = ~(
    test["ok"].eq(True) & test["conf"].ge(threshold)
)
test["candidate_label"] = test["pred"].where(~test["review_required"])
 
errors = test.loc[
    ~test["review_required"] & ~test["correct"],
    ["gold", "pred", "conf"],
]
print(errors.to_string(index=False))
 
test[["id", "pred", "conf", "candidate_label", "review_required"]].to_csv(
    "label_review_queue.csv", index=False
)

The code marks rows that need review and creates a candidate_label column for the predictions that passed the filter. It also saves the result to label_review_queue.csv.

I use the name “candidate label” deliberately. Passing the filter does not turn a prediction into a verified answer.

The two accepted mistakes are shown below.

The first prediction was failed transfer, while the expected label was beneficiary not allowed. The second was transfer timing, while the expected label was transfer not received by recipient.

Notice that the second answer had a confidence of 1.0. It was still incorrect.

TypeSafe explains that confidence summarizes how concentrated the probabilities are around an answer. It should not be read as a guarantee that a particular prediction is correct.

The source file contains the labels and predictions, but not the original messages. That means we can identify these disagreements, although we cannot explain exactly why the model made them from this file alone.

What did we learn from the experiment?

What I find useful is that we can separate the predictions into two groups: those we might use after further checks and those that need review.

However, I would not take the 0.99 threshold and apply it directly to another dataset. Different categories, languages, and message styles could produce different results.

For my own dataset, I would start with a small human-labeled sample. I would use part of it to improve the category descriptions and choose a threshold, then evaluate the final setup on data I had kept separate. I would also compare it with a simple classifier rather than assume that Jev is the best option.

One more point to remember for EDA: If we only count categories among the accepted rows, we may get a misleading picture of the full dataset. The accepted rows could contain easier examples or a different mix of topics. We should keep the rejected rows visible in our analysis.

For feature engineering, we would need another experiment. We would compare the prediction model with and without the Jev-derived features, using the same evaluation data. Better labels in this example do not prove better churn predictions.

The size of this experiment is also a limitation. Only 67 evaluation predictions passed the filter. Their approximate 95% Wilson interval is 89.8% to 99.2%, which shows how uncertain the accuracy estimate remains with a small sample. That interval does not account for every dependency in the data.

I did not independently verify the published responses against provider logs or measure new inference speed, API costs, or human-review time. The companion files include the executable analysis and source-file checksum so that readers can check the calculation.

Conclusion

Jev is a model for answering defined questions about text. We can use its categories, scores, and probabilities as values in our data science projects.

For me, its main attraction is being able to try a text-labeling task or a new feature without first training a separate classifier. Its limitations include reliance on an external API, a narrower range of tasks, and answers that still need validation.

In the experiment, a confidence filter selected 67 of 154 evaluation predictions, 65 of which were correct. The remaining 87 rows still needed review. This shows a possible way to organize labeling work, but it does not establish that Jev is better than every alternative.

I would begin with one text column and one clearly defined task. Check the predictions, compare them with a simple baseline, and decide whether the result is useful before processing the rest of the dataset.

WHERE filters rows before SQL creates the groups. HAVING filters the groups after SQL calculates COUNT, SUM, or another aggregate.

Both clauses can remove data, but moving a condition between them can change the question we answer. A status filter in WHERE might remove orders that should still be included in a customer’s total.

In this module, we’ll compare two similar requests. The first needs customers with at least two total orders and one delivered order. The second needs customers with at least two delivered orders.

Upgrade to continue with the remaining modules and keep the full set of course resources.

Subscribe now

Read more

GROUP BY combines rows that have the same value and returns one result row for each group. We often use it with COUNT, SUM, or AVG to calculate a metric for each category.

The query itself is usually short, but there are two things we need to check. First, what does one row in the source table represent? Second, what should one row in the result represent?

In this module, we’ll compare item rows, orders, and customers. After that, we’ll change the grouping columns to produce one row per month instead of one row per month and status.

Share

Read more

ORDER BY decides how SQL arranges the rows, while LIMIT decides how many rows remain in the result. We need both when a request asks for the top customers, the lowest sales, or the latest orders.

However, we still have one decision to make. What does “top” mean for the question we are answering? A customer can be at the top by total spend, number of orders, or recent growth. The query can run correctly while sorting by the wrong value.

In this module, we’ll use a small customer table to find the highest and lowest spend values. After that, we’ll compare two ways to handle customers who have the same value at the cutoff.

Share

Read more

We use WHERE to choose which rows we want from a table. It is one of the first SQL clauses we learn, but the rows we get still depend on how we write the condition. For example, order_status <> 'canceled’ removes canceled orders. It also removes orders whose status is NULL, even though we didn’t ask it to.

In this module, we’ll try that filter and see why the missing statuses disappear. After that, we’ll use WHERE to find orders from one day in São Paulo when the timestamps are stored in UTC.

Share

Read more

I have worked in data science for almost a decade now. During that time, I have seen many tools become popular. There were Hadoop and Spark, followed by easier machine learning libraries, cloud platforms, and now generative AI tools that can help us write code.

I learned many of these tools along the way. Some are still part of my work, while others do not appear much in my daily projects anymore. However, when I think about the skills I still use regularly, many of them are not new at all.

The World Economic Forum’s 2025 report expects 39% of workers’ core skills to change by 2030. This prediction is for jobs in general, not only data science. At the same time, analytical thinking remains the core skill most commonly identified by employers. For me, that describes data science quite well: we need to keep learning new tools, but we also need to improve how we think about the work.

So, which skills have aged well in my experience? I also looked at what other data professionals have learned to see where our experiences are similar or different.

Let’s get into it.

Subscribe now

SQL Is Still Everywhere

SQL was already an important skill when I started working with data, and it is still important today. According to O*NET’s analysis of U.S. data scientist job postings in 2025, SQL appeared in 51% of postings. For comparison, TensorFlow appeared in 11%, PyTorch in 10%, and Spark in 7%.

Of course, job postings do not tell us everything about what data scientists do. Still, they show that SQL remains useful even as the tools around it change.

What changed for me is how I use SQL. Earlier in my career, I focused on remembering the syntax: joins, subqueries, window functions, and so on. I still need to know them, but writing a valid query is not the difficult part of many analyses.

Take an example of joining a customer table with a transaction table. The query might run without any errors, but one customer could have many transactions. If I count the rows after the join without checking the data, I could report the number of transactions as the number of customers.

That is why understanding the data grain, table relationships, and metric definition matters more to me than memorizing every SQL command. AI can help generate the query, but I still need to check what the result represents.

Avery Smith, who has worked in analyst, data scientist, and data engineer roles, also wrote that he learned many tools on the job. In his experience, domain knowledge helped him understand the problem and interpret the data. I agree with that point. Knowing SQL gives us access to the data, but understanding the context helps us get the correct answer.

Statistics Helps Me Question the Result

Statistics is another skill I appreciate more now than when I started. At first, I mostly associated it with formulas, distributions, hypothesis tests, and model assumptions. Those topics are still important, but statistical thinking has become useful to me in a more practical way.

For example, imagine that an experiment shows a new feature performs better than the old version. Before accepting the result, I would want to know whether the groups were comparable, whether the sample was large enough, and whether the difference is meaningful for the business. I would also ask if we selected the metric before seeing the result.

We can run a statistical test in a few lines of Python. The function can give us a number, but it cannot decide whether we asked a good question or collected suitable data. The U.S. Bureau of Labor Statistics still includes mathematics, statistics, analytical skills, and logical thinking among the important parts of a data scientist’s work.

For me, the skill that aged well is knowing what we can conclude from an analysis, and when the evidence is not enough.

Share

Understanding the Problem Comes Before the Model

Earlier in my career, I was often excited about choosing a machine learning model. Should I use Random Forest? Would Gradient Boosting perform better? Could I improve the score by tuning more parameters?

Now I am more interested in the question before modeling: what decision are we trying to support?

Take a churn prediction project as an example. A model could identify customers who are likely to leave. However, we still need to know what the team can do with that information. Can they contact these customers? How early do they need the prediction? What kind of action could actually help? Without answers to these questions, even an accurate model might not be useful.

A 2025 study in Harvard Data Science Review compared data science with decision science skills. The researchers found that data science tends to emphasize coding, machine learning, and data management, while decision science gives more attention to problem framing and business knowledge. The study argues that we need both to turn analysis into useful decisions.

This also matches Yu Dong’s reflection on six years in data science. She explains how understanding the product and talking with other teams helped her find work that people could act on. One point from her experience adds to mine: we should not underestimate dashboards and reporting. A reliable dashboard can establish the baseline and shared metric definitions we need before attempting a more advanced model.

Sometimes the best solution is a model. At other times, it could be an experiment, a dashboard, a simple rule, or better data collection. I have become more comfortable choosing among these options only after I understand the problem.

Messy Data Still Needs Context

Data tools have improved a lot. We have cloud warehouses, better pipelines, automated checks, and AI tools that help write data preparation code. Yet messy data has not disappeared.

In dbt Labs’ 2025 survey of analytics professionals, 57% of respondents said they spent most of their workday maintaining or organizing datasets. The survey covers several analytics roles, not only data scientists, but the result feels familiar to me.

We can remove duplicate rows or fill missing values with code. The harder question is why those values are there. A duplicate might be a data error, or it might represent a real second transaction. A missing value might mean “unknown,” “not applicable,” or a problem in the data pipeline. Each situation requires a different response.

The same problem appears in machine learning features. A variable might have a strong relationship with the target because it was created after the outcome happened. Using it would make our model look good during testing, but the information would not be available when we actually need to make a prediction.

For me, understanding how the data was created has aged better than any particular cleaning library.

Evaluation Shows Whether the Work Is Useful

Building a model has become easier. We can train several models quickly, use AutoML, or ask AI tools to help create the code. However, we still need to decide whether the result is better than what people already do.

This is where evaluation becomes important. We need to choose a relevant metric, keep the test data separate, and compare our solution with a realistic baseline. A recommendation model might perform better than a random suggestion, but can it outperform simply showing the most popular items?

A data scientist reflecting on ten years of experience described a three-month A/B test comparing personalized recommendations with a popular-items fallback. He also looked at operating costs and found that not every project produced enough value to continue. I find this comparison useful because it evaluates the model against a real alternative rather than a score in isolation.

The same idea applies to generative AI. A demo can look impressive, but we need representative examples, clear evaluation criteria, and a baseline before saying the new system is better.

Communication Is Part of the Analysis

Communication was a skill I underestimated when I started. I thought the technical work came first and presenting the result happened afterward. My view has changed.

For example, a model might achieve an ROC-AUC of 0.87. That number could be useful to another data scientist, but a business team may ask different questions. How many customers can we reach? What happens when the model is wrong? What should we do next?

Explaining the result in those terms also helps me check my own work. If I cannot explain why I chose a metric or what the result means, I probably need to look at the analysis again.

The Airbnb data science team described a similar lesson. They found that analysis had less impact when data scientists were too far from the people making decisions. Closer collaboration helped them understand the problem and make their findings more useful. Avery Smith makes a related point in his career reflection: communication at the beginning helps avoid the wrong analysis, while communication at the end helps people use the result.

For me, communication is no longer something that comes after data science. It helps define the work and determine whether the work has value.

Conclusion

I still think learning new tools is important. Python appeared in 66% of the same 2025 U.S. data scientist job postings, and AI tools are already changing how we work. We cannot ignore these changes.

However, after almost a decade, I worry less about knowing every new tool as soon as it appears. I would rather keep improving the skills that have helped me through several changes: understanding the data, questioning the result, framing the right problem, evaluating against a useful baseline, and explaining what the analysis means.

The tools will continue to change. These are the skills I expect to keep using.

If you want to learn further about SQL in the AI era, don’t miss my latest written SQL Course.