6 Methodology

This chapter outlines the modeling techniques and formulas used to develop our novel metrics. By leveraging modern machine learning methods and contextual game data, we move beyond conventional box score statistics to better capture player impact, decision-making, and offensive progression.

At the core of this methodology are models that estimate the value of individual actions—particularly throws—by considering both their strategic potential and execution outcomes. The full modeling approach is detailed in our MIT Sloan Sports Analytics Conference paper:
“A Machine Learning Approach to Throw Value Estimation in Professional Ultimate Frisbee”.

6.1 Features

The features used in our models are drawn from four primary contexts: thrower, receiver, throw, and game situation. These features were selected based on their relevance to decision-making and throw success in ultimate.

6.1.1 Thrower Context

  • Thrower X: Cartesian coordinates of the thrower’s location on the field.
  • Thrower Distance to end zone: Distance from the thrower to the attacking end zone.

6.1.2 Receiver Context

  • Receiver X: Cartesian coordinates of the intended receiver.
  • Receiver Distance to end zone: Distance from the receiver to the attacking end

6.1.3 Throw Context

  • Throw distance: Euclidean distance between thrower and receiver.
  • Throw angle: Angle of the throw in degrees (0° forward, 180° backward, positive right, negative left).

6.1.4 Game Context

  • Time remaining: Binary indicators for whether there are ≤5, ≤10, or ≤30 seconds remaining in the quarter as well as a numerical feature for number of seconds left

We intentionally excluded variables like quarter, score differential, and point number. This omission aligns with our goal of modeling general throw value and decision quality independent of game context or team strategy. The assumption is that, outside of extreme fatigue or time constraints, optimal throw selection should remain consistent.

6.2 Modeling Choices

To model throw outcomes and offensive progression, we tested multiple algorithms and selected the most robust and interpretable options.

6.2.1 Model Type

We use logistic regression for both the Completion Probability and Field Value models. Despite experimenting with XGBoost, we found that it often produced overfit, “pocketed” predictions that lacked generalizability. Logistic regression provided smoother and more realistic probability distributions across the field.

6.2.2 Data Design Decisions

  • No field flipping: We chose not to flip field coordinates for data augmentation. This preserves current defensive trends and directional biases.
  • Minimal game context: While richer game context might improve prediction in specific scenarios, we prioritized a general-purpose model applicable across all game states.

6.3 Completion Probability (CP) Model

The CP model estimates the likelihood that a given throw is completed, based on its spatial and contextual characteristics.

Model Definition

We train a logistic regression model on all recorded throws:

\[ \text{Completion}_i \sim \text{Bernoulli}(\pi(X_{th}^{(i)}, X_{t}^{(i)}, X_{r}^{(i)}, X_{g}^{(i)})) \]

Where:
- \(\pi(\cdot)\): completion probability function
- \(X_{th}^{(i)}\): Throw-specific features (distance, angle)
- \(X_{t}^{(i)}\): Thrower location features
- \(X_{r}^{(i)}\): Receiver location features
- \(X_{g}^{(i)}\): Game context features (e.g., time remaining)

6.4 Field Value (FV) Model

The FV model estimates the probability that a possession results in a goal, given the current thrower location.

Model Definition

We train a logistic regression model at the possession level:

\[ \text{Goal}_p \sim \text{Bernoulli}(\pi(X_{t}^{(p)}, X_{g}^{(p)})) \]

Where:
- \(\pi(X_{t}^{(p)}, X_{g}^{(p)})\): Predicted probability of scoring on point \(p\)
- \(X_{t}^{(p)}\): Thrower location features
- \(X_{g}^{(p)}\): Game context features

The FV model is analogous to expected possession value (EPV) metrics in other sports, capturing the strategic value of a field location given control of the disc.

6.5 Metrics (FV/CP models)

In this section, we introduce three families of metrics derived from our Completion Probability (CP) and Field Value (FV) models, as well as a composite metric that integrates both. These metrics are designed to provide deeper insights into player contributions and decision-making in ultimate frisbee, improving upon current metrics by accounting for both the probability of success and the strategic value of plays.

6.5.1 Expected Completion Probability (xCP)

CP Model

Formula:
\[ \text{xCP} = \text{CP}(X_t, X_r, X_{th}, X_g) \]

Description:
The Expected Completion Probability (xCP) quantifies the likelihood that a pass will be completed, based on the thrower’s and receiver’s locations (\(X_t\), \(X_r\)), the type of throw (\(X_{th}\)), and the game state (\(X_g\)). This metric improves upon traditional completion rates by factoring in the difficulty of each pass, providing a context-sensitive measure of performance.

While traditional metrics simply count successful passes, xCP incorporates contextual factors (e.g., throw type, game state) to more accurately reflect the challenge involved in each pass. This gives a clearer picture of a player’s decision-making and execution under different conditions.

6.5.2 Completion Percentage Over Expected (CPOE)

CP Model

Formula:
\[ \text{CPOE} = C - \text{xCP} \]

Where \(C = 1\) if the throw was completed and \(C = 0\) otherwise.

Description:
CPOE measures the difference between the actual completion of a pass and the Expected Completion Probability (xCP). A positive CPOE indicates that the player completed a difficult throw, whereas a negative CPOE suggests a failure to complete a pass that was expected to succeed. This metric highlights player performance beyond mere completion rates.

Raw completion percentages do not account for the difficulty of passes. CPOE emphasizes overperformance or underperformance relative to expected outcomes, allowing for a more nuanced evaluation of a player’s contributions, especially when assessing high-risk or high-reward decisions.

6.5.3 Expected Contribution (EC)

FV Model

Formula:
\[ \text{EC} = \begin{cases} \text{FV}_e - \text{FV}_s & \text{if } C = 1 \\ - \text{FV}_o & \text{if } C = 0 \end{cases} \]

Where:
- \(\text{FV}_s\): field value at the starting location
- \(\text{FV}_e\): field value at the end of the throw
- \(\text{FV}_o\): opponent’s field value at the end location (if turnover)

Description:
EC quantifies the net value gained or lost from a throw. If completed, it measures the change in field value from the throw’s start to its end location. If the throw is incomplete, EC reflects the opponent’s gain from the turnover. This metric allows us to assess the strategic value of each throw beyond simple yardage.

Standard yardage metrics don’t capture the impact of field position on game outcomes. EC takes into account both the throw’s success and its position on the field, providing a richer understanding of a player’s strategic impact. It is useful for evaluating decisions that move the team closer to scoring or losing possession in critical moments.

6.5.4 Adjusted Expected Contribution (aEC)

FV Model

Formula:
\[ \text{aEC} = \begin{cases} \dfrac{\text{FV}_e - \text{FV}_s}{1 - \text{FV}_p} & \text{if } C = 1 \\ - \text{FV}_o & \text{if } C = 0 \end{cases} \]

Where \(\text{FV}_p\) is the field value of the first throw in the possession.

Description:
aEC normalizes the EC value within a possession, ensuring that the sum of aECs across all throws in a goal-scoring possession equals 1. This adjustment eliminates biases from starting positions and allows for consistent credit attribution across multiple throws and players.

While EC effectively measures a throw’s value, its absolute value can vary significantly depending on field position and is typically quite small—especially when compared to the negative effects of turnovers, resulting with most players netting a negative contribution. aEC overcomes these limitation by standardizing the metric within a possession. This makes it possible to compare throws across different lengths, starting positions, and game contexts fairly, while also aligning the metric with traditional box statistics, enabling more intuitive and consistent performance evaluations.

6.5.5 Expected Throw Value (ETV)

FV and CP Models

Formula:
\[ \text{ETV} = \text{xCP} \cdot \text{FV}_e - (1 - \text{xCP}) \cdot \text{FV}_o \]

Description:
ETV integrates both CP and FV metrics to assess the expected net benefit of a throw before it occurs. It considers both the probability of a successful throw (xCP) and the potential value of a successful play (\(\text{FV}_e\)), while factoring in the risk of turnover (\(\text{FV}_o\)). This novel metric provides a comprehensive balance between risk and reward, offering the first of its kind framework for quantifying optimal decision-making strategies from any given location or context on the field. By integrating both strategic potential and execution risk, ETV captures the true value of a throw, considering both its offensive gain and defensive risk in a way traditional metrics cannot.

6.6 Metrics (Other)

6.6.1 Adjusted Offensive Efficiency (aOE)

Formula: \[ \text{aOE} = \text{logit}^{-1}(\beta_0 + \beta_1 \cdot \text{FV}_{start} + \beta_2 \cdot \text{OE}_{teammates} + \beta_3 \cdot \text{DE}_{defenders}) \]

Where:
- \(\text{FV}_{start}\) is the starting field value of the possession.
- \(\text{OE}_{teammates}\) is the average offensive efficiency of the teammates on the line with the player, considering only games where the player is not playing with them.
- \(\text{DE}_{defenders}\) is the average defensive efficiency of the defenders guarding the player during that possession.

Description Adjusted Offensive Efficiency (aOE) evaluates the efficiency of an offensive possession for a specific player, adjusting for the quality of the starting field position, surrounding team and the defensive matchups. The logistic regression used here allows us to account for relationships between these variables and the resulting offensive success.

  • Starting Field Value (\(FV_{\text{start}}\)): The position on the field at the beginning of the possession provides context for the potential scoring opportunities. This variable captures the offensive potential of the possession at the start, considering the field location’s contribution to eventual scoring probability.

  • Teammate Offensive Efficiency (\(OE_{\text{teammates}}\)): This term adjusts for the offensive capabilities of the teammates on the field with the player. Importantly, we use the average offensive efficiency of those teammates when the player is not playing with them. This helps isolate how the player impacts offensive efficiency independently of the player’s direct contribution to their teammates’ performance. This is crucial because it removes the potential bias where a player’s presence on the field might boost or diminish their teammates’ efficiency merely by their co-presence. This term thus represents the baseline offensive efficiency that the player can expect from their teammates when not directly influencing their play.

  • Defensive Efficiency of Defenders (\(DE_{\text{defenders}}\)): This factor accounts for the quality of defense the player faces during the possession. It incorporates the defensive efficiency of the players guarding the focal player. This adjustment allows us to isolate how difficult or easy a player’s offensive actions are based on the defensive resistance they face. By adjusting for this, we are able to better assess how the player performs relative to their matchups.

Together, these factors allow aOE to provide a comprehensive, context-aware evaluation of a player’s offensive performance much better than the standard offensive efficiency. By using logistic regression, we ensure that the effects of the starting field value, teammates, and defenders are modeled in a way that respects the relationships of real-world game situations.

6.6.2 Adjusted Defensive Efficiency (aDE)

Formula: \[ \text{aDE} = \text{logit}^{-1}(\beta_0 + \beta_1 \cdot \text{FV}_{start} + \beta_2 \cdot \text{DE}_{teammates} + \beta_3 \cdot \text{OE}_{offenders}) \]

Where:
- \(\text{FV}_{start}\) is the starting field value of the possession (same as in aOE).
- \(\text{DE}_{teammates}\) is the average defensive efficiency of the teammates on the line with the defender, considering only games where the defender is not playing with them.
- \(\text{OE}_{offenders}\) is the average offensive efficiency of the players the defender is matched up against during that possession.

Description Adjusted Defensive Efficiency (aDE) measures the defensive impact of a player during a specific possession. Just like aOE, it accounts for the context of the possession by adjusting for field position, the quality of the player’s teammates, and the offensive threats faced.

  • Starting Field Value (\(FV_{\text{start}}\)): Just as with aOE, the starting field value represents the initial context of the defensive position. This variable captures the defensive potential or challenge based on where the defense starts. A defensive possession that begins closer to the defending team’s end zone is generally more challenging, and this metric adjusts for that.

  • Teammate Defensive Efficiency (\(DE_{\text{teammates}}\)): This term accounts for the defensive capabilities of the teammates on the field with the player. As with aOE, we use the average defensive efficiency of those teammates when the player is not playing with them. This adjustment is essential because it isolates the player’s individual defensive contribution to the team’s effort, removing biases that may arise if a player is typically paired with high or low-performing teammates. The goal is to determine how well the player defends in a given context, relative to their teammates’ usual performance when they aren’t involved.

  • Offensive Efficiency of Offenders (\(OE_{\text{offenders}}\)): This factor accounts for the offensive strength of the players the defender is matched up against. Just as the offensive efficiency in aOE accounts for teammates, aDE adjusts for how difficult the matchup is for the defender by looking at the offensive efficiency of the player they are guarding. The better the offensive efficiency of the opposing player, the more challenging the defensive assignment, which should be reflected in the aDE.

By adjusting for these variables through logistic regression, we generate a more accurate representation of a player’s defensive impact significantly improving on the noisy defensive efficiency metric. The resulting aDE metric is more nuanced than traditional defensive stats (such as blocks) because it integrates contextual factors like field position, defensive support, and the quality of the opposing offense.

6.6.3 Involved Offensive Efficiency (IOE)

Formula: \[ \text{IOE} = \frac{\text{Scoring Possessions with Involvement}}{\text{Total Possessions with Involvement}} \]

Description:
Involved Offensive Efficiency measures how frequently a team scores on possessions in which a player is directly involved—either through a throw attempt or a catch attempt. Traditional offensive efficiency metrics treat all possessions the same regardless of player contributions. IOE filters possessions to only those where a player was active, providing a more accurate lens on their individual offensive impact. It identifies players who consistently contribute to successful outcomes when they touch the disc.

Where involvement is defined as a throw or reception attempt by the player on a given possession.

6.6.4 Involved Efficiency Improvement (IEI)

Formula: \[ \text{IEI} = \text{IOE}_{\text{player}} - \text{OE}_{\text{player}} \]

Description:
IEI measures the difference between a player’s IOE and their team’s overall offensive efficiency. It quantifies how a player’s involvement enhances (or detracts from) team performance. A large positive IEI suggests the player boosts offensive efficiency beyond average; a small or negative IEI may highlight inefficiencies that are otherwise masked by total team performance.

Where:

  • \(\text{IOE}_{\text{player}}\) is the player’s Involved Offensive Efficiency
  • \(\text{OE}_{\text{player}}\) is the player’s baseline offensive efficiency (total scores / total possessions)

6.6.5 Offensive Involvement (OI)

Formula: \[ \text{OI} = \frac{\text{Possessions with Involvement}}{\text{Total Offensive Possessions Played}} \]

Description:
Offensive Involvement captures how frequently a player is involved in offensive possessions—either by making a throw or reception attempt. OI acts as a usage metric for ultimate. High OI indicates a central role in the offense, while low OI may suggest a more peripheral role. In combination with IOE, IEI and other metrics, it helps contextualize whether a player is being over- or under-utilized relative to their impact.