You Are What You Eat: Training Data

SplatGPT will only ever be as good as the matches it studies, but data for this purpose is difficult to source. Nintendo provides no public match API. The community instead reverse-engineered the companion app, which only exposes your most recent 50 games. Thankfully, the same community (I'm an active developer there!) built uploaders, and stat.ink now hosts 15 million player-submitted battles: manna from heaven for this project.

The catch? Despite stat.ink parsing and cleaning the data, it's still noisy. Some capture incomplete builds (builds that were literally missing abilities), others come from casual modes, and new-weapon patches destabilize the meta instantly. Finally, the classic long-tail: a few power users upload thousands of matches while most contribute only a handful, and raw statistics start to lie.

Simply mirroring the dataset would make SplatGPT average, not good. So we curate on purpose: compensating for some biases while introducing others, to nudge the model toward winning patterns.

Addressing Power User Skew

Like most community datasets, stat.ink data suffers from a "long-tail" distribution: a handful of "power users" contribute thousands of matches, while the majority of players upload a few. Splatoon adds a second wrinkle: each match uploaded contains all eight loadouts, so the same high-rank players reappear repeatedly in top level lobbies. If left unaddressed, SplatGPT might overfit to the build preferences of frequently appearing players, rather than truly learning generalized strategies.

To mitigate this issue, we implemented a contribution cap. While stat.ink is anonymized, we developed a coarse discriminator through side-channel analysis to get a probabilistic sense of distinct players, while respecting the site's privacy policy. This discriminator is based on a key insight: the internal JSON representation of gear abilities provided by stat.ink is unsorted and built incrementally. We hypothesized that this data, when combined with the specific weapon ID, could serve as a surprisingly stable and unique fingerprint for a player's particular gear setup for that weapon, even if other players used logically identical abilities. Different players would likely have different internal orderings for the same effective build due to how they acquired or modified their gear. Since gear is expensive to modify, this is surprisingly stable.

The distribution above, generated using these likely "player fingerprints," starkly illustrates the long-tail phenomenon. The counts of likley-players follow a pattern typical of community contributions when viewed on a log-log scale, confirming the significant skew we aimed to address. Armed with this understanding of the distribution, we used our discriminator to cap likely-player contributions to 100 matches. While this limit is relatively high, it was important to balance value of data from more active (and often higher skilled) players against the risk of the model overfitting to the specific builds of the most active stat.ink users. Full disclosure, this specific number was chosen somewhat arbitrarily but ultimately proved effective.

Learning from Winners

Because SplatGPT's mission is to recommend builds that win, we intentionally biased the training dataset towards successful strategies.

We began by removing all non-competitive modes, leaving only ranked matches. Next, for each weapon, we enforced a 3:2 win-loss ratio (a 60% win rate), based on the rationale weaker builds are more frequently associated with losses. Specifically, for each weapon, we counted victories, calculated how many defeats exceed the target ratio, and randomly dropped those excess losses. This exact ratio was guided by domain expertise (I'm a former competitive Splatoon player), recognizing that tuning this hyperparameter exceeded our available compute capacity. By pruning losing builds, we ensured the training corpus captured strategies that were, on average, significantly more effective.

Staying Relevant: Temporal Filtering and Meta-Awareness

The Splatoon metagame is constantly evolving. Although Nintendo has stopped introducing entirely new weapons, balance patches regularly shift weapon viability as weapons ebb and flow in the meta. Immediately following these updates, players experiment widely, temporarily destabilizing the meta.

To address this, there were two blackouts:

• Two-week blackout after any patch introducing new weapon kits.
• One-week blackout following pure balance patches.

Additionally, we applied a linear undersampling strategy to older data, ensuring the training dataset subtly favored more recent builds and maintained relevance to the current competitive landscape without losing fundamental signal from old matches.

All in all, these filtering steps cut the dataset from the original 120 million builds (15 million matches x 8 builds per match) down to roughly 14 million, an 88% reduction. While this sounds pretty drastic, it's worked remarkably well in practice!

Feeding SplatGPT: Model Training

With our high-quality, strategically-biased training set of around 14 million builds in hand, it was finally time to feed it into SplatGPT and see whether the architecture could learn effectively.

Multilabel Classification

As we saw in Part 2A, builds are naturally represented as sets in Token Space. This means that next-token prediction techniques used by language models aren't really appropriate. Instead, SplatGPT predicts all tokens at once, treating gear completion as a multilabel classification task.

This means that for every possible ability token (e.g., buckets like Quick Respawn 12-14 AP), the model outputs an independent probability of whether that token should appear in the recommended gear set.

From Builds to Training Examples

Since SplatGPT is designed as a gear autocomplete tool, it needs to handle partial build inputs gracefully. Inspired by training methods from large language models, particularly the sliding-window approach for next-token prediction, we used the following technique:

1. Duplicate Entries:
   Each complete build is duplicated until we have k copies, keeping track of each duplicate's index.

2. Randomly Drop Tokens:
   We randomly generate a list of k distinct integers ranging from 1 to N-1 (the original sequence length), skewed towards fewer drops. Each duplicated build is assigned one of these integers, indicating how many tokens will be randomly dropped.

3. Padding
   To keep batches uniform in size during training, partial sets were padded to a fixed length using a special token.

This method provided a diverse set of incomplete builds, training the model to robustly predict missing abilities in a variety of realistic contexts, knowing that the true value is likely a strong build.

Loss Function: Binary Cross-Entropy with Logits

Given the multilabel nature of our task, we chose Binary Cross-Entropy with Logits (BCELogits) as our loss function, treating each token independently. Formally, the model minimizes:

• \( y_i \) = true label (1 if token should be included, 0 otherwise)
• \( z_i \) = model’s predicted logits (raw model outputs before sigmoid)
• \( \sigma(z_i) \) = sigmoid function applied to logits
• \( N \) = total number of possible tokens

BCELogits directly operates on logits, improving numerical stability by combining the sigmoid activation and BCE calculation into a single step. It encourages SplatGPT to confidently assign high logits to correct tokens and push logits of irrelevant tokens toward negative values.

Hardware Setup: Training SplatGPT

Training a model of this complexity required serious computational resources. Early-stage experimentation and rapid prototyping were handled comfortably on my personal machine, an NVIDIA RTX 2080 Ti GPU, which was perfect for quick iterations, debugging, and preliminary testing.

However, for the final production-level training run, I upgraded to a dedicated NVIDIA H100 GPU via a DigitalOcean droplet. This final run took approximately 62 hours of continuous GPU time, a substantial (and frankly risky) investment for a side project. Fortunately, the gamble paid off: the results exceeded my expectations, making the time and resource investment entirely worthwhile.

Emergent Behavior: The <NULL> Token Story

Possibly the most fascinating discovery during training was the unexpected behavior of the special <NULL> token. Initially, the idea behind <NULL> was straightforward: I wanted SplatGPT to generate builds even with no Build Space input, using <NULL> as a placeholder. Early experiments explicitly included <NULL> tokens in the training examples, but this immediately degraded the overall build quality-model performance suffered significantly even when <NULL> wasn’t involved. Worse, occasional inclusion caused severe overfitting whenever the <NULL> token appeared.

On a whim, and driven by a completely baseless hypothesis, I tried something counterintuitive: removing <NULL> entirely from the training data while leaving it as part of the model's vocabulary. Suddenly, <NULL> began to function as a "blank slate" token: when prompted with nothing but <NULL>, the model consistently generated highly effective, weapon-specific builds—even though the model had NEVER explicitly trained on such cases. I was beside myself.

This unexpected behavior was genuinely astonishing. It strongly suggested the model wasn't merely regurgitating memorized training examples; rather, it appeared to be reasoning deeply about gear optimization principles. The <NULL> token demonstrated SplatGPT’s ability to generalize beyond explicitly trained scenarios, indicating true emergent understanding.

This discovery became one of the most compelling insights from the entire training process, significantly boosting my confidence in SplatGPT’s capabilities.

Evaluating SplatGPT's Performance: Results

With training complete, it was finally time to put SplatGPT’s recommendations to the test.

Evaluation Setup

For a quantitative evaluation, I used a specialized dataset generously provided by sendou.ink, a site run by a prominent community member that hosts builds shared by players. The site's owner, Sendou, shared with me builds submitted by verified high-level competitive players, categorized into three tiers. For this evaluation, I'm focusing exclusively on the top-tier, consisting of roughly 30 of the absolute best players in the world at any given time. This is truly the cream of the crop.

While this dataset is completely separate from my original training set, it's important to note it's not exactly a "holdout" set. Because of the significant influence these top players have, many of their builds are likely reflected in the broader player community, indirectly appearing in the filtered training data due to their effectiveness in winning games. Nevertheless, explicitly evaluating against builds directly sourced from top players remains a brutally tough benchmark.

• "Accuracy" here strictly means predicting the exact same build.
• Even when SplatGPT’s predictions differ slightly from these reference builds, the recommendations might still be highly effective and viable in practice.

Both factors mean this evaluation intentionally stacks the deck against SplatGPT.

To rigorously test SplatGPT’s capabilities, I randomly masked between 1 to 8 abilities from each original build (out of the 12 total slots in Build Space). Masking was done directly on build slots, then converted into Token Space for model inference. The scenario with 8 masks is particularly brutal, dropping 66% of the original build information.

Finally, inference was performed using beam search with a width of 5, and results were evaluated in two ways:

• Output (Best of 1): Accuracy (or Hit Rate) using only the top beam candidate, also known as Accuracy@1 or Hit@1.
• Full Beam (Best of 5): Accuracy if the correct build was anywhere within the top 5 candidates, also known as Accuracy@5 or Hit@5.

This methodology is intentionally aligned with standard best practices for evaluating state-of-the-art language models.

Quantitative Results: Exact Hit Rate and Mean Accuracy

Exact Hit Rate

This plot illustrates the percentage of times SplatGPT exactly matched the ground truth builds. As expected, the hit rate drops significantly as more abilities are masked, reflecting the increasing difficulty of the task. While these exact-hit rates may appear low at first glance, they're actually quite impressive given the extreme challenge involved—especially with up to 66% of build information removed. Successfully reconstructing builds exactly under these conditions highlights the model's exceptional ability to capture complex, context-dependent relationships from minimal information.

Mean Accuracy

While Exact Hit Rate measures only perfect matches, Mean Accuracy gives a more nuanced perspective on how close SplatGPT's recommendations are. Even if the model doesn't recreate the build exactly, we still want to see how closely it matched the intended strategy. For instance, if the original build has one main slot of Quick Respawn (10 AP) and three subs of Ink Saver (Main) (9 AP), flipping their positions would count as four incorrect slot predictions—even though strategically, these two builds are nearly identical!

The plot clearly shows that SplatGPT maintains impressive accuracy at the AP-level, consistently exceeding 80% accuracy even under severe masking conditions. Slot-level accuracy understandably decreases faster with heavier masking but remains significantly better than random guessing. This shows the model retains a strong grasp of meaningful relationships between abilities and gear slots, even when a majority of the build information is missing.

Together, these metrics demonstrate that while exact matches become increasingly challenging as masking increases, SplatGPT consistently identifies effective, strategically sound builds, successfully capturing the complex nuances essential to gear optimization.

Qualitative Insights: "Inaccuracy" is not "Inferiority"

I hinted at this earlier, but these quantitative accuracy measures have a significant limitation: they're measuring how closely SplatGPT matches a specific set of builds from top players, not necessarily how strong or viable the alternative builds it proposes are. A build that differs from the reference isn't necessarily a "mistake"; often it's just a different yet equally effective strategy.

To illustrate this, let me share a real-world anecdote. For context, the last time I actively played Splatoon competitively was around August 2024, shortly before starting serious work on SplatGPT. Fast-forward to May 2025: while finalizing my beam-search implementation, I noticed something deeply confusing. SplatGPT kept recommending a single sub-slot of Sub Resist Up, a so-called "utility sub" (a strategically chosen ability offering significant value for minimal investment), for the weapon I was most familiar with. This recommendation completely baffled me, because the meta as I remembered it absolutely did not value Sub Resist Up as a utility sub. Convinced this had to be some sort of bug or oversight, I began digging through logs and debugging code. Frustrated, I vented to my co-developer for the splatgpt.ink website about the confusion:

To my amazement, it turned out the confusion wasn't due to a bug at all. Instead, SplatGPT had anticipated a subtle meta shift occurring after my October 2024 data cutoff. The competitive community had gradually rediscovered the strategic value of using Sub Resist Up, specifically due to its interaction with certain high-impact weapons. Although signs of this shift existed as early as October, it hadn't fully taken hold until months later. In other words, SplatGPT had successfully predicted a meta shift, a shift I initially mistook for a bug, simply because the model had identified something strategically valuable before I did.

This realization was genuinely shocking. If the emergence of the <NULL> token behavior hadn't already convinced me, this experience hammered home the idea that SplatGPT isn't just blindly mimicking player strategies, it's genuinely capable of uncovering nuanced, strategically powerful gear optimizations that can even outpace player consensus.

Broader Community Reactions from Top Players

Beyond my own experiences, responses from top competitive players who've tested SplatGPT have been overwhelmingly positive, ranging from impressed surprise to outright disbelief at how effectively the model generates viable, strategic builds. These reactions underscore that the model isn't merely good at recreating known builds but can genuinely capture and even anticipate subtle strategic nuances valued by experienced competitive players.

Practical Impact and Community Deployment (and a Small Lament)

Perhaps most meaningful of all, SplatGPT wasn't merely an academic exercise, it was deployed publicly through the splatgpt.ink website and enjoyed early enthusiastic usage from the community. An API endpoint even collected user feedback, laying groundwork for future improvements via reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO).

Unfortunately, with Nintendo ratcheting up their security on their companion app one more degree, stat.ink suffered a dramatic dropoff in data collection. With the data well dried up, SplatGPT's recommendations inevitably lost their edge as the game's meta moved forward while the model remained frozen in time, stuck forever in October 2024. Understandably, traffic and enthusiasm declined accordingly. Alas, if you choose to build under the Sword of Damocles, you must accept the consequences.

Still, the successful deployment demonstrated SplatGPT's practical potential. The fact that the set completion engine proved so effective strongly suggests it could find similar success elsewhere... perhaps in other complex, noisy optimization domains beyond Splatoon.

For now, though, my focus shifts from deployment back to understanding: I've built a powerful gear-optimizing "brain," but how exactly does it think? In Part 3, I'll dive deep into the interpretability of SplatGPT, leveraging a Sparse Autoencoder (SAE) to map out the surprisingly intricate internal representations of the model.

Embracing Sets and Structured Prediction

The failure of naive recommendation systems isn't just about data scarcity or noise, it's a representation mismatch. A Splatoon gear build is not an ordered list, it is a set whose value comes from which abilities appear at which quantities and how they interact, not so much where they sit. Modeling the data as a set automatically satisfies our first requirement, permutation invariance.

The other two requirements, capturing non-linear interactions and understanding deep contextual relationships, are strengths of Transformer architectures like those found in Large Language Models (LLMs). However, standard LLMs are typically designed for ordered sequences: they emit token after token and score themselves on next-token accuracy. Even without positional embeddings, the causal mask inherently encodes an order bias.

So I asked: "What if I built an LLM that uses sets instead of sequences of tokens?" SplatGPT was my solution: fuse GPT-2's residual stack with the permutation invariance provided by using Set Transformers(Lee et al., 2019). However, this hybrid approach demanded a shift in the prediction task itself: the fused model no longer predicted the "next token", but should instead predict the entire collection of abilities likely to be within the set all at once. This naturally framed the problem as a multilabel classification task: for every possible "ability token" (more on this later), the model predicts the probability of that token being a member of the target set. This changes the training process, loss functions, data handling, and anything else downstream of adopting this set-based, multilabel paradigm.

Build Space and Token Space

Predicting probabilities for "ability tokens", however, buries the lede a little bit. How exactly do we define these tokens to accurately capture Splatoon's gear mechanics? A naive approach might be to create tokens for each ability and the type of slot it occupies (main or sub). But this runs into a significant issue: within the game's design, three "sub" slots (3 AP each, totaling 9 AP) have nearly the same internal point value as a single "main" slot (10 AP). Indeed, the community often sees these quantities as essentially equivalent for build purposes. Listing main vs subs would throw away this crucial fungibility of AP. Additionally, enforcing slot limits directly in the model would be a heavy ask adding needless complexity.

I solve this with two separate spaces:

Example: From Build to Tokens and Back

Consider the input build fed into the system:

• Player Input Build (Build Space):
  • Headgear Main: Ink Saver (Main) (10 AP)
  • Clothing Sub 1: Ink Saver (Main) (3 AP)
  • Clothing Sub 2: Quick Super Jump (3 AP)
  • (Other slots empty for simplicity)
• "Melting-down" to Token Space:
  • Ink Saver (Main): 13 AP (falls into the ISM_12_14_AP bucket, as an example)
  • Quick Super Jump: 3 AP (falls into the QSJ_3_5_AP bucket)
  • The model receives this as ["ISM_12_14_AP", "QSJ_3_5_AP"]

Let's say that SplatGPT returns the exact same combination for this example. We now have to reconstruct, which has two equivalent representations:

• "Reconstruction Step": Finding Valid Builds for the Predicted Tokens
  • Option 1 (Mirroring Original Structure):
    • 1 main slot to Ink Saver (Main) (10 AP)
    • 1 sub slot to Ink Saver (Main) (3 AP)
    • 1 sub slot to Quick Super Jump (3 AP)
    • Result: ISM = 13 AP, QSJ = 3 AP. Both targets hit.
  • Option 2 (Prioritizing Sub Slots for ISM):
    • 4 sub slots to Ink Saver (Main) (12 AP)
    • 1 sub slot to Quick Super Jump (3 AP)
    • Result: ISM = 12 AP, QSJ = 3 AP. Both targets hit.

Both of these options are valid ways to achieve the AP targets predicted by the model in Token Space. This becomes significantly more complex once you consider a full build, requiring the reconstruction algorithm to pick one based on its own heuristics or possibly offer multiple choices to the user. This example also demonstrates how Token Space abstracts away the main/sub concepts altogether and allows the model to focus entirely on total AP, while reconstruction handles the combinatorial arrangement of the final predictions.

Introducing SplatGPT

At its heart, SplatGPT processes a given weapon and (potentially incomplete) set of gear abilities, aiming to predict a complete and optimal set of ability tokens.

• Embedding Layer
  Discretised ability tokens and weapon IDs are embedded in the same vector space. We add the weapon embedding to every ability embedding (rather than concatenate), giving each token built-in context about the weapon and letting the model transfer synergies from popular to data-poor weapons. This is an important as abilities can either enhance a weapon's strengths or shore up its weaknesses, abilities can therefore only be understood within the context of the weapon they're supporting. This has a nice secondary effect that encourages the model to learn and transfer synergistic relationships observed with popular, data-rich weapons to less common, data-poor weapons (or equivalently, observe high amounts of signal from data-poor weapons and transfer it to noisy data-rich weapons).

• SetTransformer Layers (x N)
  The core of SplatGPT's power lies in the stack of SetTransformerLayer blocks. Each layer is a three-stage block designed to progressively deepen the model's understanding of the interactions within the set of abilities while preserving permutation invariance and sequence length.

  1. Set Transformer:
     The input stream is first processed by a Set Transformer module, which uses Induced Set Attention Blocks and Set Attention Blocks followed by a Pooling by Multihead Attention operation, internal attention mechanisms to analyze all-to-all interactions between the input tokens within the current set. The result is a condensed, fixed-size summary representation that essentially captures the essence of the input as a collective. This is foundational for permutation invariance.
  2. Cross-Attention
     One of the key challenges of fusing Set Transformers with a GPT-like architecture is that the pooling operation in Set Transformers requires a predetermined output sequence length. However, this interferes with the ability to have residual connections throughout the model which requires both input and output sequences to keep a fixed length.
     SplatGPT elegantly solves this by employing cross-attention. Here, the original input sequence for the SetTransformerLayer (before the Set Transformer module) acts as the query. The global summary vectors produced by the Set Transformer module serve as both the key and value. In essence, each token effectively "asks" the global summary what information from that summary is most relevant to it and updates itself based on that result. This allows the model to broadcast the rich, set-level summary back to each individual input token while maintaining the original sequence length.
     By reinjecting this global perspective, each input token becomes more aware of its role, dramatically enhancing the contextual understanding at each layer. Because attention mechanisms are inherently permutation equivariant, the later Masked-Mean layer achieves full permutation invariance.
  Interpretability aside

  I should note that it is difficult to understand what the later layers of SetTransformerLayer might represent, the first layer allows the raw input tokens to be aware of their relationship to all other input tokens. Perhaps the second layer allows these newly context-aware tokens to become aware of not only their own status but also the understanding the other tokens have of their goals. Characterizing past the first layer is quite challenging, and further interpretation work must be done to meaningfully characterize what the third layer (and possibly beyond) might represent.
  3. Feed-Forward Network (FFN)
     After the cross-attention step has enriched each input token in the stream with the global set context, each token in the sequence is passed through a standard Feed-Forward Network. This is a two-layer MLP: it first expands the dimensionality of the embedding, applies the GELU (Gaussian Error Linear Unit) activation function, and then contracts the dimensionality back. This allows nonlinear transformations to be applied to each token individually, and some research suggests this is where "knowledge" resides in a model.

  The entire three-stage block is wrapped in a residual connection: the output of this entire block is added back to its original input. These connections not only help train deep networks, but as Anthropic's research into transformer circuits demonstrates, it also allows for higher order contextual representations to flow through the model.

• Masked-Mean Pool
  Once the data flows through the entire transformer circuit, it's processed one last time by the Masked Mean Pooling layer. Its job is to aggregate the now highly refined and informed token embeddings into a single vector that holistically represents the entire (potentially partial) input gear set. It does this while respecting padding masks, which are used to handle sets of abilities of varying sizes within a batch, so that they do not influence the final average. This averaging process is inherently permutation-invariant and provides a final summary of the set's characteristics.

• Output Head
  This final vector is fed into the Output Head. This is a simple linear projection that transforms the vector's dimensionality to match the total number of unique tokens in SplatGPT's vocabulary. These resulting scores (logits) are then passed through a sigmoid activation function. This allows the model to assign an independent probability for each ability token in the vocabulary, deliberately framing the problem as multilabel classification. This is essential, as it allows the model to predict high probabilities for multiple different ability tokens simultaneously.

Together, these blocks let SplatGPT capture stacking curves, weapon synergy, and deep context in a single, permutation-invariant architecture.

This deep dive into SplatGPT's architecture reveals a model fundamentally built to understand Splatoon 3 gear in its native, set-based context. By splitting the problem into Token Space and Build Space, we have engineered a model designed for set completion with considerations for weighted multisets (sets where elements can appear multiple times and have different weights).

In part 2B, we'll walk through the dataset. How was it scraped, cleaned, how some biases were removed or compensated for while others were intentionally introduced. We'll go through the training regiment and discuss results, including the fascinating <NULL> token that convinced me the model is reasoning deeply rather than memorizing. Stay tuned!

Hidden Complexity in Splatoon’s Gear System

Nintendo's Splatoon series is one of contradictions. At first glance, its vibrant, cartoony aesthetic suggests simple, kid-friendly fun. Yet beneath that cheerful exterior lies remarkable depth and complexity across its interconnected gameplay systems. Nothing illustrates this better than the gear building system. On the surface, it functions as a form of self-expression, allowing players to build fun outfits for their character to wear during battle. Indeed, many players become quite attached to their uniquely customized avatars. However, hidden beneath this playful facade is a system of surprising strategic depth where choices can decisively shape gameplay outcomes. It is precisely this hidden complexity that makes it a fascinating and formidable challenge, especially for automated Machine Learning approaches attempting to discover the "perfect" build.

Splatoon’s Gear System Explained

Released in 2022, Splatoon 3 is Nintendo's wildly successful competitive team-based shooter, where players compete in fast-paced battles over territory and objectives. Before entering a match, players face a critical strategic choice that shapes their character's effectiveness in battle: their gear.

You can think of Splatoon's gear system as choosing your character's outfit for battle, composed of three distinct types: headgear, clothing, and shoes. Each piece of gear contains one high-impact primary (main) ability slot and three smaller secondary (sub) ability slots. Strategic depth arises from the existence of powerful, build-defining abilities restricted exclusively to the main slot of specific gear types. For example, the versatile Comeback ability, only available as a main ability on headgear, is strong enough to become the centerpiece around which a player's entire strategy revolves.

This elegant setup is intuitive enough for younger players to grasp easily while simultaneously introducing a deceptively intricate combinatorial puzzle, one challenging even for seasoned veterans. This poses a particularly fascinating yet formidable problem for optimization for both man and machine; a challenge so formidable that entire websites have been built just to help players advise each other on the most effective builds.

The Weapon-Gear Relationship

While gear provides abilities to adjust your character's performance, the foundation of every loadout is your weapon choice. This is typically the first (arguably the most impactful) decision a player makes, as the weapon defines fundamental aspects of gameplay, providing both powerful strengths that gear can enhance as well as notable weaknesses that gear can compensate for. Splatoon organizes its weapons within a clear hierarchy:

• Weapon Class (e.g. Shooter, Roller, Charger)
  • Defines the core movement mechanics, attack patterns, and overall gameplay archetype.
• Specific Weapon (e.g., Splattershot, Carbon Roller, E-liter 4K)
  • Dictates precise weapon statistics and handling characteristics, shaping the nuances of play.
• Weapon Kit (e.g., Splattershot vs. Tentatek Splattershot)
  • Includes a specific pairing of a secondary (sub) weapon, such as a bomb or utility, and a powerful special weapon, comparable to ultimates in other competitive games.
  • Shapes gear optimization even between variants of the same specific weapon.

This structured design both simplifies and complicates gear selection. On one hand, similar weapons kits often allow interchangeable gear builds, enabling players to experiment without too much investment. On the other hand, subtle differences between even closely related weapon kits might require entirely different gear setups.

The Sheer Size of the Search Space

Each piece of gear has one main slot and three sub slots. Headgear has four main-slot-only abilities tied to it, Clothing has five, and Shoes has three. This leads to a massive search space of 1.3 billion possible gear options (1,262,451,960 exactly). There's also 130 unique weapon kits, which balloons the figure up to over 160 billion possibilities (164,118,754,800 exactly). Given that the largest repository of data (stat.ink) has, as of May 2025, about 15 million recorded battles, each involving 8 players, this translates to a maximum representation of approximately 120 million gear instances. Even in the highly optimistic scenario where each instance represents a unique gear combination, this coverage is still three orders of magnitude smaller than the total search space. This is what makes the problem especially difficult for both players and algorithms alike.

The Intricacies of Ability Stacking

Much of the strategic depth arises directly from the ability stacking mechanics. Splatoon quantifies the investment in each ability using an internal point system, often colloquially referred to as Ability Points (AP) where the different slot types carry distinct weights:

• A main ability slot grants a hefty 10 AP towards that ability, offering a substantial boost.
• A sub ability slot grants a comparatively modest 3 AP, offering smaller, but cumulatively significant, gameplay effects.

Critically, each ability also follows a unique, non-linear effectiveness curve, typically characterized by diminishing returns as the total AP invested increases:

This non-linear scaling introduces key strategic consequences:

• Utility Subs: Some abilities provide a significant benefit with minimal investments (often just 3 AP), making them efficient and valuable additions to many builds. This specific phenomenon presents the biggest hurdle for traditional ML approaches: is a single sub-slot inclusion strategically optimal, or simply random noise? More on this later.
• Stacking-Dependent Abilities: Other abilities offer negligible benefit at low AP investments, becoming effective only when heavily stacked, making them powerful but niche options.

This interplay transforms gear building into a complex resource allocation puzzle, as the optimal choice for each ability depends on:

• The effectiveness of the ability at the current level of investment.
• The opportunity cost of investing AP elsewhere.
• Synergies with other abilities in the build.
• Compatibility with the equipped weapon.
• Individual playstyle considerations.

Like a snowball rolling downhill, these small but important pieces of the system compound, creating significant difficulty for both players and algorithms to properly optimize gear while taking everything into consideration.

Data Complexity and Noise

As if the system wasn't complex enough already, there's one final wrinkle making the optimization problem particularly difficult to solve: noise. While Splatoon 3 reduces randomness significantly compared to its predecessors, there remain four distinct layers of noise that complicate the interpretation of gathered data:

• Random Ability Assignment: Gear sub-slots are initially filled based on probability distributions tied to each gear's "brand". Since brand information is typically excluded from real-world datasets, baseline observations inherently contain unpredictable elements that are not easy to compensate for, if possible at all.
• Optimization Costs: Crafting a min-maxed gear set demands substantial time and in-game resources. Consequently, many observed loadouts may represent incomplete optimizations, practical compromises ("good enough" builds), or gear setups recycled across multiple weapons and evolving competitive landscapes (the broader competitive context known as the metagame), despite being suboptimal.
• Player Expertise Distribution: Gear data aggregates choices from players with widely varying game knowledge and priorities. Some choices represent deep, strategic understanding by experienced players; others reflect aesthetic preferences, outdated practices, limited knowledge of current community best practices, or experimental off-meta strategies by skilled players that may not prove viable.
• Multimodality and Meta Evolution: Many weapons support multiple viable builds, making it challenging to identify a singular "optimal" choice. Conversely, certain weapons have clearly defined niches with straightforward optimal builds and broad consensus. Additionally, continuous meta shifts mean previously optimal gear configurations can become outdated or suboptimal over time.

These factors combine to create substantial ambiguity in the data. Determining whether a single sub-slot ability (such as Quick Super Jump) is strategically optimal, randomly assigned, part of an incomplete setup, or intentionally suboptimal is a non-trivial challenge. Without careful consideration of context, what appears to be irrelevant "noise" could in fact be essential strategic signal, posing a formidable challenge for conventional machine learning approaches.

Why Conventional ML Models Struggle

These layers of noise and system complexity underscore a fundamental requirement for gear optimization: context. Interpreting any individual gear choice, like that lone Quick Super Jump sub-slot mentioned earlier, is impossible in isolation. Is it:

• An optimal "utility sub," carefully chosen to complement the specific weapon kit?
• A randomly assigned ability yet to be scrubbed or overwritten?
• Part of a gear piece primarily optimized for a different weapon or strategy?
• An intentional, albeit suboptimal, choice reflecting a niche strategy or personal preference?

The correct interpretation hinges entirely on interacting factors: the chosen weapon kit, the complete set of equipped abilities across the loadout, the prevailing competitive meta, and potentially unobservable factors like player intent or resource constraints.

This deep contextual dependency is exactly why conventional machine learning approaches struggle here. Standard techniques often identify noise through statistical deviations or treat features as somewhat independent variables. They might assume outliers represent errors to filter out, or that frequently occurring patterns inherently represent the primary signal. In the Splatoon gear system, however, these assumptions quickly break down. An apparent outlier (an uncommon ability choice) might be a highly effective niche strategy in the right context—crucial signal disguised as noise. Conversely, blindly copying the most common builds can lead to suboptimal setups if key contextual nuances are overlooked, especially when dealing with multimodality.

Models not explicitly designed to handle set-based inputs (where order doesn't matter, but combinations do), non-linear interactions, and deep contextual dependencies simultaneously, are fundamentally ill-equipped for this task. Simply averaging popular builds or filtering perceived noise, the approach many community resources currently take because it’s straightforward, cannot meaningfully optimize loadouts. Instead, we need an approach capable of capturing the holistic interplay between every element of a loadout within its specific gameplay environment.

Model Comparisons

This makes Set Transformers the most attractive option to address this problem, which we'll address in the next part.

The Road Ahead

We've journeyed through the intricate layers that make Splatoon 3 gear optimization such a deceptively difficult challenge. From the non-linear scaling of abilities and deep weapon dependencies to the ambiguous noise embedded in player data, it's clear that context is paramount. Conventional machine learning methods, often reliant on simpler assumptions about data independence or clearly separable signals, aren't designed to navigate this nuanced, combinatorial landscape effectively.

But complexity doesn't mean impossibility; it simply calls for a more specialized solution. In the next post (Part 2) of this series, I'll introduce SplatGPT, a deep learning model specifically built to handle these intricate challenges, leveraging architectures optimized for set-based inputs and deep contextual awareness.

Yet building the model is just the beginning. The deeper—and potentially more exciting—question is: how exactly does it make these optimization decisions? In Part 3, I'll dive into interpretability, showing how I've trained a monosemantic Sparse Autoencoder (SAE) on SplatGPT to uncover thousands of internal neurons that map neatly onto distinct strategic concepts. Given the scale of these neurons, I'm using doc2vec embeddings and automatic TF-IDF-based labeling to rapidly identify, label, and interpret these monosemantic features.

Finally, in Part 4, we'll push beyond passive interpretation into active collaboration. Using custom PyTorch hooks, I'll demonstrate how we can actively steer the gear completion process. Rather than passively accepting whatever optimal build the model generates, we'll be able to influence the direction it takes, nudging the optimization toward specific strategic goals: whether that's an aggressive push, a more defensive playstyle, or even entirely novel, niche configurations. By visualizing builds before optimization, after standard optimization, and following targeted steering, we'll unlock an entirely new dimension of control, turning SplatGPT into an interactive partner in strategic gear-building.

Join me as we move from clearly defining the complexity of gear optimization, to understanding the intricacies of an advanced AI solution, and finally, actively collaborating with that solution to build gear tailored precisely to specific strategic intentions in the rich, nuanced world of Splatoon 3.

At first glance, optimizing gear loadouts in competitive games seems straightforward: identify the best combinations of abilities for each situation. However, when these systems allow for complex stacking mechanics and context-dependent effectiveness, the problem becomes remarkably challenging for traditional machine learning approaches. Nintendo’s Splatoon 3 presents an especially fascinating case study, where the interaction between gear abilities, weapon choice, and player strategy creates an optimization problem that reveals fundamental limitations in how we typically approach set-based prediction tasks.

The Deceptive Complexity of Loadout Systems

Imagine trying to build a recommender system for the optimal gear setup for a player. A naïve approach may suggest looking at what high-level players use and replicating their choices. Indeed, this is what multiple websites have attempted to do. However, this runs into problems. How do you account for multimodality? Can you still build proper recommendations for highly unpopular weapons? How do we distinguish between intentional choices and noise?

The Splatoon Gear System

Splatoon 3 is Nintendo’s wildly successful 2022 team-based competitive shooter where players battle as squid-kids. Before even entering a match, players face a strategic decision that affects how they play: they must select a gear loadout that complements their weapon choice and personal playstyle.

You can think of Splatoon’s gear system as the outfit your character is going to wear, and it consists of three distinct types: headgear, clothing, and shoes. Each piece of gear contains one primary (main) ability slot, three secondary (sub) ability slots, and a set of four unique main-slot-only abilities specific to that gear type. For example, the ability Comeback is only found on headgear.

Weapons and Gear

Understanding weapon mechanics is crucial for gear optimization because different weapons demand different ability configurations. Weapons as they’re set up in Splatoon 3 follow a hierarchical structure:

• Weapon Class (e.g. Shooter, Roller, Charger)
  • Defines basic movement, attack patterns, and controls; archetype-level
• Specific Weapon (e.g. Splattershot, Carbon Roller, E-liter 4K)
  • Determines precise statistics and handling
• Weapon Kit (e.g. Splattershot vs Tentatek Splattershot)
  • Provides specific sub-weapon and special weapon combination
  • Influences optimal gear choices even with the same specific weapon

Ability Stacking

The complexity of gear optimization emerges from the ability stacking mechanics:

• Standard abilities can be placed in any slot
  • Primary slot abilities have a weight of 10
  • Secondary slot abilities have a weight of 3
• Each ability follows a unique, non-linear effectiveness curve:
  • Some abilities provide significant benefits from the minimum possible investment ("utility subs")
  • Others require substantial investment to reach meaningful effectiveness.

This creates an optimization problem where the value of each ability slot depends on:

• The ability itself
• Total investment in that ability across all gear
• Synergies with other equipped abilities
• Synergies with selected weapon
• Playstyle considerations

Data Challenges

As yet another wrinkle, the gear system in Splatoon 3 introduces several layers of noise into any data gathered:

• Random Ability Assignment: Secondary slots are initially filled randomly based on the gear's brand. Each brand has its own probability distribution as to how those secondary slots are filled.

• Optimizing is Expensive: While players are able to change abilities of their piece of gear, it requires significant time investment. As a result, observations may be an incomplete optimization. Additionally, players will reuse gear optimal for one weapon across multiple weapon setups.

• Player Expertise Distribution: Not all players fully understand the complex mechanics behind ability stacking. Some choices might reflect a poor understanding of the game rather than a well-understood strategic decision. Some players might simply value aesthetics, regardless of the abilities.

• Build Multimodality: Many weapons have multiple viable optimal builds, while others have a single niche. Meta shifts can also change a weapon's niche, invalidating previously optimal gear configurations.

More Context Needed

What makes this system particularly challenging for traditional machine learning approaches is that noise in the data cannot be identified without understanding the full context. A single Quick Super Jump ability in a secondary slot might be:

• An optimal choice for the current weapon
• A random ability from initial brand probabilities that hasn't been changed
• Part of optimal gear for one weapon, used for another weapon

The interpretation depends entirely on:

• The weapon being used
• Other abilities present in the build
• The current competitive meta

This contextual dependency means we can't simply treat unexpected configurations as noise to be filtered out – they might represent valid strategic choices that only make sense when considering the full picture. Traditional approaches to handling noise rely on the assumption that noise can be identified through statistical patterns or outlier detection. However, in the Splatoon 3 gear system, what looks like noise in one context may, in fact, be raw signal in another.

Requirements for High Quality Recommendations

Given all of these interacting complexities that compound upon each other, any model attempting to achieve a high performance on gear prediction must simultaneously satisfy multiple requirements:

1. Set Processing

• Must handle unordered collections of abilities
• Needs to maintain identical predictions regardless of ability slot ordering
• Should process all gear pieces together as a complete loadout

2. Nonlinear Interactions

• Must capture complex stacking effects between abilities
• Should implicitly understand ability breakpoints and effectiveness curves
• Needs to model synergies between different abilities

3. Contextual Understanding

• Must process weapon information alongside gear choices
• Should recognize valid variants in builds for each weapon
• Needs to distinguish between intentional choices and noise based on context

4. Multimodal Predictions

• Must be capable of suggesting multiple valid configurations
• Should avoid collapsing to a single recommendation regardless of input

These requirements suggest the need for an architecture that can process sets while maintaining awareness of the relationship between all elements and their context. Traditional sequence models and standard neural networks are fundamentally ill-suited for this task. Given these stringent requirements, a novel approach is needed. One that can handle the set-based nature of the problem while capturing intricate interactions.

Attention

To tackle the complexities of Splatoon's gear optimization, I drew inspiration from the attention mechanisms in transformer models. Attention inherently provides a way to incorporate context by allowing each element in a sequence to interact with every other element. Crucially, when positional embeddings are omitted, attention becomes permutation invariant—making it ideal for processing sets where order doesn't matter.

This property aligns perfectly with our challenge. In the gear system, the effectiveness of an ability depends not just on its presence but also on its interaction with other abilities and the chosen weapon. By using attention mechanisms, we can model these interactions holistically, ensuring that each ability's impact is considered in the context of the entire set.

A Two-Stage Approach

Realizing that attention mechanisms could address our needs, I embraced the set-based nature of the problem and modeled it as a multilabel classification task. Here's how:

1. Discretization and Tokenization: Standard abilities are discretized into buckets based on their stacking weights and effectiveness, transforming continuous ability values into categorical tokens. Main-slot-only abilities are straightforward to tokenize because they are tied to a single gear type and cannot be stacked.

• However, discretization introduces a loss of exact information. To construct a valid gear set that adheres to game constraints (like slot limitations, mutual exclusivity between pairs of abilities, etc.), we need a method to translate these probabilities into a feasible loadout.

2. Probability Prediction: A deep learning model predicts the probability of each discretized ability token as being part of the optimal gear set. By treating these tokens as labels, we can handle multiple abilities simultaneously, reflecting the multilabel nature of gear configurations.

3. Gear Set Construction: Turning the probabilities into a gear build has multiple viable approaches. Instead of relying solely on the model's initial output, we employ a deterministic algorithm; specifically, beam search. This algorithm iteratively builds gear sets by selecting combinations of abilities that maximize the overall predicted probability while satisfying all constraints. It effectively bridges the gap between probabilistic predictions and practical applications. While the specifics of this algorithm are beyond the scope of this post, it serves as a crucial bridge between probabilistic predictions and practical gear set construction.

By splitting the problem into these two components, we leverage the strengths of deep learning in modeling complex, context-dependent interactions and use algorithmic strategies to ensure valid, optimized outputs.

Introducing SplatGPT

By integrating these strategies, I developed SplatGPT, a model tailored for the Splatoon gear optimization challenge. SplatGPT enhances the Set Transformer architecture with elements inspired by GPT-2, specifically designed to handle the unique aspects of this problem.

• Embedding Layer: The model starts by converting discretized ability tokens and weapon IDs into embeddings. By adding the weapon embeddings to the ability embeddings, we provide contextual information that allows the model to understand how weapon choices influence ability effectiveness while retaining the ability for the model to transfer synergies from more popular weapons to less popular weapons.

• SetTransformer Layers: SplatGPT employs multiple layers of what I call the SetTransformerLayer. Each layer processes the input set through a Set Transformer module which, due to its design, maintains permutation invariance. This ensures the model's predictions are based solely on the combination of ability tokens and not their order.

  • Cross-Attention Mechanism: Within each layer, the output of the Set Transformer serves as the key and value in a cross-attention mechanism, with the original input as the query. Using the original input as the query allows the model to reference the initial state of the abilities, ensuring that the refined understanding doesn't drift from the original data while retaining the input sequence length. This setup allows the model to refine its understanding by focusing on relevant interactions between abilities.
  • Feedforward: Each SetTransformerLayer ends with a feedforward layer that increases dimensionality, passes the input through an activation function (in this case GeLU, or Gaussian Error Linear Unit, which helps model complex patterns), and returns dimensionality to what it was before.
  • Residual Connections: Through the use of residual connections, the layers build higher-order relationships that capture complex interactions between abilities and weapons. These connections enable what's sometimes referred to as "virtual attention," enhancing the model's capacity to learn intricate patterns.

• Masked Mean Layer: To aggregate information across the set and ensure permutation invariance, the inputs pass through a masked mean pooling layer. This mechanism averages the embeddings while accounting for variable input lengths, effectively summarizing the set without imposing any order.

• Output Layer: The final output first is projected to the size of the dictionary before it passes through a sigmoid activation function, generating probabilities for each ability token. This framing as a multilabel classification problem enables the model to predict multiple abilities simultaneously, capturing the multimodal nature of optimal gear configurations.

SplatGPT effectively models the intricate, nonlinear relationships within the gear system, accounting for ability stacking, weapon synergy, and contextual dependencies. By integrating attention mechanisms and set-based processing, it overcomes the limitations of traditional models, providing accurate and practical gear recommendations.

Cross-Attention Mechanism in SplatGPT

The key breakthrough that allowed me to create SplatGPT by putting in a Set Transformer into a GPT-2-like architecture was cross-attention. Here's a more detailed explanation:

Maintaining Sequence Length

The Set Transformer module inherently includes a pooling operation that reduces the sequence length by summarizing the input into a condensed representation. While this is beneficial for capturing global set information with permutational invariance, it poses a challenge when integrating to the residual stream, as the sequence length must remain consistent across every connection back into the stream in order to work effectively.

To address this, I introduced a cross-attention mechanism that maintains the original sequence length. Here's how it works:

• Cross-Attention Setup: The output of the Set Transformer (the pooled, set-level representation) serves as the key and value, while the original input sequence acts as the query in the cross-attention mechanism.

• Functionality: This setup allows each position in the original sequence to attend to the set-level context captured by the Set Transformer. Essentially, it broadcasts the set-level information back to each individual token while preserving the sequence length.

Adding Context

This process effectively distributes the aggregated set-level information back to each individual token, enriching their representation. In essence, each token now:

• Holds Context: Each token incorporates some information about its role within the entire gear set and its interaction with other abilities in the build.

• Captures Complex Relationships: With each additional SetTransformerLayer, the model refines these representations, enabling it to model increasingly complex interactions between abilities.

• Benefits from Deeper Layers: The stacking of layers allows the model to build upon previous representations. Since each SetTransformerLayer is part of the residual stream, the higher-order relationships that can be captured as a result of preserving sequence length are much more than the sum of their parts. For more information on this, check out A Mathematical Framework for Transformer Circuits by Anthropic.

Why It Matters

• Context-Dependent Abilities: The effectiveness of an ability often depends on other abilities equipped and the chosen weapon.

• Nonlinear Synergies: Certain ability clusters behave differently based on the surrounding context. While Comeback and Quick Respawn with 15 Ability Points (AP) have high amounts of synergy, if the input tokens include Quick Respawn with 40 Ability Points, the amount of synergy between Comeback and Quick Respawn 15 is much lower.

• Comprehensive Modeling: By enriching each token with the set-level context, the model can make more informed predictions about which abilities will optimize performance.

Why Not Self-Attention?

As mentioned above, attention is set-based in nature and needs to be provided positional embeddings for sequence-to-sequence tasks. So why not use self-attention? Why use a Set Transformer at all with Cross-Attention?

The primary reason is because self-attention works as pairwise relationships. Though the residual stream allows significantly more complex interactions over multiple layers of self-attention, we are still dealing with sets. That is, we have to deal with \( 2^n \) possibilities where \( n \) is the number of tokens in our vocabulary. We’d need many layers of self-attention to replicate what we can get with a few layers of cross-attention with the set transformer output.

Results

Evaluation

Evaluating a gear optimization model for Splatoon 3 presents a unique challenge. The true test of a build’s effectiveness can only be measured across hundreds of games with multiple players – a scale of testing that proves infeasible given Nintendo’s locked-down API and the demographic challenges of convincing younger players to systematically track their games.

Where are the Standard Metrics?

Traditional evaluation metrics would require large-scale user testing under controlled conditions. Since there is no direct efficacy score that we can extract from just the gear that the model produces, we would need to rely on players testing these out and aggregating results over dozens of matches per build in order to properly evaluate. Unfortunately, there are a few issues here. First, Splatoon 3's closed API and Nintendo's strict data policies make gathering such data nearly impossible. Second, coordinating players to pull this off would itself be a herculean logistical feat. Finally, meta shifts can quickly invalidate any static benchmark. As a result, we have to rely on qualitative feedback from top competitive players. While not the usual setup, the anecdotal evidence so far has been quite promising and has revealed both strengths and limitations of the model.

Initial Validation

Initial validation came from my own experience as a player ranked in the top 10,000 globally. Once the model began consistently surpassing my build optimization abilities, I sought more rigorous validation from top competitive players. These experts were asked to select random weapons they felt comfortable with and provide partial builds with varying levels of completion – ranging from nearly complete setups to just weapon selection. This tested the model’s ability to both complete existing builds and construct entirely new ones from minimal input.

Surprising Results

The response from top players was unanimously positive, with reactions ranging from impressed surprise to outright disbelief. The most compelling example came when the model perfectly reconstructed a top player’s build for an extremely unpopular weapon, working from just a half-completed build – and did so without having seen that specific configuration or similar builds in its training data. This demonstrated not just pattern matching, but a genuine understanding of gear optimization principles.

However, the model isn’t without limitations. Weapons with minimal representation in the dataset, particularly when combined with unpopular abilities, can produce suboptimal results. While this seems unavoidable given the current dataset constraints, it’s a clear area for future improvement.

Production Deployment

Perhaps the most meaningful validation is that this isn’t just a proof of concept – the model is already in production, actively helping players optimize their gear loadouts. It demonstrates that achieving superhuman performance isn’t necessary to provide real value to the community. Like a solid NBA bench player, it consistently performs at a professional level while maintaining room for growth. Additionally, an API endpoint was created to take player feedback on the autocompleted build. This will be used with RLHF or DPO to improve future builds.

Training Dataset

The primary dataset was sourced from stat.ink, a community platform where players upload their match data using third-party tools. While the platform provides anonymized data with identifiable information stripped, this presents potential sampling biases from the site's users being overrepresented. Several preprocessing steps were necessary to ensure data quality and manage potential biases.

Player-Level Bias Correction

• Limited entries to 50 per likely-player through a discriminator (not an identifier) created via side-channel analysis
• While imperfect, this is the best available compensation for sampling bias within the dataset constraints

Performance-Based Filtering

• Enforced a minimum 3:2 win-loss ratio on a per-weapon basis
• This intentionally introduces a bias against losing configurations
• Less effective builds or less skilled players are more likely to produce losses

Temporal Filtering

• Post-Update Periods:
  • Dropped two weeks of data after new weapon introductions
  • Dropped one week of data after balance patches
  • Players experiment heavily during these periods, leading to unstable data quality
• Historical Data:
  • Progressively dropped older data
  • Accounts for meta shifts and evolution of optimal strategies
  • Helps ensure model reflects current game state

Completeness

• Dropped all incomplete builds
• Ensures model trains on complete, valid configurations

Impact

The preprocessing decisions were carefully chosen using domain expertise to balance between availability and quality. The introduction of biases might seem counter-intuitive but serves to help the model produce optimal builds rather than replicating the dataset.

Training Process

Initial Success

The training process was surprisingly smooth. Even early runs produced builds that, while far from optimal, demonstrated a logical consistency that an experienced player could understand. This early success suggested that the architecture was well-suited for this problem space.

Data Preparation

The training approach drew inspiration from large language model techniques, specifically the sliding window method for next token prediction. Each build generated multiple training examples through progressive token dropping:

• Before training, select \( n \) for how many times each build will be used in the dataset.
• Generate a list of random numbers from 1 to \( k \) where \( k \) represents the sequence length, these will represent the number of ability tokens dropped for each training example.
• Initial attempts dropped largest ability buckets first
  • Later switched to random drops, which improved learning of cross-bucket ability associations
• Padding tokens added to maintain consistent sequence length within batches
• Binary cross-entropy with logits selected as loss function
• Learning rate guided by validation set F-score after each epoch.

The development process utilized two distinct training scenarios:

• "Toy" models: Single RTX 2080Ti, 10-to-20-minute training runs for rapid iteration
• Production model: Single H100 GPU, 62-hour training period. While the training time could be optimized further, this was sufficient for production-quality results.

Emergent Behavior: The NULL Token

Possibly the most fascinating discovery emerged halfway through development with the introduction of a <NULL> token. Early experiments with <NULL> revealed two extremes:

• Including <NULL> in all training data degraded overall build quality across the board
• Sparse inclusion of <NULL> tokens led to overfitting on builds lucky enough to get <NULL>
• Anything in-between would lessen overfitting but also decrease overall build quality.

I made a completely baseless hypothesis: completely excluding the <NULL> token from training but including it as part of the vocabulary might make the model treat it as a root token for generating an optimal build given no input. This proved correct, with <NULL> consistently producing highly-effective weapon-specific builds – an emergent behavior that it was not explicitly trained for.

This unexpected success with the <NULL> token highlights that the model is actually developing optimization strategies and not simply regurgitating its training dataset. This is quite fascinating, as in a way the model is attempting to "reason" about what it should do given an unknown input.

Insights

To better understand how SplatGPT makes decisions, I developed a utility that applies the logit lens to each layer of the model. The process involves progressively truncating SetTransformerLayers until only the embeddings flow into the masked mean, followed by projection and activation. I then analyzed the results across a comprehensive set of inputs: every ability token paired with each weapon, at each layer depth. By tracking how probability distributions change between layers, we can effectively peek at the model’s “thought process” as it incorporates more contextual data and refines gear recommendations. See the appendix for excessive details on this.

Through this careful investigation using the logit lens, we can observe several patterns in how different types of ability tokens are processed:

• Role-defining abilities tend to show accurate predictions from early layers. Their inclusion in the input will shift the output distribution towards generally viable builds immediately.
• Multimodal abilities hedge significantly, with input probabilities clustered tightly around \( 0.5 \). As more context is added and it goes through more layers, these abilities will either have higher probabilities in the output distributions or go towards zero.
• Ubiquitous abilities that require contextual understanding show a mesa-shaped distribution, with most values between \( 0.4 \) and \( 0.6 \). This requires significantly more context, and the mesa-shape represents their high appearance rate in the training data.
• <NULL> Token exhibits weapon-specific distribution patterns and undergoes more significant change in the first two layers compared to other tokens.

Conclusion

SplatGPT demonstrates that deep learning models based on LLM architectures can solve complex set-based optimization problems while capturing intricate domain constraints. Through careful architecture decisions, I created a model capable of understanding and leveraging the complex interactions between weapons, abilities, and playstyles in Splatoon 3.

The most fascinating aspects of this project were not the solutions we explicitly engineered but rather the behaviors that emerged. The <NULL> token’s ability to generate strong baseline builds despite never being trained on them suggests the model developed genuine optimization capabilities rather than simply memorizing common configurations. The fact that the model has reached such a high level of performance despite being trained on such a wide skillset is also very impressive and goes to show how we can extract the wisdom of the crowds through careful, thoughtful data preprocessing. Additionally, the logit lens analysis reveals how the model progressively refines its understanding of ability interactions across layers, with different patterns for role-defining, multimodal, and ubiquitous abilities.

While SplatGPT has proven effective enough to deploy in production, there’s still much to explore. Future work includes using sparse autoencoders to better understand the features being detected at each layer, expanding the training dataset to better handle rare weapon-ability combinations, and potentially applying similar architectures to optimization problems in other games. The model’s deployment was also equipped with a feedback endpoint tied to the API’s request id. This allows users to give both positive or negative feedback on builds. This is specifically to set up for the next stage of model improvement: having a dataset of judged model outputs in order to apply either RLHF or DPO. This should take the model from the current mid-level competitive player skill tier it is at and take it to top competitive.

The immediate success of the model suggests it is especially good at dealing with set completion tasks with a lot of contextual noise. While I can’t think of many processes that this would apply to outside of Splatoon 3, the existence of this solution will hopefully inspire other solutions the way LLMs and Set Transformers inspired me.

Appendix

Logit Lens Analysis

The figure below illustrates the distributions for nine different weapon/ability token input combinations. The first six are for the weapon Splatana Stamper as a case study for how the distribution is affected by the token selection as it is one of the weapons that is multimodal in nature. The last three are specifically chosen weapon/ability token pairs that illustrate some interesting quirks.

Shown above are a selection of nine logit lens distributions at different points of analysis, colored by their position on the visible spectrum:

• Input Layer -> Red
• Layer 0 -> Orange
• Layer 1 -> Green
• Layer 2 -> Blue

The first two rows are for the Splatana Stamper, a multimodal weapon with multiple viable builds. The selection of abilities to start with are from left to right then top to bottom: no ability, the single most commonplace ability, a highly commonplace ability for builds for this weapon, the two defining tokens for the different builds, and a minimal investment case that is signal here but is noise in almost any other context. The final row represents three selected weapons all with a representative of their builds. Below is a supplementary graph which has the differences between layers.

What we can tell from these differences between layers is that most of the feature selection is done by the first layer, with iterative refinements happening with each layer. What each feature actually is is not something I have looked into yet, though I will look into it via sparse autoencoders as detailed in this Anthropic paper