Optimizing Fraud Detection Systems: Leveraging Machine Learning and Experiment Automation

Our goal was to improve the detection of fraudulent patterns specific to individual businesses, products, or countries, while minimizing false positives for legitimate payments. Despite a training data volume of approximately 800 million transactions, our fraud prevention model was found to use only a fraction of the available history. Most payments without reported fraud were deliberately excluded to maintain training efficiency.

We hypothesized that more comprehensive historical data would help the model better distinguish fraud from unusual but legitimate activities. However, training on a larger data volume posed a challenge for both our infrastructure and model development. A single pass through the data could take up to a day. Moreover, the latest payments, from which we needed to learn, often suffered from incomplete fraud reports, making it difficult to assess model progress. Addressing these issues required more intensive experimentation, with each experiment demanding significant effort from machine learning engineers in data preparation, training monitoring, and bug fixing.

The intention to learn from a broader network of transactions necessitated faster training and a deeper understanding of delayed fraud reports. We developed tools that allow automated agents to investigate overlooked fraud and conduct experiments. These agents used model explanations and results from earlier investigations to test changes. By scaling training approaches and automating the experimental cycle, we were able to improve our model's success rate by more than 8%.

For each weekly batch of payment data, our training system retained all known fraudulent payments and randomly sampled a subset of the remaining ones. This subsampling accelerated regular retraining. Training on a larger portion of the network would provide the model with more examples of less common activities and more data for learning representations of entities such as payment cards or merchants.

In the first experiment, we eliminated subsampling and included all relevant payments from a nine-month period. This expanded the training set from approximately 800 million to 11 billion transactions. The results were encouraging: after the first pass through the data, the model was already approaching the performance of the existing solution, and after the third pass, it surpassed it in initial evaluation.

Each pass, or epoch, took approximately 24 hours. Our goal was to run several experiments per week and aim for daily testing of new ideas. At this rate, even a three-epoch comparison required roughly three days of training. Testing a new feature or configuration meant waiting days for a result, which significantly slowed down the exploration of new ideas.

We optimized processes by shortening each epoch from 24 hours to 4.8 hours (an 80% reduction), while maintaining the dataset of 11 billion examples. Keeping GPUs fully utilized required harmonizing data loading, computation, and communication. Accelerating one component invariably exposed the limitations of others.

Our training system downloaded data from object storage to disk, from where data loaders then read it. At this scale, it was not feasible to cache the dataset in memory, so disk throughput limited the speed of data delivery to the GPUs. We modified the data loaders to stream data directly from storage into memory, cutting the epoch time in half to 12 hours.

Tensor operations in the model's training step were compiled using modern optimization tools, and the compiler configuration was also fine-tuned. We also used lower-precision arithmetic to speed up computations. The model's architecture includes parallel branches; their concurrent execution, along with improved transfers between CPU and GPU, reduced the time to seven hours.

Memory utilization limited the amount of data we could prepare ahead of the GPUs. Reducing peak training memory usage from 1.3 TB to 500 GB allowed us to increase the number of data loading workers per GPU and prefetch more data. Compressing gradients to BF16 format during inter-GPU communication, optimizing communication block size, and using asynchronous transfers from CPU to GPU ultimately reduced the epoch time to 4.8 hours.

The larger dataset also led to a wider embedding layer. The model grew to approximately 30 million parameters, three times its previous size, without increasing latency when used in load tests or production. We could now train at the desired scale while still making fast decisions to meet our latency requirements.

Unfortunately, the larger model initially appeared better, but this result changed as more fraud reports arrived.

The new model appeared superior in offline tests and early production experiments. However, with the arrival of additional fraud reports, its measured performance dropped below that of the previous model, forcing us to revert to the model trained with the original sampling strategy.

For a payment made with a stolen card, it can take weeks for the cardholder to notice the fraud and report it to their bank. Once the platform receives a report of a fraudulent transaction, we obtain a 'fraud' label for a payment we evaluated weeks earlier. Labels in our training datasets were assigned based on available fraud reports at that moment. The most recent weeks of training data therefore contained the least mature labels, as insufficient time had passed for all reports to arrive.

We had to find a balance between the speed at which the model learns new fraudulent patterns and the reliability of these labels. Recent payments helped the model keep pace with attackers, but including too many meant training on fraud that was incorrectly labeled as legitimate. Waiting for reports provided us with more reliable labels, but at the cost of learning new behaviors later.

The flow of fraud reports is gradual. Payments that appear legitimate in a recent training or evaluation dataset may only receive 'fraud' labels weeks later. The set of payments we subsampled contained both legitimate payments and fraud that had not yet been reported. Retaining only a fraction of this set also kept most mislabeled fraud out of training.

When we tried to remove subsampling, we added more legitimate examples, as intended, but also roughly 20 times more fraudulent examples incorrectly labeled as legitimate. The number of known fraudulent examples remained the same, as we were already retaining all of them. The model subsequently saw many more examples that taught it that fraudulently reported patterns, even with a delay, were legitimate, especially in the most recent weeks of training. It became worse at detecting fraud that would only be reported weeks later. Understanding such failures required repeated experimentation with data and model training. Faster training facilitated these experiments, but each still demanded significant engineering effort from hypothesis to result.

Each experiment followed the familiar cycle in machine learning engineering: investigate failures, formulate a hypothesis, implement a change, and evaluate it. Preparing training data, monitoring jobs, and debugging errors consumed time in every experiment.

We developed a system called Agentic MLE (short for Machine Learning Engineer) to take over a larger portion of this work. It is a set of tools and instructions that enable AI agents to operate within the existing machine learning infrastructure and conduct experiments from idea to outcome: exploring data, implementing features, computing their historical values, training a model, and evaluating it against a baseline.

Agents work across the entire experiment lifecycle. Engineers review hypotheses and results and decide what to deploy. Feature experiments are built on our feature platform, which computes historical training features and serves their online equivalents from the same definitions.

It was necessary to make operational knowledge accessible to agents. They need to have the correct datasets, entity keys, timestamps, and sampling rules for feature experiments, be able to reproduce knowledge at the time of payments for historical feature backfilling, and diagnose failed distributed training jobs before restarting them. We organized this knowledge into reusable procedures for the platform and model-specific documentation that the agent loads for a given task.

The Agentic MLE system uses tools for querying data, inspecting workflow execution, and reading experiment results. It writes code for a candidate feature and launches the jobs necessary to test it. Engineers review the proposed experiment and its results at defined checkpoints. During execution, it checks for failures, correct configurations, and retries failures, much like a human would.

We deployed these tools to address a fraud attack where the model incorrectly assigned low-risk scores to fraudulent payments. We provided agents with attack data and results from previous experiments and investigations into model behavior, including our scaling work. The agent used model explanations to identify which features influenced scores up or down for overlooked fraud, giving us testable hypotheses. We could remove features that made transactions appear safe, or more explicitly define combinations of suspicious activities. Human experts helped prioritize 38 ideas, after which agents implemented changes, trained models, debugged failed runs, and evaluated the results.

Several agents proposed experiments with Stochastic Weight Averaging (SWA), which averages snapshots of parameters from a training run into a single model. The model already used SWA, but the averaging only started in the second half of the run. Training loss was decreasing while validation loss was increasing, giving agents reason to test whether including earlier snapshots could reduce overfitting.

Early stopping of training improved overall ranking but reduced recall in our narrow range of false positives. The best results combined averaging from epoch 0 and shortening the training run to eight epochs.

In a controlled comparison, the new configuration improved recall by more than 8%, which translates to an estimated tens of millions of dollars annually in additional fraud prevention for merchants.

We learn from every experiment, especially those that do not lead to production deployment, and we use these insights to improve our models and systems. Infrastructure improvements that accelerated our training allowed us to quickly test a wide range of hypotheses through the Agentic MLE system. Our goal is for agents to conduct longer investigations between checkpoints, while engineers prioritize, evaluate evidence, and decide what gets released.