Synthetic Data Generation for Training and Evaluation
For a fraud-detection classification task, the rare-but-critical cases — sophisticated fraud attempts, as opposed to obvious ones — were exactly the ones we had the fewest real labeled examples of, which is the usual problem with rare-event data: the cases that matter most for evaluation are the hardest to collect enough of. We turned to LLM-generated synthetic examples to fill out that category for both fine-tuning and evaluation.
The first version of our synthetic generation prompt produced examples that were too easy — the model generated fraud patterns that were internally consistent with what it already understood as 'fraud-like,' which meant the synthetic examples were more detectable than real sophisticated attempts. Training and evaluating against them gave us a false sense of security; the resulting classifier scored well on synthetic hard cases and then missed real ones in production that didn't match the synthetic pattern.
We fixed this by seeding generation with real (anonymized) structural patterns from actual historical fraud cases and asking the model to vary surface details — merchant names, amounts, timing — while preserving the underlying pattern that made the original case hard to catch, rather than asking it to invent fraud patterns from general knowledge. This kept the synthetic data grounded in patterns we knew were realistic rather than patterns that were merely plausible-sounding.
We also built an explicit check into our pipeline comparing the distribution of synthetic examples against the real (small) sample of rare cases we did have, on features like transaction amount distribution and timing patterns, and rejected synthetic batches that drifted too far from the real distribution. Two early batches failed this check and would have subtly skewed training if we'd used them unchecked.
Synthetic data ended up being genuinely useful, but only as an augmentation on top of a real, if small, seed set — never as a full replacement for real examples, and never trusted without validating its distribution against whatever real data existed. The failure mode to watch for isn't synthetic data being obviously wrong; it's synthetic data being subtly, plausibly wrong in a way that teaches the model (and your evaluation) the wrong lesson with high confidence.
