No. 07of 22LandmarkResearch
AI-text detection, robustness testing and plagiarism analysis
Textegrity
A content-integrity pipeline: an ensemble of eight detectors for AI-written text, a back-translation stage that tests how easily detection breaks, and a plagiarism check. Its own evaluation shows 97.86% accuracy on its training distribution and 55.7% on text from newer models.
- When
- Feb – May 2026
- Status
- Built
- Team
- RV University course project: a four-member team with a faculty guide.
- Role
- All 25 commits in the project repository are mine, and the April training runs were on my machine. The project documents list me under AI detection.
Progress
Progress70%
All three features run end to end from the command line and the API. The web front end was never deployed, and detection of text from newer models is weak.
What comes next
- Build a clean held-out test set from different sources and report the real out-of-distribution number.
- Correct the draft paper before it goes anywhere.
- Fix the failing API tests and deploy a working detection demo.
97.86%
accuracy of the four-model meta-classifier on 2,099 held-out rows
In-distribution only: the held-out rows come from the same RAID and HC3 pool as the training data, and there is no clean held-out split. Read it together with the next figure.
Training run output (step1_progress/STEP1_COMPLETE)Verified. Backed by a file or public page that has been checked.Training run output (step1_progress/STEP1_COMPLETE) — Read in the project audit of 13 November 2026.55.7%
accuracy on 140 texts from newer language models
78 of 140 right on a balanced set: 37 of 70 AI texts and 41 of 70 human texts. Chance is 50%.
The project's benchmark results file (benchmark_results.csv)Verified. Backed by a file or public page that has been checked.The project's benchmark results file (benchmark_results.csv) — Read in the project audit of 13 November 2026.97.25%
accuracy of the best fine-tuned detector alone, DeBERTa-v3-large, on 6,994 samples
Same data pool. Two of the four fine-tuned detectors are weak (Longformer 64.36%, XLM-RoBERTa 36.75%), and the best released detector scores 96.67% on the same set.
Training log and metrics file (step1_output.log, REAL_METRICS.json), recomputed from saved scoresVerified. Backed by a file or public page that has been checked.Training log and metrics file (step1_output.log, REAL_METRICS.json), recomputed from saved scores — Read in the project audit of 13 November 2026.62.0% → 27.9%
average detector score before and after back-translation
Three test texts. One barely moved, from 99.7% to 82.6%.
The project's benchmark notes (BENCHMARK_RESULTS.md); the per-text row is from the project paper, table IVVerified. Backed by a file or public page that has been checked.The project's benchmark notes (BENCHMARK_RESULTS.md); the per-text row is from the project paper, table IV — The average was read in the project audit of 13 November 2026.8
detectors in the ensemble: four fine-tuned by the team, four used as released
This counts detectors. Counting every transformer the team fine-tuned across detection, plagiarism and rewriting gives seven, plus a partly trained PEGASUS; several of them are weak.
Project presentation and training logsVerified. Backed by a file or public page that has been checked.Project presentation and training logs — Confirmed in the project audit of 13 November 2026.93.57%
accuracy of the plagiarism cross-encoder on 9,908 sentence pairs
PAWS, MRPC and STS pairs; F1 0.930. The run output survives only as console text saved in a session log.
Training console output saved in a session logVerified. Backed by a file or public page that has been checked.Training console output saved in a session log — Read in the project audit of 13 November 2026.Exhibit 01 · Chart
In the lab, and on newer models
Exhibit 02 · Scorecard
Every detector, and what it scored
Run outputs read in the project audit of 13 November 2026. All detector rows are on the same 6,994-sample RAID and HC3 set unless a row says otherwise; none of it is a clean out-of-distribution test except the 140-text row.
| Measure | Result | Note | Source |
|---|---|---|---|
| DeBERTa-v3-large (fine-tuned) | 97.25% | AUROC 0.9989. | step1_output.log, REAL_METRICS.json |
| RoBERTa-large (fine-tuned) | 94.48% | AUROC 0.9872. | step1_output.log, REAL_METRICS.json |
| Longformer (fine-tuned) | 64.36% | Weak. AUROC 0.777. | step1_output.log, REAL_METRICS.json |
| XLM-RoBERTa (fine-tuned) | 36.75% | Below chance on a two-class task. AUROC 0.6418. | step1_output.log, REAL_METRICS.json |
| Fakespot RoBERTa (as released) | 96.67% | Within a point of the best detector the team trained. | step15_progress score files |
| ModernBERT (as released) | 90.06% | step15_progress score files | |
| Binoculars (as released) | 86.17% | step15_progress score files | |
| GLTR (as released) | 63.93% | step15_progress score files | |
| Four-model meta-classifier, 2,099 held-out rows | 97.86% | F1 0.984, AUROC 0.998. Held-out rows of the same pool. | step1_progress/STEP1_COMPLETE |
| Four-model meta-classifier, all 6,994 samples | 97.36% | False-positive rate 6.82%. | evaluation_report.json |
| After adversarial retraining | 94.71% | AUROC 0.984, F1 0.961. | step5_output.log |
| Later 14-feature meta-classifier, 56,994 samples | 76.49% | The eight-detector, 14-feature version, on 56,994 samples as its metrics file records it. F1 0.7718, AUROC 0.8472. Lower than the four-model figure, and listed so that nothing is hidden. | meta_classifier_metrics.json |
| 140 texts from newer language models | 55.7% | 70 AI, 70 human. The honest out-of-distribution number. | benchmark_results.csv |
| Plagiarism cross-encoder, 9,908 pairs | 93.57% | F1 0.930, AUC 0.979, on PAWS, MRPC and STS. | Training console output saved in a session log |
| Sentence-embedding similarity (validation) | Spearman 0.733 | similarity_evaluation_validation_results.csv | |
| Plagiarism index | 15,000 documents | step6_output.log | |
| Rewriting models | Not a working result | The fully fine-tuned Flan-T5-XL mostly copies its input; PEGASUS stopped at 13,000 of 39,000 steps. | step3_output.log, trainer_state.json |
Exhibit 03 · Compare
What back-translation does to the score
Before
62.0%
The average "AI" score the detector gave three test texts as written.
After back-translation
27.9%
The same three texts after a round trip through other languages, keeping the variant that scored lowest. One technical text barely moved, from 99.7% to 82.6%.
Three texts is a tiny sample. It is shown as a robustness test of detection, not as a tool: it shows how fragile the detector is.
01
What it is
Textegrity checks a piece of text three ways: is it AI-written, does detection survive rewriting, and is it copied. It was a university course project by a four-member team with a faculty guide. Its paper is an unpublished draft.
02
The hard part
Single detectors disagree, and the disagreement carries information. When one detector is sure a text is AI-written and another is sure it is human, the text has often been rewritten. A plain average hides that and lands near the middle.
03
What was built
Eight detectors. Four are transformer classifiers we fine-tuned on the RAID and HC3 datasets: DeBERTa-v3-large, RoBERTa-large, Longformer and XLM-RoBERTa. Four are used as released: Binoculars, a Fakespot RoBERTa detector, ModernBERT and GLTR. A logistic-regression meta-classifier stacks their scores with interaction features that encode agreement and disagreement between detectors. The robustness stage back-translates a text through five languages, scores each variant with the detector and keeps the one that scores lowest. The plagiarism stage combines a MinHash index of 15,000 documents, web and scholarly search, a fine-tuned sentence-embedding model and a cross-encoder. It all runs from a command-line pipeline and a FastAPI service; the web front end was never deployed.
04
What was found
On 2,099 held-out rows of its own data pool, the four fine-tuned detectors stacked together score 97.86% accuracy; DeBERTa alone scores 97.25%. On 140 texts written by newer language models the detector scores 55.7%, which is close to chance, and that is the number that matters. Two of the four fine-tuned detectors turned out weak (Longformer 64.36%, XLM-RoBERTa 36.75%), and the best released detector, at 96.67%, is within a point of the best one we trained. Across three test texts, back-translation lowered the average detector score from 62.0% to 27.9%; one technical text barely moved, from 99.7% to 82.6%.
05
What I'd change
There is no clean held-out split: every in-distribution figure comes from the same pool as the training data. An eight-detector version scored higher still on that pool, but the pool very likely overlaps its training sources, so that figure is not quoted here. The next version needs a test set built from different sources, and a larger, current benchmark than 140 texts, before it quotes any headline number. The rewriting stage belongs here as a robustness test: it shows how fragile detection is, and that is the finding.
Figure
One question
The answer is on this page. Get it right and it goes in your passport on the map.
Challenge· stamped
The detector scores 97.86% on held-out rows of its own data. What does it score on 140 texts written by newer language models?
55.7%: 78 of 140 texts right on a balanced set, where chance is 50%. Source: the project's own benchmark results file (benchmark_results.csv), read in the project audit of 13 November 2026.
Show the answer
55.7%. 55.7%: 78 of 140 texts right on a balanced set, where chance is 50%. Source: the project's own benchmark results file (benchmark_results.csv), read in the project audit of 13 November 2026.
- Stack
- Python
- PyTorch
- Hugging Face Transformers
- sentence-transformers
- scikit-learn
- datasketch (MinHash/LSH)
- FastAPI
- Helsinki-NLP opus-mt translation models
- llama-cpp (optional local models)
- Kubeflow (A100) and Apple MPS for training