The Machinery Data You Are Allowed to Train On

Every survey of machinery fault datasets prints the same columns: sampling rate, number of records, fault classes, total size. Almost none print the one that decides whether a model trained on the data will work — the number of distinct physical assets. Not files, not recordings, not operating conditions. Individual bearings, gearboxes, machines.

It matters because a model that has seen a defect on one bearing can recognise that bearing again with almost no effort, and this looks identical to diagnosis until you show it a different one. Counted properly, the public record is far thinner than it appears. CWRU is one rig and twenty-eight seeded bearings across 161 files. MaFaulDa's 1,951 records are one machine. Paderborn is thirty-two bearings. IMS is twelve bearing positions, FEMTO seventeen, XJTU-SY fifteen. The largest genuinely multi-asset sets in the public record are not vibration at all: MIMII is twenty-eight machines recorded acoustically, and CARE is thirty-six wind turbines of SCADA tags.

So the first honest question is not which dataset is best. It is which datasets you are permitted to use at all, and what is left once you have applied the requirements any real project has.

CWRU is the most-cited bearing dataset in machine-learning literature. Its pages carry no licence, no terms of use, no citation request — the words simply do not appear. That is not permission. Absent an explicit grant, default copyright applies, and a company training a model it intends to sell is relying on an assumption nobody at Case Western ever wrote down.

Paderborn is explicit and inconvenient: CC BY-NC 4.0, with the terms stating that noncommercial academic use is allowed and that commercial use requires contacting the author. Paderborn is also the only open set that pairs motor current with vibration — so the single best public resource for current-signature diagnosis is the one a commercial team may not touch by default. Worse, a Zenodo mirror of the same data declares CC BY 4.0, a licence the rightsholder never granted. Download from the mirror, believe the label, and you are infringing while documenting your compliance.

Others fail differently. MFPT has gone: mfpt.org redirects to asnt.org and the dataset URL now returns a web page instead of the archive, which survives only in the Wayback Machine. NREL's wind-gearbox benchmark — two real 750 kW gearboxes, one healthy and one damaged, the most operationally realistic vibration data in the list — states plainly that you cannot share the datasets with others, and adds a pre-publication review clause. MathWorks ships bearing data under CC BY-NC-SA with a carve-out that covers MathWorks and nobody else.

Apply the filter honestly and a commercial trainer is left with four small CC BY sets — two from Ottawa, one from UConn, one from HUST — every one of them seeded defects on a handful of assets, plus two CC BY-SA sets that are acoustic or SCADA rather than vibration. There is no licence-clean, multi-asset, run-to-failure vibration corpus in the public record. That is the finding, and it is worth sitting with before designing a project around one.

The obvious response is to buy the data instead. It is not for sale. Snowflake's marketplace was enumerated in full — 4,166 listings, zero machinery vibration. Databricks, 2,076 listings via its API, zero. Datarade returns no results for the term. What look like listings on AWS Data Exchange carry the embedded type proServ — professional services, not data. No company selling machinery fault data could be found, and the only route to commissioning it that surfaced was the Paderborn chair's own contract test-rig work, which is a pointed irony given how they license the dataset they already published.

The traffic runs the other way. Condition-monitoring vendors take rights over the data your plant generates, and the clauses are considerably broader than most buyers realise. Schaeffler's terms — Version 02, dated 15 March 2022 — cover data from vibration measurements on a worldwide, perpetual, irrevocable, non-exclusive, free of charge, sublicensable and assignable basis, including the use of artificial intelligence. The same document says the customer is not entitled to demand an extract or overview of the data it supplied, and that Schaeffler retains what it derived even after deletion of the data.

One clause deserves separate attention. Section 4.6 extends that grant to the customer's own labels and comments. Labels are the expensive part of a training corpus — they are what an engineer produced by opening the machine and looking. It is the only clause found in any vendor contract that names them explicitly, and it converts your maintenance history into somebody else's training set.

The others vary in how far they go. Waites takes an exclusive, transferable, irrevocable, perpetual licence over data the client is elsewhere told it owns — exclusive being the word that should stop a reader, since it can lock out the owner. Fluke's agreement grants Fluke ownership of aggregated data with unrestricted use and sharing for any purpose, commercial or otherwise, and a full-text search of all 37,850 characters finds no customer-ownership clause anywhere in it. SKF's is narrower and better drafted: raw data comes back on request, but only raw data, never the analyses, and only within three months of termination.

Two honest caveats. Augury's and Petasense's machine-data terms could not be reached at all, so nothing is claimed about them — Petasense's only public 'Terms and Conditions' is a contact-form privacy notice containing no terms, which is itself informative. And the Waites and Fluke documents carry no date or version anywhere in them, which is a material fact about a contract you are asked to sign.

Suppose the licensing is settled and you train on CWRU anyway, as thousands of papers have. The accuracy you report depends almost entirely on a decision that has nothing to do with your algorithm.

We downloaded CWRU's 12 kHz drive-end and fan-end records directly from Case Western, scraped the file-to-label mapping from the pages rather than hard-coding it, and ran the same random forest on the same twenty-six condition-monitoring features over 7,051 windows from eleven physical bearings. The only thing that changed between runs was how the windows were divided into training and test.

Split the windows at random and the model scores 99.97%. Split so that no window from a test recording appears in training — the fix most careful papers apply — and it scores 99.66%. Split by physical bearing, leaving one bearing entirely unseen, and it scores 24.14% against a majority-class baseline of 46.75%. The model does worse than guessing the commonest class, because on a bearing it has never met it systematically confuses inner-race with outer-race.

Read the middle number again, because it is the one that matters to practice. Run-level splitting closes four tenths of one percent of a seventy-six-point gap. The model was never memorising the individual window; it was memorising the individual bearing. Overlap between windows is a red herring too — set the overlap to zero and the leaky protocol still returns 99.83%. Under the random split, 86.7% of test windows have their nearest training neighbour in the same file, and 100% have it on the same physical bearing.

The 24.14% deserves the same scepticism we just applied to the 99.97%. It is a mean across leave-one-bearing-out folds with a standard deviation of 39.5 points — enormous — and the pooled accuracy across all folds is 17.81%. A four-fold grouped variant of the same protocol collapses to under 1%. The honest reading is not 'the true accuracy is 24%'; it is 'once you remove the leakage there is almost no transferable signal in this protocol, and the variance is so wide that any single number overstates what was learned'.

CWRU also confounds asset with fault size, so a leave-one-bearing-out fold changes both at once. We ran a cleaner separation to check: train on drive-end bearings and test on fan-end bearings at matched severities. That gives 37.61% against a 46.66% baseline, with inner-race recall of 0.004 — while the leaky within-fan-end protocol on the very same records returns 99.83%. The collapse is not an artefact of confounded severity. It survives every attempt to explain it away.

There is a control experiment worth running before anyone trains anything. Take the envelope spectrum, compute the energy in narrow bands at BPFO, BPFI and 2×BSF, and simply declare the largest one the fault. No training, no split, no parameters fitted to the data, so no leakage is even possible.

That rule scores 68.34% on the same 7,051 windows — forty-four points above the trained model's honest score, and twenty-two above the majority baseline. A calculation from the bearing geometry, which any analyst can do by hand, outperforms a random forest that reported 99.97% two sections ago.

State its weakness as plainly. The rule is good at outer-race faults (recall 0.90) and inner-race (0.79), and poor at ball faults (0.20) — which is the expected result, because ball-spin energy is modulated by the cage and smears across the spectrum rather than standing up as a clean line. It is not a replacement for a trained model. It is the baseline that a trained model has to beat before anyone should believe it has learned anything, and on this dataset, honestly evaluated, it does not.

This is also why bandwidth deserves attention. Reduce the data to the 2 kHz a typical route collector delivers and the leaky number falls by less than a point, to 99.25%, while the honest number halves to 13.63%. A benchmark that barely notices losing five-sixths of the diagnostic content is not measuring diagnosis.

Almost all of the licence-clean data is seeded — a notch cut by electrical discharge machining, a hole drilled, a tooth ground away. A machined notch has sharp edges and a fixed geometry from the first revolution. A real spall begins as subsurface fatigue, breaks through, and grows with debris circulating through the raceway. They do not look the same, and models know it.

Lessmeier and colleagues built the Paderborn dataset specifically to measure this, and their own tables give the number: classifiers trained on artificially damaged bearings and tested on naturally degraded ones drop from 98.3% to 75.0%, a 23-point fall, and 23 to 34 points for the algorithms that work best in-domain. On motor current the drop is 47 points. Their conclusion, in their words, is that artificial training data is apparently not sufficient.

The same caution applies to speed. Transfer-learning benchmarks on Paderborn show that moving from 1500 to 900 rpm costs around 56 points, while a sevenfold torque change costs almost nothing. Speed changes the physics of what the sensor sees; load largely does not. This is worth remembering when someone proposes a diagnostic method that depends on a run-up or coast-down, because most plant machines run at one fixed speed and will never provide one.

And label provenance is weaker than the tables suggest. FEMTO's own documentation states that nothing is known about the nature and the origin of the degradation. An independent analysis of IMS test set 3 found no outer-race signature at all in the data labelled as an outer-race failure. If the label is wrong, every number computed from it is decoration.

One last mismatch, and it is the one that turns a good model into an ignored alarm. CWRU is 81% faulty. A plant is not. Most machines on most days are fine, and a detector's usefulness is governed by how it behaves against a large healthy majority — which is exactly the population the public datasets barely contain. CWRU has one healthy bearing, so even a perfect result bounds the per-asset false-alarm rate no more tightly than about 0.95 — the exact one-sided 95% bound for zero events in one trial, 1 − 0.05^(1/n), of which the familiar rule of three is only the large-sample approximation. A dataset cannot demonstrate a low false-alarm rate it never had the healthy samples to measure.

If the public record cannot supply a licence-clean, multi-asset, honestly-labelled corpus, the remaining option is to generate one. A fault simulator is the obvious instrument, and it is worth being precise about what it does and does not solve, because the temptation is to oversell it.

It does not remove the domain gap. A benchtop rig is a small machine at one mounting, and it inherits several of the limitations catalogued above — most importantly, a bearing adapter with a machined defect sits on the artificial side of exactly the artificial-versus-natural divide that cost Lessmeier 23 points. It is also not a new category of data: MaFaulDa, one of the public datasets in the table, is itself recorded on a commercial fault simulator. A rig corpus is the same kind of thing, with better provenance and a cleaner licence. The honest verdict is augment and validate, not replace.

What a rig genuinely fixes is the specific list of defects this post has been documenting. You know the fault because you installed it, and you can confirm it by teardown rather than inferring it from a rig configuration. You can run many distinct bearings, which is the axis every public dataset is thinnest on. You can sweep operating points deliberately instead of accepting the four somebody else recorded. You can acquire through the same sensor and the same mounting your route actually uses, instead of discovering later that the model was trained at 48 kHz on a stud mount and deployed at 2 kHz on a magnet. And you own the result outright, with no licence to read and no clause quietly claiming your labels.

For that corpus to be credible rather than merely ours, it has to meet conditions we would demand of anyone else: at least twenty distinct bearings per class rather than twenty recordings; more than one rig, so the model cannot memorise one foundation; several teardown-and-rebuild replicates, because reassembly changes the signature; a run-to-failure subset kept strictly separate from the seeded one; blind teardown confirmation with the disagreement rate between the installed label and the teardown finding published rather than hidden; an asset-wise canonical split shipped with the data so nobody can accidentally report 99.97%; and every model scored out-of-rig alongside its in-rig number.

That last item is the whole argument in one line. A dataset's value is not the accuracy it enables anyone to report. It is whether the number survives being tested on a machine it has never seen.

If you are training a machinery diagnosis model, the binding constraint is rarely the algorithm. It is that the data you can legally use is seeded, single-asset and acquired under conditions you will never meet in the field — and that the honest accuracy on it is far below the published one.

We build the rigs and the acquisition chain that let you generate labelled fault data with provenance you can defend, across as many distinct assets as your protocol demands, on terms with nobody else's name on them.