Expec Reclaim
Reclaim storage without guessing.
Daily, evidence-backed answers to which Hive tables and partitions can be purged on Cloudera CDP — with the coverage proof that makes the answer safe to act on.
Expec Reclaim
Know what is cold before you delete it.
Your cluster already keeps the evidence: who queried what, who opened which file, how large every table and partition is. Expec Reclaim turns that evidence into a daily answer to one question — what hasn't been read, and can it be removed — with enough proof for a data owner to sign off on.
No NameNode changes. No agents on the cluster. No passwords. Everything is read-only.
HDFS cannot tell you when a file was last read
On modern CDP clusters, access-time tracking is off. Every file reports 1970. Housekeeping ends up running on modification time, directory listings, and guesswork — storage keeps growing, and nobody can actually prove what's safe to delete.
The evidence that does exist is hard to use:
Audit files are enormous. A single day of HDFS audit can run over 100 GB, and the audit keeps daily folders for the current month only.
A SQL statement isn't a data read. A query planner can touch thousands of tables in a day; only a fraction of them actually open a file.
The same table can look busy and dead at the same time. One shared service account reads the whole catalogue; a person never opens it.
Deleting is irreversible. A false positive costs far more than a missed saving.
What it gives you
How it works
Metastore ──► table, view and partition snapshot (daily diff)
Reports host ──► file-system image, read offline (no NameNode load)
Ranger audit ──► Impala and HiveServer2 queries ─┐
Ranger audit ──► HDFS file opens ────────────────┴──► classification, coverage, candidates
│
review app, views, exports
Five steps, run in order by a single daily command:
Snapshot the Metastore — tables, views, partitions, and what's changed since yesterday.
Pull the file-system image — read offline with the standard Hadoop tool, so the NameNode is never scanned.
Aggregate the inventory — size, files, blocks, small files and partition health per table and partition.
Ingest the audits — SQL queries and HDFS file opens, closed days only, with a completeness verdict for every source and day.
Classify and report — resolve views, apply reader classes, combine the signals, and publish the status, candidates and reports.
Each step overwrites its own day. A re-run is safe, a partial run replaces only what it covers, and a missed day can be caught up.
Why the answer can be trusted
Cold means proven cold. A table is only marked COLD when every source that could have seen a reader of it was complete for the whole window. Otherwise it's NOT_SEEN, with the number of days actually observed. Missing data never turns into a purge.
Every source is checked every day. Each audit day gets a verdict from its record count against the same weekday over the previous weeks. Short, empty and unparsable days are flagged and break coverage; a spike is treated as real data, not as a gap.
Views are resolved. A read of a view counts for the tables underneath it, so reporting layers don't make their source tables look dead.
Readers are classified, not assumed. Engines, ETL, BI accounts, people, maintenance jobs and shared catalogue-wide accounts are treated differently — and reads that are ignored stay visible, so you can see what a shared account touched.
Partition-level evidence. File opens from the HDFS audit are mapped to individual partitions, so old partitions of a healthy table can be separated from the data that's still in use.
It's built to say no. In its first full production run, it classified 67,378 tables and 734,672 cold partitions — and nominated no candidate at all, because two shared service accounts read most of the catalogue. It surfaced those accounts for classification instead. A false purge avoided is worth more than a headline number.
Proven at production scale
The first run deliberately produced no purge candidate. That's the point: the tool shows you the evidence, and tells you where the evidence isn't yet strong enough.
The review application
A self-hosted web application for the people who approve deletions:
Overview — the state of the cluster: size, classes, coverage and confidence
Candidates — purge candidates, filterable by owner, database and size
Written, not read — producers writing data nobody queries
Heatmaps — read recency by database and table
Cold partitions — per table, with retention suggestions
Growth — size history from the daily inventory
Trash — space recoverable from the Trash, by account and checkpoint
Small files and partitions — where the NameNode cost sits
Tables without data, orphan directories — metadata and space clean-up
Run status and legend — what ran, what's complete, and what every flag means
CSV export on every page — for owner review and approval records
Accounts are hidden by default. The same data is exposed as Impala views for SQL users, with validation queries included for spot-checking any single table.
Deployment
Runs in the Cloudera AI workspace, scheduled once a day, or deployed as a Cloudera AI Application for the review UI.
CDP Private Cloud Base 7.x with Hive, Impala and Ranger audit. Reads the Impala, HiveServer2 and HDFS audit sources.
Requires: Ranger audit enabled on those services, read access to the audit location, and SSH access for the file-system image.
Storage: Iceberg tables in your own schema, sized to the number of tables and partitions.
Security: key-based SSH and keytab only. No passwords anywhere in the configuration or the code.
Read-only against your cluster. It never deletes, moves or compacts anything.
Who it's for
Platform and data engineering teams who own storage growth and NameNode pressure.
Data governance and CDO teams who need a defensible, auditable retention decision.
Data owners and BI teams who have to approve what gets removed.
Capacity planners who need to know whether to buy storage or reclaim it.
Questions
Does it delete anything? No. It produces evidence, candidates and reports. Removal stays with your change process and the data owner.
Do we need to turn on access-time tracking? No. That's the last option, not the design. The tool works from audit and image evidence that already exists.
How soon is it accurate? SQL reads are usable from the first run, once the audit is backfilled. Tables read only by direct Spark jobs need about 90 days of HDFS audit history; until then, results are marked provisional and carry a MEDIUM confidence.
What if a read source is missing for a day? That day is flagged incomplete, coverage stops there, and nothing is called cold on the strength of it.
Will it overload the cluster? No. The image is read offline from a copy, the audits are plain files, and the work is done in Spark on data that's already on disk.
Talk to us
Want to see what your cluster would report? We can run Expec Reclaim against your existing audit and image data and walk you through the first results — including the cases it refuses to nominate.
Expec Consulting — get in touch to arrange a walkthrough.
Comments