Clara Jiménez Arias

Data lakes promise flexibility at scale, but without governance controls the lake becomes an unusable data swamp. In this article, we explore the challenges organisations face and a range of solutions that can help pave the way to success. The challenge Why data quality and governance matter more than ever The low cost of cloud storage has made data ingestion significantly easier. However, this creates a paradox: storing information is now easier than organising, validating or deleting it. As a result, datasets accumulate without a clear owner, with inconsistent structures and lacking metadata. This leads to: Loss of productivity: analysts spend up to 40% of their time validating data before they can use it. Regulatory risk: data without traceability or retention controls complicates audits and legal obligations. Hidden costs: queries on poorly organised data drive up compute costs and make expenditure difficult to predict. Duplication of effort: without confidence in available data, teams rebuild processes and create uncontrolled copies. Governance Governed data, smarter decisions. The control plane of the data lake Data governance is not a documentation layer. It is the control plane that defines who can ingest data, modify schemas, grant access and manage the data lifecycle. Without this control plane, a data lake inevitably tends to become a data swamp. Mandatory data ingestion contract The most effective way to prevent chaos is to require every ingestion process to comply with a minimum contract. When this contract is optional, the data lake quickly becomes a repository with no traceability or context. Distributed accountability model Centralising responsibility within the platform team creates bottlenecks and weakens accountability. Source system: defines and ensures data quality rules at source. Platform team: technically enforces governance policies. Domain owners: oversee semantic consistency within their area. Ingestion owners: implement validation controls at the point of entry. Data quality Data quality is your most valuable asset. Six dimensions that must be measured, not assumed Quality is not a binary condition. It is a multidimensional concept with objective criteria, acceptable thresholds and metrics for continuous monitoring. Quality gates at ingestion Quality must be assured at the point of entry into the data lake. Post-ingestion validation alone is not enough to prevent errors from spreading. Schema validation: data types, mandatory fields and constraints. Referential integrity checks against related systems. Detection of duplicates and management of late-arriving data. Automated monitoring with alerts when thresholds are breached. Invalid records are quarantined — they are never silently discarded. Metadata The DNA of data. The operational index of the data lake Metadata is not optional documentation; it is an essential operational index that reduces discovery times and prevents duplication of effort. But what exactly is metadata? Metadata provides the descriptive framework that enables an organisation to understand what data it owns, how it originates, where it resides and what it means to the business. It includes information about technical and business processes, rules and constraints, as well as both logical and physical data structures. This knowledge spans three dimensions: The data itself: databases, elements, models and schemas. The concepts they represent: business processes, systems, code and infrastructure. The connections between them: relationships, dependencies and flows. — There are three types of metadata (TON):      Technical: Where is the data stored?      Organisational / Operational: Who generates the data?      Business: How is the data classified? Organisational value Integrated management of these three types of metadata enables: Traceability from data origin through to consumption Governance based on clear responsibilities Interoperability across systems and teams Trust in the quality and meaning of information — Without metadata, data is simply volume without context.     With metadata, it becomes a strategic asset that can be understood and governed. Lifecycle The alpha and omega of data. Retention is not infinite Many data lakes are designed on the assumption of indefinite retention, which is unsustainable due to regulatory requirements and storage costs. Every dataset requires a retention class with a clearly defined outcome: Deletion: scheduled removal once the defined retention period has expired. Archiving: immutable cold storage with an auditable chain of custody. Legal hold: suspension of all actions while a legal or regulatory obligation remains in force. — The system must be able to demonstrate at any time:      What data existed.      Who accessed it.      Why it was deleted or retained.      Without this, any audit or e-discovery exercise becomes a manual process with high levels of risk. Security Security, the non-negotiable asset. Least privilege by design The concentration of data within a lake increases the attractiveness of privileged access. Access control must be derived from data classification, not its location. Policy-based: access rules are derived from data classification and purpose, not from manually maintained user lists. Just-in-time: elevated privileges are granted temporarily, with justification and full traceability. Segregation of administrative privileges: no single role has unrestricted access to all data. Centralised auditing: consolidated records of who accessed what, when and in what context. Diagnosis Analysis is key. Symptoms and root causes Recommendations Practical solutions for every company. Practical actions Implement ingestion contracts before scaling: do not onboard new domains unless metadata capture and quality validation are mandatory. Define data owners: the platform enables, but responsibility for accuracy rests with the business. Automate quality gates: manual validation does not scale. Controls must be integrated into ingestion processes from day one. Make retention policies executable: the system should automatically archive or delete data when retention policies expire. Monitor cost by workload: separate exploratory computing from production analytics to avoid budget overruns. Integrate security with data classification: access controls should be driven by information sensitivity, not by physical location. Conclusion The data lake is not an option; it is a necessary evolution. An evolution that must embrace data governance without compromise, not only to understand what data exists and where it resides, but also to take a significant step forward in data utilisation and Artificial Intelligence. Governance + intelligence = success Where, Governance: data governance ensures consistency, quality, traceability and trust, establishing the rules that underpin and strengthen the entire data ecosystem. Intelligence: Artificial Intelligence interprets data, learns from it and creates value from it, unlocking its full potential while relying on trusted and reliable foundations.   AUTHORS Héctor García Data governance specialist Clara Jiménez Data governance specialist * * * AI & Data From suspicion to trust: the real journey to better Data Governance May 20, 2025