However, data warehouses do not support flexible data analytics and processing. On the other hand, DLA provides powerful computing capabilities, allowing DG to analyze ad serving on a monthly and quarterly basis, accurately calculate the number of activities for each brand, and analyze the ad performance of each activity in terms of media, markets, channels, and data management platforms (DMPs). However, the rapid development of Internet advertising has created several challenges for the advertising industry. The two computing models of batch processing and stream computing are unified by improving the stream computing concurrency and increasing the time window of streaming data. Alibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology. With DIS, DLI defines all types of data points, which can be used by Flink jobs as sources or sinks. As shown in Figure 21, the platform allows each merchant to build its own data lake in one click. What data lake solutions are provided by major players in the field of cloud computing? Terms, This site is protected by reCAPTCHA and the Google, Eight Data Management Requirements for the Enterprise Data Lake, So whether your enterprise data warehouse is on life support or moving into maintenance mode, it will most likely continue to do what its good at for the time being: operational and historical reporting and analysis (aka rear view mirror). For any data lake project to be successful, you have to have the right people. Many articles emphasize the differences between a data lake and a data warehouse, but a data lake is essentially a specific implementation of converged data management. A data lake provides diverse analytics capabilities, including batch processing, stream computing, interactive analytics, and machine learning, along with job scheduling and management capabilities. For many fast-growing game companies, a popular game often results in extremely fast data growth in a short time. Batch processing is combined with stream computing to meet the needs of many emerging applications. Each data element in a lake is assigned a unique identifier and tagged with a set of extended metadata tags. This data can be converted from the JSON format to Parquet format. The platform cannot provide a full range of SaaS-based analytics functions to meet the various customization needs of all types of merchants. Download the report for free to learn more. A data lake provides big data capabilities, including the ultra-large storage space and scalability needed to process data on a large scale. Hot data is stored at the data center's upper layer for easy access by applications. If you plan to perform actions like these, you need to thoroughly understand the throughput requirements from a data ingestion standpoint, which will dictate throughput for storage and network as well as whether you can process the data in a timely manner. The official documentation for AWS Glue states "Each AWS account has one AWS Glue Data Catalog per AWS region.". This achieves a balance between data storage capacity and cost. Phase 1: This shows the offline data processing infrastructure, as represented by Hadoop. AWS's data lake solution provides all of the functions shown in the reference architecture except quality management and data governance. Data flows through the Lambda platform from left to right, as shown in Figure 3. It provides a unified area for the storage of the internal data of an enterprise or organization. The data lake has become a middle ground between EDWs and data dumps in offering systems that are still agile and flexible, but have the safeguards and auditing features that are necessary for business-critical data. Already, enterprises are using Azure or S3 in the cloud to store data persistently, but then spinning up clusters dynamically and spinning them down again when processing is finished. It also integrates seamlessly with operational stores and data warehouses so you can extend current data applications. DLI can be connected to Huawei Cloud Data Migration (CDM) and Data Ingestion Service (DIS). The data lake setup process is business-oriented and different from the process of building a data warehouse or data mid-end, which are also popular technologies. The stored data can include data of any format and of any type. To implement these capabilities, a data lake provides a series of data management components, including data access, data migration, data governance, quality management, asset catalog, access control, task management, task orchestration, and metadata management. Currently, cloud-native is the ideal way to build a data lake. I recommend using the stream computing model illustrated in Figure 24. Editors note: Ben Sharma is speaking on the topic of Building a modern data architectureat Strata + Hadoop World in San Jose, March 28-31,2016. In addition, the data lake monitors and records inter-system data mobility so you can trace the data flow. http://www.gartner.com/newsroom/id/2809117? A data lake supports a wide variety of data sources. After data is ingested, it should be quickly and efficiently analyzed using data analytics and machine learning tools to derive valuable insights and move vetted data into a data warehouse. Figure 8: Permission Separation for AWS's Data Lake Solution. If thats the case, what is the persistent storage layer? A data lake is the infrastructure for next-generation big data analytics and processing. As shown in Figure 2, Hadoop is a batch data processing infrastructure that uses HDFS as its core storage and MapReduce (MR) as the basic computing model. Data warehouses are large storage locations for data that you accumulate from a wide range of sources. Access permissions for actually stored data are further divided into data access permissions and data storage permissions. Scalability means a data lake is scalable in terms of size and capabilities. The final results of batch processing and stream computing are provided to applications through the service layer, ensuring access consistency. All these features are easy to use and allow the data lake to better meet business needs. Due to its fidelity and flexibility, a data lake stores at least two types of data: raw data and processed data. In this section, I will focus on how to build a data lake using the products of the data business unit. A data swamp is a deteriorated and unmanaged data lake that is either inaccessible to its intended applications or provides little value. Alibaba Cloud's data lake solution implements data integration and development by using two methods: DMS provides powerful data management and data security capabilities. Use cases are emerging even in sensitive industries like health care, financial services, and retail. Download the full report Architecting Data Lakes by Alice LaPlante and Ben Sharma to learn more. Data lake technology enables agile data processing and modeling, helping you quickly adapt to business growth and changes. I could not find a diagram of DLI's overall architecture on the Huawei website, so I tried to draw one based on my knowledge. A data lake pushes the results of data analytics and processing to appropriate storage engines, which support access from different applications. 5. Business Support: Many data lake solutions provide standard access interfaces, such as JDBC, to external users. Enterprise data accumulates over a long period of time, which is analogous to how rain fills a lake. Data access permission is similar to the access permission that a database grants to its tables. A data lake provides more features than a big data platform, such as metadata, data asset catalogs, permission management, data lifecycle management, data integration and development, data governance, and quality management. After migration, we integrated DLA and OSS to provide superior analytics capabilities for DG. Huawei's official documentation offer suggestions for building a data governance organization. In short, the big data infrastructure has evolved from the Hadoop architecture to the Lambda and Kappa architecture. Business data inflow is completed through extract, transform, and load (ETL). Azure allows you to develop a custom data lake solution based on Visual Studio. Figure 6 illustrates the overall data lifecycle in a data lake. You should follow three basic principles: separation of computing and storage, elasticity, and independent extensions. Therefore, it is increasingly important to create an economic and efficient precision data operations system to better support business development. A comprehensive data lake solution can direct the raw data flow to Kafka. 2022, OReilly Media, Inc. All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. Get a free trial today and find answers on the fly, or master something new and useful. One exception is that AWS Glue accesses AWS Lake Formation only at the database and table levels. Data flows from left to right. DAYU covers the core processes of data lake governance and provides tool support. Methods, Process, Examples, Techniques, Build a Cloud Data Lake Using E-MapReduce, Using Apache Spark for Data Processing and Analysis, MySQL Deep Dive - Implementation and Acquisition Mechanism of Metadata Locking, ClickHouse: Analysis of the Core Technologies behind Cloud Hosting to Cloud-Native, PolarDB-X Kernel V5.4.14: A New Version with More Refined Data Management Capabilities, Senior Technical Experts from Alibaba Discuss Data Warehouse Tuning Part 2, Senior Technical Experts from Alibaba Discuss Data Warehouse Tuning Part 1, Database Autonomy Services Lead the New Era of Database Autonomous Driving, Case Analysis of PostgreSQL Transaction Rollback, See the Past and Future of Redis from Redis 7.0, New-Gen Cluster Non-Inductive Data Migration of Alibaba Cloud In-Memory Database Tair, An In-depth Analysis of Atomicity in PolarDB, Commercially Available Data Lake Solutions. In practice, we recommend pushing the data processed in a data lake to data engines that support online businesses to improve the application experience. Instead, DG can scale out its infrastructure as needed. This field cannot be empty, Please enter your comment. However, in reality, the data stored in a data lake is not frequently accessed, and data lake-related applications are still in the exploration stage. Now that data storage and technology is cheap, information is vast and newer database technologies dont require an agreed upon schema up front, discovery analytics is finally possible. This means AWS Glue is closely integrated with AWS Lake Formation and has higher access permissions for the data stored in AWS Lake Formation. Note the following points when using a stream computing engine: The quasi-stream computing model is similar to batch processing. AWS defines a data lake in a more direct manner: A data lake is a centralized repository that allows you to store all of your structured and unstructured data at any scale. These components continuously improve big data platforms' data processing capabilities, such as HBase for online key-value (KV) operations, Hive for SQL, and Pig for workflows. by ApsaraDB. A schema is essential for any data application. It manages full data throughout its lifecycle. Azure's data lake solution consists of the data lake storage layer, interface layer, resource scheduling layer, and computing engine layer, as shown in Figure 15 (source: Azure website.) This is critical if you want to build a managed and governed data lake instead of the much-maligned data swamp.. Both DLA and AnalyticDB inherently support OSS access. This enables effective adaptation between data and your businesses but increases the cost of data warehouse maintenance at the early stage. A data lake can include structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs), and binary data (images, audio, video). This component connects to external data sources, detects the data format and schema, and creates metadata for the data lake in the corresponding data catalog. YM is a data intelligence service provider. If you need these two functions, you can develop them as needed. The data lake solution introduces AnalyticDB, a cloud-native data warehouse, to support low-latency interactive analytics that otherwise cannot be implemented by DLA. Data privacy and security are critical, especially for sensitive data such as protected health information (PHI) and personally identifiable information (PII). Data in the ODS is used to build a multidimensional subject DM by using the dimensional modeling method. Metadata inflow involves two steps: data source preparation and metadata crawling. Huawei's data lake solution provides an ecosystem with a more sophisticated upstream-downstream relationship than that of AWS's data lake solution. DAYU generates quality rules and a transformation model based on the data schema and metric model. Just as you must think of metadata from an enterprise-wide perspective, you need to be able to integrate your data lake with external tools that are part of your enterprise-wide data view. Azure implements YARN-based resource scheduling. Even schema-less databases, such as MongoDB, recommend using identical or similar structures as a best practice. Therefore, a data lake is more suitable for innovative enterprises and enterprises with rapid business changes and growth. Data in the ODS is processed based on the predefined EDW paradigm and then transferred to the EDW. It also provides multiple computing engines, such as U-SQL, Hadoop, and Spark. The Lambda architecture integrates stream computing and batch processing. All DMs are associated through consistent dimensions to form an enterprise- or organization-wide generic data warehouse. The analyzed data can be used for basic analytics functions, such as behavior statistics, customer profiling, customer selection, and ad serving monitoring. If you understand DLI differently, you are welcome to share your views. Alibaba Cloud provides a wide range of data products. Naturally, you may not have all, the answers upfront, so it might be necessary to perform a proof of concept to get some experience and to tune and learn along the way. Data from data sources of an operational or transactional system is extracted or loaded to the ODS. A highly scalable ingestion-layer system that extracts data from various sources, such as websites, mobile apps, social media, IoT devices, and existing Data Management systems, is required. Figure 11 illustrates the implementation of the DAYU data governance methodology (source: the Huawei Cloud website. Figure 19: YJ's Data Lake Solution After the Transformation. Clarify your needs based on data applications, and generate business-adapted data during the data ETL process. Editor's note: This is an excerpt from the new report "Architecting Data Lakes", by Alice LaPlante and Ben Sharma. DLA is deeply integrated with AnalyticDB in the following two aspects: The combination of DLA and AnalyticDB integrates data lakes and data warehouses under cloud-native. This requires robust data management capabilities, which cover data sources, data connections, data formats, and data schemas. In addition, traffic costs are constantly increasing. As shown in Figure 24, to apply the real-time stream computing model to the data lake, you can introduce Kafka-like middleware as the data forwarding infrastructure. With constant changes in data processing capabilities and processing demand, you may find it impossible to achieve high real-time performance in certain processing scenarios no matter how you improve the batch processing performance. Many enterprises are building or plan to build their own data lakes. A data lake is a type of big data infrastructure with an excellent total cost of operation (TCO) performance. To meet your business needs at an optimal TCO, data lakes provide some basic technical features, such as the separate extension of storage and computing, a unified storage engine, and a multi-mode computing engine. In recent years, the cost of traffic acquisition has been increasing, forcing many companies to invest heavily to attract new online customers. To reduce the frequency of writing intermediate results from data processing, computing engines, such as Spark and Presto, cache data in the memory of compute nodes whenever possible. Were on a journey towards connecting enterprise data together. Azure Data Lake works with existing IT investments for identity, management, and security for simplified data management and governance. The big data infrastructure is specially designed for storage and computing, but it ignores data asset management. An effective way to break out of ineffective online advertising methods is to use data tools to convert more of your website visitors into paying customers and refine ad serving comprehensively. DLI can be viewed as a combination of AWS Lake Formation, AWS Glue, Amazon Athena, and Amazon EMR (based on Flink and Spark.) For decades, the foundation for business intelligence and data discovery/storage rested on data warehouses. Specifically, a data lake not only provides sufficient storage and computing capabilities for an increasing amount of data but also constantly provides new data processing models to meet emerging needs. With CDM, DLI can access the data of Internet Data Centers (IDCs) and third-party cloud services. It should be flexible to run in batch, one-time, or real-time modes, and it should support all types of data along with new data sources. DMS manages data at four granularities, database, table, column, and row, providing control over data security required by enterprises. For example, are you going to use decoupling between the storage and the compute layers? In short, we transformed YJ's architecture to help it build a data lake, as shown in Figure 19. It connects disparate data sources to computing engines to form an upstream-downstream relationship. DG is a leading global provider of intelligent marketing services to enterprises looking to expand globally. Figure 9: Mapping of AWS's Data Lake Solution to the Reference Architecture. How can we build basic data lake architecture? Quality management and data governance are closely related to the organizational structure and business type of an enterprise, requiring a great deal of customization and development work. As part of your technology platform and architecture, you must think about what the scale-out capabilities of your data lake will look like. Data lake solutions are likely to evolve in the following directions in the future: 1 . The industry already has many universal practices for data lake technology selection. Data storage permission specifies the explicit or implicit access permissions for each specific data catalog in Amazon S3. Based on Alibaba Cloud's database ecosystem, including OLTP, OLAP, and NoSQL databases, DLA provides SQL-based external data processing capabilities. 2. Data Flexibility: As shown in the "Schema" row of the preceding table, schema-on-write or schema-on-read indicates the phase in which the data schema is designed. This solution provides a specialized component to retrieve metadata from external data sources. To make better use of data, enterprises and organizations must take the following measures to manage data assets: A data lake not only provides the basic capabilities of a big data platform, but also data management, data governance, and data asset management capabilities. Then, the engine writes the data processing results to OSS, a relational database management system (RDBMS), NoSQL database, or data warehouse as needed. For instance, a biotechnology company can implement a data lake that receives manufacturing data, research data, customer support data, and public data sets and provide real-time visibility into the research process for various user communities via different user interfaces. A data lake is designed based on a consideration of asset management. Data Lake Analytics (DLA): An Interactive Analytics Service That Utilizes Serverless Architecture, Exploring Application and Analysis of Big Data in The Cloud, Quick Implementation of Data Lake House Based on MaxCompute, What is Data Analysis? It stores structured, semi-structured, and unstructured data, It supports access to all types of disparate data sources, It discovers, manages, and synchronizes metadata, It provides the built-in SQL and Spark computing engines, which can process various types of data more effectively. Figure 16: DG's Transformed Advertising Data Lake Solution. AWS effectively implements all its services in the ecosystem. 2. Technology Selection: Select the technologies used to build a data lake based on the data analysis results. This section analyzes the data lake solutions of major cloud service providers and illustrates them in a data lake reference architecture to compare the advantages and disadvantages of different solutions. Data warehouses are popular with mid- and large-size businesses as a way of sharing data and content across the team- or department-siloed databases. Support for multi-modal computing engines, such as SQL, batch processing, stream computing, and machine learning, 3 . Increasing Internet advertising costs have made companies give up the strategy of expanding their customer bases by buying traffic. There are currently heated discussions among frontline personnel on the best way to build a data lake. A data lake uses a scalable distributed file system for storage. DLA pushes the intermediate results to the data warehouse with a relatively fixed data schema, so you can implement business applications based on the data warehouse. Figure 14 illustrates a classic implementation of data lake-data warehouse integration in the gaming industry. If so, has this solution been applied in actual scenarios? 5. Data Rich Computing Engines: A data lake supports a diversity of computing engines, including batch processing, stream computing, interactive analytics, and machine learning engines. I currently work in the data business unit. 2. Ralph Kimball proposed the top-down (DM-DW) data schema. Data Lake gives business users immediate access to all data. After all, the data lake needs to provide value that the business is not getting from its EDW. They can find correlations and insights within the data as they get to know it. We can further analyze the characteristics of a data lake in terms of data and computing: 1. Data Fidelity: A data lake stores data as it is in a business system. DLA's SQL syntax is fully compatible with that of AnalyticDB, allowing you to develop applications based on data lakes and data warehouses using the same technology stack. Each stage consists of one or more tasks, which are executed concurrently to improve the computing process parallelism. Stream computing engines will be described later. A data lake supports comprehensive data lifecycle management. A data lake provides centralized storage for the data of an enterprise or organization. The platform provides multi-client SDKs for merchants to access tracking data in diverse forms, such as webpages, apps, and mini programs. Data lakes can give retailers profitable insights from raw data, such as log files, streaming audio and video, text files, and social media content, among other sources, to quickly identify real-time consumer behavior and convert actions into sales. The data lake is the only place to store your enterprise or organization's raw business data. Is there a single system to solve all these problems? Does Alibaba Cloud have a mature data lake solution?
Sitemap 18