> For the complete documentation index, see [llms.txt](https://ankit-apdc.gitbook.io/system-design/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ankit-apdc.gitbook.io/system-design/building-blocks/database.md).

# Database

## Resources

* <https://www.youtube.com/watch?v=8ogJlOIxKVE&ab_channel=KeertiPurswani>

## Data Lake vs Data Warehouse

### Data Lake

* is a system or repository of data stored in its natural/raw format, usually object blobs or files
* is a single store of data including raw copies of source system data, sensor data, social data etc
* A data lake can include structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs) and binary data (images, audio, video)
* A data lake can be established "on premises" (within an organisation's data centres) or "in the cloud" (using cloud services from vendors such as Amazon, Microsoft, or Google)
* [**Wikipedia link**](https://en.wikipedia.org/wiki/Data_lake)

### Data Warehouse

* Transformed data is used for tasks such as reporting, visualisation, advanced analytics and ML&#x20;
* [Extract, transform, load](https://en.wikipedia.org/wiki/Extract,_transform,_load) (ETL) and [extract, load, transform](https://en.wikipedia.org/wiki/Extract,_load,_transform) (ELT) are the two main approaches used to build a data warehouse system

### Blobs

* BLOB stands for **Binary Large OBject**. A blob is a data type that can store binary data
* This is different than most other data types used in databases, such as integers, floating point numbers, characters, and strings, which store letters and numbers
* BLOB is a large complex collection of binary data which is stored in Database
* Basically BLOB is used to store media files like images, video and audio files
* Due to its ability to store multimedia files it takes a huge disk space
* Also length of BLOB may go upto 2, 147, 483, 647 characters
* BLOB provides fast multimedia transfer

## **ETL vs ELT Approach**:&#x20;

### ETL Approach

### **ELT Approach**&#x20;

* **Extract, load, transform** (**ELT**) is an alternative to (ETL) used with data lake implementations
* the data is not transformed on entry to the data lake, but stored in its original raw format, this enables faster loading times
* ELT requires sufficient processing power within the data processing engine to carry out the transformation on demand, to return the results in a timely manner
* Since the data is not processed on entry to the data lake, the query and schema do not need to be defined a priori

## Data Pipeline

## Lambda Architecture

## Big Data

![](https://679135566-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ma8GKQU92FWew4_578V%2F-McrhXvGymIqT-rMTj-L%2F-Mcrjni5Ud7ZOyXbQu1o%2Fimage.png?alt=media\&token=0eb1ecd1-c795-4600-b630-caea88588600)

{% hint style="success" %}

* **Latency**: Response time for single request
* **Throughput**: Number of transactions per sec
  {% endhint %}

## Different Processing Types

### Stream Processing

### Batch Processing

## References for Further Reading

{% embed url="<https://towardsdatascience.com/scalable-efficient-big-data-analytics-machine-learning-pipeline-architecture-on-cloud-4d59efc092b5>" %}
In case unable to open the link, open in incognito window
{% endembed %}