# Parallel Processing

## What is the difference between concurrency and parallelism?

<https://stackoverflow.com/questions/1050222/what-is-the-difference-between-concurrency-and-parallelism>

**Concurrency** is when two or more tasks can start, run, and complete in overlapping time **periods**. It doesn't necessarily mean they'll ever both be running **at the same instant**. For example, *multitasking* on a single-core machine.

**Parallelism** is when tasks *literally* run at the same time, e.g., on a multi-core processor.<br>

I'd perhaps characterise concurrency as a property of a program or system (and parallelism as the run-time behaviour of executing multiple tasks at the same time)

## Worker vs Threads vs Processes

## Multi-Threading

* List are  not thread save, should not be used in multi-threading
* Deque is a little more complex. The documentation clearly states that both the .append() and .pop() operations are atomic, meaning that they won’t be interrupted by a different thread. Other methods in deque are not thread-safe&#x20;
* LifoQueueis designed to be fully thread-safe. All of its methods are safe to use in a threaded environment. It also adds optional time-outs to its operations which can frequently be a must-have feature in threaded programs

## Multi-Processing

## Multi-Threading vs Multi-Processing

threading: [https://nyu-cds.github.io/python-concurrency/03-threads/#:\~:text=The%20optimal%20number%20of%20threads,threads%20to%20run%20at%20once.](https://nyu-cds.github.io/python-concurrency/03-threads/)

{% embed url="<https://timber.io/blog/multiprocessing-vs-multithreading-in-python-what-you-need-to-know/>" %}

{% embed url="<https://blog.floydhub.com/multiprocessing-vs-threading-in-python-what-every-data-scientist-needs-to-know/>" %}

{% embed url="<https://www.guru99.com/difference-between-multiprocessing-and-multithreading.html#:~:text=Multiprocessing%20allocates%20separate%20memory%20and,as%20that%20of%20the%20process.&text=Multithreading%20avoids%20pickling.,to%20send%20to%20other%20processes>." %}

## Parallel Processing

## Implementation

### Multi threading

###

### Multi-processing

* <https://www.analyticsvidhya.com/blog/2021/04/a-beginners-guide-to-multi-processing-in-python/>

### Multi-processing and Scheduling using Celery

#### with Redis

#### with AWS SQS

### Distributed Processing

Expand number of workers and contract as per requirements (traffic on the site)

## Reference for Further Reading