> For the complete documentation index, see [llms.txt](https://ankit-apdc.gitbook.io/python-1/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ankit-apdc.gitbook.io/python-1/processing-related/parallel-processing.md).

# Parallel Processing

Multi-Processing, Parallel Processing, and Distributed Processing

## What is the difference between concurrency and parallelism?

<https://stackoverflow.com/questions/1050222/what-is-the-difference-between-concurrency-and-parallelism>

**Concurrency** is when two or more tasks can start, run, and complete in overlapping time **periods**. It doesn't necessarily mean they'll ever both be running **at the same instant**. For example, *multitasking* on a single-core machine.

**Parallelism** is when tasks *literally* run at the same time, e.g., on a multi-core processor.<br>

I'd perhaps characterise concurrency as a property of a program or system (and parallelism as the run-time behaviour of executing multiple tasks at the same time)

## Worker vs Threads vs Processes

## Multi-Threading

* List are  not thread save, should not be used in multi-threading
* Deque is a little more complex. The documentation clearly states that both the .append() and .pop() operations are atomic, meaning that they won’t be interrupted by a different thread. Other methods in deque are not thread-safe&#x20;
* LifoQueueis designed to be fully thread-safe. All of its methods are safe to use in a threaded environment. It also adds optional time-outs to its operations which can frequently be a must-have feature in threaded programs

## Multi-Processing

## Multi-Threading vs Multi-Processing

threading: <https://nyu-cds.github.io/python-concurrency/03-threads/#:~:text=The%20optimal%20number%20of%20threads,threads%20to%20run%20at%20once.>

{% embed url="<https://timber.io/blog/multiprocessing-vs-multithreading-in-python-what-you-need-to-know/>" %}

{% embed url="<https://blog.floydhub.com/multiprocessing-vs-threading-in-python-what-every-data-scientist-needs-to-know/>" %}

{% embed url="<https://www.guru99.com/difference-between-multiprocessing-and-multithreading.html#:~:text=Multiprocessing%20allocates%20separate%20memory%20and,as%20that%20of%20the%20process.&text=Multithreading%20avoids%20pickling.,to%20send%20to%20other%20processes>." %}

## Parallel Processing

## Implementation

### Multi threading

###

### Multi-processing

* <https://www.analyticsvidhya.com/blog/2021/04/a-beginners-guide-to-multi-processing-in-python/>

### Multi-processing and Scheduling using Celery

#### with Redis

#### with AWS SQS

### Distributed Processing

Expand number of workers and contract as per requirements (traffic on the site)

## Reference for Further Reading