Parallel Processing

Multi-Processing, Parallel Processing, and Distributed Processing

What is the difference between concurrency and parallelism?

https://stackoverflow.com/questions/1050222/what-is-the-difference-between-concurrency-and-parallelism

Concurrency is when two or more tasks can start, run, and complete in overlapping time periods. It doesn't necessarily mean they'll ever both be running at the same instant. For example, multitasking on a single-core machine.

Parallelism is when tasks literally run at the same time, e.g., on a multi-core processor.

I'd perhaps characterise concurrency as a property of a program or system (and parallelism as the run-time behaviour of executing multiple tasks at the same time)

Worker vs Threads vs Processes

Multi-Threading

• List are not thread save, should not be used in multi-threading

• Deque is a little more complex. The documentation clearly states that both the .append() and .pop() operations are atomic, meaning that they won’t be interrupted by a different thread. Other methods in deque are not thread-safe

• LifoQueueis designed to be fully thread-safe. All of its methods are safe to use in a threaded environment. It also adds optional time-outs to its operations which can frequently be a must-have feature in threaded programs

Multi-Processing

Multi-Threading vs Multi-Processing

threading: https://nyu-cds.github.io/python-concurrency/03-threads/#:~:text=The%20optimal%20number%20of%20threads,threads%20to%20run%20at%20once.

Multiprocessing vs. Threading in Python: What you need to know.Timber.io

Multiprocessing vs. Threading in Python: What Every Data Scientist Needs to KnowFloydHub Blog

Multithreading vs Multiprocessing – Difference Between ThemGuru99

Parallel Processing

Implementation

Multi threading

Multi-processing

• https://www.analyticsvidhya.com/blog/2021/04/a-beginners-guide-to-multi-processing-in-python/

Multi-processing and Scheduling using Celery

with Redis

with AWS SQS

Distributed Processing

Expand number of workers and contract as per requirements (traffic on the site)

Reference for Further Reading