Real Life Application II
Last updated
Was this helpful?
Last updated
Was this helpful?
Was this helpful?
Create a topic
List all topics
Delete a topic
Consume messages from a Kafka topic
Produce messages to a Kafka topic
Writing data to ES index (writing in bulk)
Query ES data
Update ES data
Different data types in Redis and their use
Writing data to Redis
Updating Redis data
Elasticsearch (data writer using python to ES)
Flask REST API (for GET request)
We will be creating two search engines:
Book search engine
Movie search engine
Input should be:
where to search: books or movies
Search fields?
For books: author, sold copies, search based on keywords, language, genre, etc.
Dataset for books: Use the JSON file prepared with scrapping
Dataset for movies: Use the attached CSV file
For the detailed (more attributes) and bigger movie dataset (85000): refer to this Kaggle link
We will be creating two indices:
book_dataset
movie_dataset
Write and test query in "dev tools" (within Kibana)
Create Flask API for communication with the application
to be updated....
Elasticsearch (data writer using python to ES)
Kafka reader and ES writer, data pipeline example:
Scrap data of books from Wikipedia and store to JSON file
moving data from JSON file to a Kafka topic
read from Kafka topic and write to designated ES index
Use this ES index for search engine
Kafka functionalities
Kafka to ES writer
Here, we are just bringing together different elements, different applications we have built in the previous sections.
link to be updated....
Writing data to ES index (writing in bulk)
Query ES data
Update ES data
Different data types in Redis and their use
Writing data to Redis
Updating Redis data
Elasticsearch (data writer using python to ES)
Flask REST API (for GET request)
We will be creating two search engines:
Book search engine
Movie search engine
Input should be:
where to search: books or movies
Search fields?
For books: author, sold copies, search based on keywords, language, genre, etc.
Dataset for books: Use the JSON file prepared with scrapping
Dataset for movies: Use the attached CSV file
For the detailed (more attributes) and bigger movie dataset (85000): refer to this Kaggle link
We will be creating two indices:
book_dataset
movie_dataset
Write and test query in "dev tools" (within Kibana)
Create Flask API for communication with the application
to be updated....
Elasticsearch (data writer using python to ES)
Kafka reader and ES writer, data pipeline example:
Scrap data of books from Wikipedia and store to JSON file
moving data from JSON file to a Kafka topic
read from Kafka topic and write to designated ES index
Use this ES index for search engine
Kafka functionalities
Kafka to ES writer
Here, we are just bringing together different elements, different applications we have built in the previous sections.
link to be updated....
Number of results (keep default as 10)
Number of results (keep default as 10)