> For the complete documentation index, see [llms.txt](https://ankit-apdc.gitbook.io/system-design/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ankit-apdc.gitbook.io/system-design/building-blocks/database/implementation/elasticsearch.md).

# Elasticsearch

Properties

Keywords

* Document, Index, partition, shards

Installation

Dev-tools

Discover

Create index, field mapping

Loading Data to ES using python

Query data using dev-tools

Query data using python (Create a search engine using python)

* BM25 algorithms for searching
* Lucene Software
* Query Functionalities available - radial search, weighted search

### Update data using python

List indices using python

from elasticsearch import Elasticsearch

es = Elasticsearch(\[{'host': '', 'port': 9200}])

indices=es.indices.get\_alias().keys()

## Mapping or Schema in ES

## Querying in ES

term match

must vs should term search

exact match

radial search

Weighted search on different fields

## Lucene Search Engine

## Scoring in ES

* The scoring function is a mathematical expression for finding a value for the relative importance of different documents
* Scoring is very much is based on the TF-IDF, used in the document/information retrieval  

$$
score (q,d) = queryNorm(q) \* coord(q,d) \* idf(t)2 \* t.getBoost() \* norm(t,d) (t in q)
$$

![](/files/-MgLedXQBeSUo8JvBY9w)

* `score (q,d)`:  a relevance score of a document `d` for query `q`
* `queryNorm (q)`: query normalization factor
  * queryNorm = 1 / √sumOfSquaredWeights
  * `sumOfSquaredWeights` is computed by adding together the IDF of each term in the query, squared
  * QNF is the ratio that aims to make the results of different queries comparable. It is calculated at the beginning of each query using the above formula
* `coord (q,d):` query coordination factor
  * term score \* number of matching terms / total number of terms in the query. 
  * In the case of a multi-term query, the coordination factor rewards the documents that contain a higher number of terms of that query. 
  * The more query terms appear in the document the more relevance it might have.
  * For simplicity, let’s say you have a query with three terms:  “nice,” “red,” and “carpet,” each with a 1.5 score.
  * So, for example, the document that matches “nice red” will have 3.0 \* 2 / 3 = 2.0 score. Of course, the documents that contain all three terms will be much more relevant than the document that contains just two of them.
* `tf (t in d)`*:* term frequency of the term `t` in document `d` 
  * `=`tf(t in d) = √frequency
  * Number of times each term appears in the document
* `idf (t)`: inverse document frequency for term `t` 
  * idf = 1 + ln(numDocs/(docFreq + 1))
  * Inverse document frequency (IDF) assigns low weight/relevance to terms that appear frequently in all of the documents in the index
* `t.getBoost()`:  the boost applied to the query 
* `norm(t,d)` –  the field-length norm
  * norm = 1/sqrt(numFieldTerms)
  * The value of this parameter depends on the document field length in which a match with the query was found

![](/files/-MgLebZEW698cGK4Ine1)

![](/files/-MgLeb0I_gFl42egANjA)

#### Querying same index twice can return documents in different order

<https://www.elastic.co/guide/en/elasticsearch/reference/current/consistent-scoring.html>

## Updating a Document in ES

## Reference for Further Reading

* [qbox scoring and relevancy](https://qbox.io/blog/practical-guide-elasticsearch-scoring-relevancy)
* <https://coralogix.com/blog/42-elasticsearch-query-examples-hands-on-tutorial/>