Search Engine Indexing

Search engines help users find relevant information among billions of web pages.

Definition

A search engine is a system that indexes web pages and allows users to search for content using keywords.

How Search Engines Work

Worked Example

Worked Example: Search Engine Process

Step 1: Crawling
• Web crawlers (spiders/bots) browse the web
• Start from known pages, follow links to discover new pages
• Continuously revisit pages to find updates
• Respect robots.txt files (instructions for crawlers)

Step 2: Indexing
• Analyse page content (text, images, videos)
• Extract keywords and their positions
• Note page metadata (title, description)
• Store in massive searchable database (the index)
• Build relationships between pages (links)

Step 3: Ranking
• When user searches, find matching pages in index
• Apply ranking algorithm (like PageRank)
• Consider hundreds of factors
• Return results in order of relevance

Step 4: Serving Results
• Display search results page
• Show title, URL, snippet of text
• User clicks through to visit page

Card 1 of 250%

Question

Search Engine Indexing

Click or drag to flip

Answer

Building a database of the words found on web pages, each stored with links to the pages that contain it and its position on those pages. A search is answered from this database instead of by scanning the live web.

Click or drag to flip back

Browse all 2 flashcards on Search Engine Indexing

Search Engine Indexing
Building a database of the words found on web pages, each stored with links to the pages that contain it and its position on those pages. A search is answered from this database instead of by scanning the live web.
Search Engine Indexing
Programs called crawlers or spiders visit web pages, moving from page to page by following hyperlinks. For each page they record information such as its text, meta tags and the position of each word, and store it in the search engine's index.

Exam questions on Search Engine Indexing

2 OCR past paper questions (16 marks) with mark schemes

Question 1H446/01 · June 202412 marks
Discuss why search engine indexing and page ranking will be important in achieving this. You should include: what search engine indexing means and what information is collected, what PageRank means and possible factors that can affect a PageRank score, possible ways for the company to improve the PageRank score.
Your answer (optional, saved in this browser)
Show mark scheme
Levels of response question (12 marks) AO1 Search engine indexing: Bots/spiders/crawlers crawl the web finding web pages, navigate from page to page following hyperlinks, record information such as text/metatags/position of words, store in search engine's database. PageRank: Used to find and rank website pages, list results in search engine, higher score appears higher in list, calculated using weighted directed graph (pages are nodes, hyperlinks are edges), damping factor used. AO2 To get more presence: Make effective use of meta tags, H1 tags with suitable headings, increase number/quality of incoming links from other websites, increase outgoing links, keywords in meta/title/header tags. AO3 PageRank is only one algorithm used. Company needs to consider who links to their page, ensure site is malware-free/secure.
Mark yourself: how many marks did you get?
/ 12
Question 2H446/01 · June 20214 marks
Describe how a website is indexed by a search engine.
Your answer (optional, saved in this browser)
Show mark scheme
Web crawler/spider visits site (from existing list or following links) Records information such as text/metatags/position of words Storing them in an index Follows links to other sites Robots.txt file can be used to instruct web crawlers (4 marks - 1 per bullet)
Mark yourself: how many marks did you get?
/ 4

How Search Engines Work

Exam questions on Search Engine Indexing

Past paper questions