Effective Seed URL Selection and Scope Extension Algorithm for Web Crawler

Zülfü Alanoğlu; Mehmet Akçayol

Makale detayları

Effective Seed URL Selection and Scope Extension Algorithm for Web Crawler

Yazarlar:

Zülfü Alanoğlu | Mehmet Akçayol

Yayıncı:

Marmara University

Ana Başlıklar:

Mühendislik

Dergi:

International Journal of Advances in Engineering and Pure Sciences Vol.35 Issue 1, pp.27-38

Belge Tipi:

Makale

Yayın Tarihi:

3/2023

Belge Dili:

Türkçe

Document ID (Org No.):

2001286640

pISSN:

2636-8277

Anahtar kelimeler:

Özet:

 The web is a huge data source which is rapidly growing and which keeps all kinds of data. Users use search engines to get the data they want from this data source. Search engines obtain these data through web crawlers. Web crawlers retrieve, parse, and index data on all pages they reach by tracking uniform resource locators URL on web pages. The most important issues in the web crawling process are which URLs to start from, and the scope of the crawl. In this study, seed URL selection and scope expansion methods of a general web crawler were presented. In the selection of seed URLs, three different seed URL sets were created based on the daily hours spent by the visitors in 102 different countries, the number of daily page views per visitor, the percentage of traffic from the search, and the total number of affiliate sites, and their performance was analyzed thoroughly. Furthermore, a new search algorithm based on link score was proposed to expand the scope quickly, searches were made, compared, and detailed analyzes were performed using seed URL sets.

 The web is a huge data source which is rapidly growing and which keeps all kinds of data. Users use search engines to get the data they want from this data source. Search engines obtain these data through web crawlers. Web crawlers retrieve, parse, and index data on all pages they reach by tracking uniform resource locators URL on web pages. The most important issues in the web crawling process are which URLs to start from, and the scope of the crawl. In this study, seed URL selection and scope expansion methods of a general web crawler were presented. In the selection of seed URLs, three different seed URL sets were created based on the daily hours spent by the visitors in 102 different countries, the number of daily page views per visitor, the percentage of traffic from the search, and the total number of affiliate sites, and their performance was analyzed thoroughly. Furthermore, a new search algorithm based on link score was proposed to expand the scope quickly, searches were made, compared, and detailed analyzes were performed using seed URL sets.

 The web is a huge data source which is rapidly growing and which keeps all kinds of data. Users use search engines to get the data they want from this data source. Search engines obtain these data through web crawlers. Web crawlers retrieve, parse, and index data on all pages they reach by tracking uniform resource locators URL on web pages. The most important issues in the web crawling process are which URLs to start from, and the scope of the crawl. In this study, seed URL selection and scope expansion methods of a general web crawler were presented. In the selection of seed URLs, three different seed URL sets were created based on the daily hours spent by the visitors in 102 different countries, the number of daily page views per visitor, the percentage of traffic from the search, and the total number of affiliate sites, and their performance was analyzed thoroughly. Furthermore, a new search algorithm based on link score was proposed to expand the scope quickly, searches were made, compared, and detailed analyzes were performed using seed URL sets.

 Web, hızla büyüyen ve her türden verilerin bulunduğu devasa bir veri kaynağıdır. Kullanıcılar bu veri kaynağından istedikleri verileri almak için arama motorlarını kullanırlar. Arama motorları bu verileri web tarayıcıları ile elde ederler. Web tarayıcıları web sayfalarındaki tek düzen kaynak bulucuları URL-Uniform Resource Locator izleyerek ulaştıkları tüm sayfalardaki verileri alır, ayrıştırır ve indekslerler. Web tarama sürecindeki en önemli konular hangi URLlerden başlanacağı ve taramanın kapsamıdır. Bu yazıda kapsamı tüm web olan genel bir tarayıcının tohum URL seçim ve kapsam genişletme yöntemleri sunulmuştur. Tohum URL seçiminde 102 farklı ülkede ziyaretçinin günlük harcadığı saat, ziyaretçi başına günlük sayfa görüntüleme sayısı, aramadan gelen trafiğin yüzdesi ve toplam bağlı site sayısı temel alınarak oluşturulmuş üç farklı tohum URL seti oluşturulup detaylı bir şekilde performansları analiz edilmiştir. Ayrıca kapsamı hızlı bir şekilde genişletmek için link skoruna dayalı yeni bir tarama algoritması önerilmiş, tohum URL setleri kullanılarak taramalar yapılmış, karşılaştırılmış ve detaylı analizleri yapılmıştır.

Makale detayları

Effective Seed URL Selection and Scope Extension Algorithm for Web Crawler

Yazarlar:

Zülfü Alanoğlu | Mehmet Akçayol

Yayıncı:

Marmara University

Ana Başlıklar:

Mühendislik

Dergi:

International Journal of Advances in Engineering and Pure Sciences Vol.35 Issue 1, pp.27-38

Belge Tipi:

Makale

Yayın Tarihi:

3/2023

Belge Dili:

Türkçe

Document ID (Org No.):

2001286640

pISSN:

2636-8277

Web crawler

Seed URL Selection

Scope extension

Link score calculation

Web Tarayıcıları

Tohum URL Seçimi

Kapsam Genişletme

Link Skoru Hesaplama

Sayı

Makale detayları

Yazarlar: Zülfü Alanoğlu | Mehmet Akçayol

Yayıncı: Marmara University

Ana Başlıklar: Mühendislik

Dergi: International Journal of Advances in Engineering and Pure Sciences Vol.35 Issue 1, pp.27-38

Belge Tipi: Makale

Yayın Tarihi: 3/2023

Belge Dili: Türkçe

Document ID (Org No.): 2001286640

pISSN: 2636-8277

Sayı

Yazarlar:

Zülfü Alanoğlu | Mehmet Akçayol

Yayıncı:

Marmara University

Ana Başlıklar:

Mühendislik

Dergi:

International Journal of Advances in Engineering and Pure Sciences Vol.35 Issue 1, pp.27-38

Belge Tipi:

Makale

Yayın Tarihi:

3/2023

Belge Dili:

Türkçe

Document ID (Org No.):

2001286640

pISSN:

2636-8277