selenium_scholar.py

This module provides classes for querying Google Scholar using selenium and parsing returned results. It currently only processes the first results page. It is not a recursive crawler.

class AddArticleTask(result, article)

Bases: object

Task that adds an article to the result

apply(querier)

get_citation_data(querier)

Given an article, retrieves citation link. Note, this requires that you adjusted the settings to tell Google Scholar to actually provide this information, prior to retrieving the article.

class ParseTask(result)

Bases: snowballing.scholar.ScholarArticleParser120726

Task that parsers articles

apply(querier)

handle_article(art)

The parser invokes this callback on each article parsed successfully. In this base class, the callback does nothing.

handle_num_results(num_results)

The parser invokes this callback if it determines the overall number of results, as reported on the parsed results page. The base class implementation does nothing.

class QueryTask(query)

Bases: object

Task that queries the scholar

apply(querier)

class Result(query, html)

Bases: object

Represent a result with articles

set_navigation(driver, name, text)

class ScholarSettingsTask(pages=10, citform=0, new_window=False, collections=1)

Bases: object

This class lets you adjust the Scholar settings for your session.

CITFORM_BIBTEX = 4

CITFORM_ENDNOTE = 3

CITFORM_NONE = 0

CITFORM_REFMAN = 2

CITFORM_REFWORKS = 1

COLLECTIONS_ARTICLES_AND_PATENTS = 1

COLLECTIONS_ARTICLES_ONLY = 0

COLLECTIONS_CASE_LAW = 2

SETTINGS_URL = 'http://scholar.google.com/scholar_settings?hl=en&as_sdt=0,5&sciodt=0,5'

apply(querier)

citform

collections

new_window

per_page_results

class SearchScholarQuery

Bases: snowballing.scholar.ScholarQuery

This version represents the search query parameters the user can configure on the Scholar website, in the advanced search options.

SCHOLAR_QUERY_URL = 'http://scholar.google.com/scholar?'

get_url()

Returns a complete, submittable URL string for this particular query instance. The URL and its arguments will vary depending on the query.

set_author(author)

Sets names that must be on the result’s author list.

set_include_citations(yesorno)

set_include_patents(yesorno)

set_phrase(phrase)

Sets phrase that must be found in the result exactly.

set_pub(pub)

Sets the publication in which the result must be found.

set_scope(title_only)

Sets Boolean indicating whether to search entire article or title only.

set_timeframe(start=None, end=None)

Sets timeframe (in years as integer) in which result must have appeared. It’s fine to specify just start or end, or both.

set_words(words)

Sets words that all must be found in the result.

set_words_none(words)

Sets words of which none must be found in the result.

set_words_some(words)

Sets words of which at least one must be found in result.

class SeleniumScholarQuerier(driver=None)

Bases: object

ScholarQuerier instances can conduct a search on Google Scholar with subsequent parsing of the resulting HTML content. The articles found are collected in the articles member, a list of ScholarArticle instances.

apply_settings(*args, **kwargs)

Applies settings

continue_tasks()

get_response(url, log_msg=None, err_msg=None, condition=None)

Helper method, sends HTTP request and returns response payload.

send_query(query)

Initiates a search query (a ScholarQuery instance)

class URLQuery(url, start=None)

Bases: snowballing.scholar.ScholarQuery

Represent a Google Scholar query using a generic query We use it to navigate on the citations

get_url()

Returns a complete, submittable URL string for this particular query instance. The URL and its arguments will vary depending on the query.

check_captcha(driver, condition)

Check if it expects a captcha

click(parent, selector)

Click on selector

get_scholar_url(work)

Get scholar url from work

wait_for(driver, condition, delay=5)

Wait for an element appear during :delay seconds