scholar.py

This module provides classes for querying Google Scholar and parsing returned results. It currently only processes the first results page. It is not a recursive crawler.

class ClusterScholarQuery(cluster=None)

Bases: snowballing.scholar.ScholarQuery

This version just pulls up an article cluster whose ID we already know about.

SCHOLAR_CLUSTER_URL = 'http://scholar.google.com/scholar?cluster=%(cluster)s&num=%(num)s'

get_url()

Returns a complete, submittable URL string for this particular query instance. The URL and its arguments will vary depending on the query.

set_cluster(cluster)

Sets search to a Google Scholar results cluster ID.

exception Error

Bases: Exception

Base class for any Scholar error.

exception FormatError

Bases: snowballing.scholar.Error

A query argument or setting was formatted incorrectly.

exception QueryArgumentError

Bases: snowballing.scholar.Error

A query did not have a suitable set of arguments.

class ScholarArticle

Bases: object

A class representing articles listed on Google Scholar. The class provides basic dictionary-like behavior.

as_citation()

Reports the article in a standard citation format. This works only if you have configured the querier to retrieve a particular citation export format. (See ScholarSettings.)

as_csv(header=False, sep='|')

as_txt()

set_citation_data(citation_data)

class ScholarArticleParser(site=None)

Bases: object

ScholarArticleParser can parse HTML document strings obtained from Google Scholar. This is a base class; concrete implementations adapting to tweaks made by Google over time follow below.

handle_article(art)

The parser invokes this callback on each article parsed successfully. In this base class, the callback does nothing.

handle_num_results(num_results)

The parser invokes this callback if it determines the overall number of results, as reported on the parsed results page. The base class implementation does nothing.

parse(html)

This method initiates parsing of HTML content, cleans resulting content as needed, and notifies the parser instance of resulting instances via the handle_article callback.

class ScholarArticleParser120201(site=None)

Bases: snowballing.scholar.ScholarArticleParser

This class reflects update to the Scholar results page layout that Google recently.

class ScholarArticleParser120726(site=None)

Bases: snowballing.scholar.ScholarArticleParser

This class reflects update to the Scholar results page layout that Google made 07/26/12.

class ScholarConf

Bases: object

Helper class for global settings.

COOKIE_JAR_FILE = None

LOG_LEVEL = 3

MAX_PAGE_RESULTS = 20

SCHOLAR_SITE = 'http://scholar.google.com'

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0'

VERSION = '2.10'

class ScholarQuerier

Bases: object

ScholarQuerier instances can conduct a search on Google Scholar with subsequent parsing of the resulting HTML content. The articles found are collected in the articles member, a list of ScholarArticle instances.

GET_SETTINGS_URL = 'http://scholar.google.com/scholar_settings?sciifh=1&hl=en&as_sdt=0,5'

class Parser(querier)

Bases: snowballing.scholar.ScholarArticleParser120726

handle_article(art)

The parser invokes this callback on each article parsed successfully. In this base class, the callback does nothing.

handle_num_results(num_results)

The parser invokes this callback if it determines the overall number of results, as reported on the parsed results page. The base class implementation does nothing.

SET_SETTINGS_URL = 'http://scholar.google.com/scholar_setprefs?q=&scisig=%(scisig)s&inststart=0&as_sdt=1,5&as_sdtp=&num=%(num)s&scis1=%(scis)s%(scisf)s&hl=en&lang=all&instq=&inst=569367360547434339&save='

add_article(art)

apply_settings(settings)

Applies settings as provided by a ScholarSettings instance.

clear_articles()

Clears any existing articles stored from previous queries.

get_citation_data(article)

Given an article, retrieves citation link. Note, this requires that you adjusted the settings to tell Google Scholar to actually provide this information, prior to retrieving the article.

parse(html)

This method allows parsing of provided HTML content.

save_cookies()

This stores the latest cookies we’re using to disk, for reuse in a later session.

send_query(query)

This method initiates a search query (a ScholarQuery instance) with subsequent parsing of the response.

class ScholarQuery

Bases: object

The base class for any kind of results query we send to Scholar.

get_url()

Returns a complete, submittable URL string for this particular query instance. The URL and its arguments will vary depending on the query.

set_num_page_results(num_page_results)

class ScholarSettings

Bases: object

This class lets you adjust the Scholar settings for your session. It’s intended to mirror the features tunable in the Scholar Settings pane, but right now it’s a bit basic.

CITFORM_BIBTEX = 4

CITFORM_ENDNOTE = 3

CITFORM_NONE = 0

CITFORM_REFMAN = 2

CITFORM_REFWORKS = 1

is_configured()

set_citation_format(citform)

set_per_page_results(per_page_results)

class ScholarUtils

Bases: object

A wrapper for various utensils that come in handy.

LOG_LEVELS = {'debug': 4, 'error': 1, 'info': 3, 'warn': 2}

static ensure_int(arg, msg=None)

static log(level, msg)

class SearchScholarQuery

Bases: snowballing.scholar.ScholarQuery

This version represents the search query parameters the user can configure on the Scholar website, in the advanced search options.

SCHOLAR_QUERY_URL = 'http://scholar.google.com/scholar?as_q=%(words)s&as_epq=%(phrase)s&as_oq=%(words_some)s&as_eq=%(words_none)s&as_occt=%(scope)s&as_sauthors=%(authors)s&as_publication=%(pub)s&as_ylo=%(ylo)s&as_yhi=%(yhi)s&as_sdt=%(patents)s%%2C5&as_vis=%(citations)s&btnG=&hl=en&num=%(num)s'

get_url()

Returns a complete, submittable URL string for this particular query instance. The URL and its arguments will vary depending on the query.

set_author(author)

Sets names that must be on the result’s author list.

set_include_citations(yesorno)

set_include_patents(yesorno)

set_phrase(phrase)

Sets phrase that must be found in the result exactly.

set_pub(pub)

Sets the publication in which the result must be found.

set_scope(title_only)

Sets Boolean indicating whether to search entire article or title only.

set_timeframe(start=None, end=None)

Sets timeframe (in years as integer) in which result must have appeared. It’s fine to specify just start or end, or both.

set_words(words)

Sets words that all must be found in the result.

set_words_none(words)

Sets words of which none must be found in the result.

set_words_some(words)

Sets words of which at least one must be found in result.

citation_export(querier)

csv(querier, header=False, sep='|')

encode(s)

main()

txt(querier, with_globals)