Extracting schema.org microdata using Scrapy selectors and XPath

We have released an lxml-based version of this code as an open-source library called extruct. The Source code is on Github, and the package is available on PyPI. Enjoy!

Web pages are full of data, that is what web scraping is mostly about. But often you want more than data, you want meaning. Microdata markup embedded in HTML source helps machines understand what the pages are about: contact information, product reviews, events, etc.

Web authors have several ways to add metadata to their web pages: HTML "meta" tags, microformats, social media meta tags (Facebook's Open Graph protocol, Twitter Cards).

Applying Schema.org

Then there is Schema.org, an initiative from big players in the search space (Yahoo!, Google, Bing, and Yandex) to “create and support a common set of schemas for structured data markup on web pages.”

Here are some stats from the Web Data Commons project on the use of microdata in the Common Crawl web corpus:

In summary, we found structured data within 585 million HTML pages out of the 2.24 billion pages contained in the crawl (26%).

Let's focus on Schema.org syntax for the rest of this article. The markup looks like this (example from schema.org "Getting started" page):

<div itemscope itemtype ="http://schema.org/Movie">
<h1 itemprop="name">Avatar</h1>
   <span>Director: <span itemprop="director">James Cameron</span> (born August 16, 1954)</span>
   <span itemprop="genre">Science fiction</span>
   <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a>
</div>

Let's assume you want to extract this microdata and get the movie item from the snippet above.

Using Scrapy Selector let's first loop on elements with an itemscope attribute (this represents a container for an item's properties) using .//*[@itemscope], and for each item, get all properties, i.e. elements that have an itemprop attribute:

>>> from scrapy.selector import Selector
>>> selector = Selector(text="""
...


<div itemscope itemtype ="
http://schema.org/Movie
">
...


<h1 itemprop="name">Avatar</h1>


...   <span>Director: <span itemprop="director">James Cameron</span> (born August 16, 1954)</span>
...   <span itemprop="genre">Science fiction</span>
...  <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a>
... </div>


""", type="html")
>>> selector.xpath('.//*[@itemscope]')
[<Selector xpath='.//*[@itemscope]' data=u' <div itemscope itemtype="http://schema.o'>] >>>

Let's print out more interesting stuff, like the item's itemtype and the properties' values (the text representation of their HTML element):

>>> for item in selector.xpath('.//*[@itemscope]'): ... print item.xpath('@itemtype').extract() ... for property in item.xpath('.//*[@itemprop]'): ... print property.xpath('@itemprop').extract(), ... print property.xpath('string(.)').extract() ... [u'http://schema.org/Movie'] [u'name'] [u'Avatar'] [u'director'] [u'James Cameron'] [u'genre'] [u'Science fiction'] [u'trailer'] [u'Trailer'] >>>

Hm. The value for the trailer isn't that interesting. We should have selected the href of <a href="../movies/avatar-theatrical-trailer.html">Trailer</a>.

For the following extended example markup, taken from Wikipedia,

<div itemscope itemtype="http://schema.org/Movie"> <h1 itemprop="name">Avatar</h1> <div itemprop="director" itemscope itemtype="http://schema.org/Person"> Director: <span itemprop="name">James Cameron</span> (born <time itemprop="birthDate" datetime="1954-08-16">August 16, 1954</time>) </div> <span itemprop="genre">Science fiction</span> <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a> </div>

we see it's not always href attributes that are interesting, but datetime perhaps, or src for images, or content for meta element like ``... any attribute really.

Therefore we need to get itemprop text content and elements attributes (we use the @* XPath expression for that).

Let's do that on the 2nd HTML snippet:

>>> selector = Selector(text=""" ... <div itemscope itemtype=" http://schema.org/Movie "> ... <h1 itemprop="name">Avatar</h1> ... <div itemprop="director" itemscope itemtype=" http://schema.org/Person "> ... Director: <span itemprop="name">James Cameron</span> ... (born <time itemprop="birthDate" datetime="1954-08-16">August 16, 1954</time>) ... </div> ... <span itemprop="genre">Science fiction</span> ... <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a> ... </div> """, type="html") >>> for item in selector.xpath('.//*[@itemscope]'): ... print item.xpath('@itemtype').extract() ... for property in item.xpath('.//*[@itemprop]'): ... print property.xpath('@itemprop').extract(), ... print property.xpath('string(.)').extract(), ... print property.xpath('@*').extract() ... [u'http://schema.org/Movie'] [u'name'] [u'Avatar'] [u'name'] [u'director'] [u'n Director: James Cameron n(born August 16, 1954)n '] [u'director', u'', u'http://schema.org/Person'] [u'name'] [u'James Cameron'] [u'name'] [u'birthDate'] [u'August 16, 1954'] [u'birthDate', u'1954-08-16'] [u'genre'] [u'Science fiction'] [u'genre'] [u'trailer'] [u'Trailer'] [u'../movies/avatar-theatrical-trailer.html', u'trailer'] [u'http://schema.org/Person'] [u'name'] [u'James Cameron'] [u'name'] [u'birthDate'] [u'August 16, 1954'] [u'birthDate', u'1954-08-16'] >>>

Wait a minute! We're getting only attribute values, not attribute names (and itemprop attribute twice for that matter, once for @itemprop and once for @*).

To get names of attributes, one cannot apparently use name() on an attribute... BUT you can do name(@*[i]), i being the attribute position:

>>> for item in selector.xpath('.//*[@itemscope]'): ... print "Item:", item.xpath('@itemtype').extract() ... for property in item.xpath('.//*[@itemprop]'): ... print "Property:", ... print property.xpath('@itemprop').extract(), ... print property.xpath('string(.)').extract() ... for position, attribute in enumerate(property.xpath('@*'), start=1): ... print "attribute: name=%s; value=%s" % ( ... property.xpath('name(@*[%d])' % position).extract(), ... attribute.extract()) ... print ... print ... Item: [u'http://schema.org/Movie'] Property: [u'name'] [u'Avatar'] attribute: name=[u'itemprop']; value=name Property: [u'director'] [u'n Director: James Cameron n(born August 16, 1954)n '] attribute: name=[u'itemprop']; value=director attribute: name=[u'itemscope']; value= attribute: name=[u'itemtype']; value=http://schema.org/Person Property: [u'name'] [u'James Cameron'] attribute: name=[u'itemprop']; value=name Property: [u'birthDate'] [u'August 16, 1954'] attribute: name=[u'itemprop']; value=birthDate attribute: name=[u'datetime']; value=1954-08-16 Property: [u'genre'] [u'Science fiction'] attribute: name=[u'itemprop']; value=genre Property: [u'trailer'] [u'Trailer'] attribute: name=[u'href']; value=../movies/avatar-theatrical-trailer.html attribute: name=[u'itemprop']; value=trailer Item: [u'http://schema.org/Person'] Property: [u'name'] [u'James Cameron'] attribute: name=[u'itemprop']; value=name Property: [u'birthDate'] [u'August 16, 1954'] attribute: name=[u'itemprop']; value=birthDate attribute: name=[u'datetime']; value=1954-08-16 >>>

There's still something wrong, right? Duplicate properties.

It's because the 2nd HTML snippet is using embedded items. Indeed, James Cameron is the director of "Avatar", but he's also a Person (yes, he is!). That's why the markup says <div> with an itemscope attribute.

How can we fix that, only selecting properties at the current scope, and leave nested properties when we reach the nested item?

Well, it happens that Scrapy Selectors support some EXSLT extensions, notably the sets operations. Here, we'll use set:difference: we'll select itemprops under the current itemscope, and then exclude those that are themselves, children of another, itemscope, under the current one.

>>> for item in selector.xpath('.//*[@itemscope]'): ... print "Item:", item.xpath('@itemtype').extract() ... for property in item.xpath( ... """set:difference(.//*[@itemprop], ... .//*[@itemscope]//*[@itemprop])"""): ... print "Property:", property.xpath('@itemprop').extract(), ... print property.xpath('string(.)').extract() ... for position, attribute in enumerate(property.xpath('@*'), start=1): ... print "attribute: name=%s; value=%s" % ( ... property.xpath('name(@*[%d])' % position).extract(), ... attribute.extract()) ... print ... print ... Item: [u'http://schema.org/Movie'] Property: [u'name'] [u'Avatar'] attribute: name=[u'itemprop']; value=name Property: [u'director'] [u'n Director: James Cameron n(born August 16, 1954)n '] attribute: name=[u'itemprop']; value=director attribute: name=[u'itemscope']; value= attribute: name=[u'itemtype']; value=http://schema.org/Person Property: [u'genre'] [u'Science fiction'] attribute: name=[u'itemprop']; value=genre Property: [u'trailer'] [u'Trailer'] attribute: name=[u'href']; value=../movies/avatar-theatrical-trailer.html attribute: name=[u'itemprop']; value=trailer Item: [u'http://schema.org/Person'] Property: [u'name'] [u'James Cameron'] attribute: name=[u'itemprop']; value=name Property: [u'birthDate'] [u'August 16, 1954'] attribute: name=[u'itemprop']; value=birthDate attribute: name=[u'datetime']; value=1954-08-16 >>>

But wouldn't it be nice to keep a reference to the Person item as property of the Movie director itemprop?

We'd need to uniquely identify items in the markup, maybe the position of each itemscope element, or its number.

You can use XPath count() function for that, counting other itemscopes that are siblings before (preceding::*[@itemscope]) or ancestors of the current one (ancestor::*[@itemscope]):

>>> for item in selector.xpath('.//*[@itemscope]'): ... print "Item:", item.xpath('@itemtype').extract() ... print "ID:", item.xpath("""count(preceding::*[@itemscope]) ... + count(ancestor::*[@itemscope]) ... + 1""").extract() ... Item: [u'http://schema.org/Movie'] ID: [u'1.0'] Item: [u'http://schema.org/Person'] ID: [u'2.0'] >>>

Here's a cleaned-up example routine, referencing items by their ID when itemprops are also itemscopes:

>>> for item in selector.xpath('.//*[@itemscope]'): ... print "Item:", item.xpath('@itemtype').extract() ... print "ID:", item.xpath("""count(preceding::*[@itemscope]) ... + count(ancestor::*[@itemscope]) ... + 1""").extract() ... Item: [u'http://schema.org/Movie'] ID: [u'1.0'] Item: [u'http://schema.org/Person'] ID: [u'2.0'] >>>

This dict output is not quite what W3C details in the microdata specs, but it's close enough to leave the rest to you as an exercise 😉

You can find a Github Gist of the above code here.