Do you want to find the best keywords for your content? Do you need to do a competitive analysis on your blog or website? If so, Screaming Frog is an invaluable tool. This article will walk through how to use it and what information can be found with this tool.

The screaming frog extractor allows users to select which fields they would like extracted from a webpage and then save these as CSV files. These can then be opened in Excel and filtered into specific ranges of data types such as dates, numbers, words. For example, if you wanted all the URLs on a page that contain “keyword” in them but only pages with more than 5 matches per URL, this could easily be done by filtering out any URLs that meet those requirements. Keep reading to learn more.

How to Use Screaming Frog Custom Extraction

  1. Go to the Custom Extraction Setting

    screaming frog custom extraction

    In ScreamingFrog, go to Configuration > Custom > Extraction.

  2. +Add the extraction rules

    custom extraction

    Next, you will need to +Add and set up your extraction rules.

     

    Add a title, select if you need CSSPath, XPath, or Regex, then add your search function. If you aren’t sure which selector or function you need, look at the examples below.

Example:

Here is an example of how you would scrape for a Facebook Pixel ID

Results, as you can see, one of my pages are missing a Facebook Pixel:

Here are more prebuilt syntaxes to help you crawl and search for content:

Basic Syntax for XPath Web Scraping

SYNTAXFUNCTION
//Search anywhere in the document
/Search within the root
@Select a specific attribute of an element
*The wildcard is used to select any element.
[ ]Find a specific element.
.Specifies the current element
..Specifies the parent element

XPath functions

OPERATORFUNCTION
starts-with(x,y)Checks if x starts with y
contains(x,y)Checks if x contains y
last()Finds the last item in a set
count(XPath)Counts occurrences of the XPath extraction

How to Extract Common HTML Elements

XPATHOUTPUT
//h1Extract all H1 tags
//h3[1]Extract the first H3 tag
//h3[2]Extract the second H3 tag
//div/pExtract any <p> contained within a <div>
//div[@class=’author’]Extract any <div> with class “author”
//p[@class=’bio’]Extract any <p> with class “bio”
//*[@class=’bio’]Extract any element with class “bio”
//ul/li[last()]Extract the last <li> in a <ul>
//ol[@class=’cat’]/li[1]Extract the first <li> in a <ol> with class “cat”
count(//h2)Count the number of H2’s (set extraction filter to “Function Value”)
//a[contains(.,’click here’)]Extract any link with anchor text containing “click here.”
//a[starts-with(@title,’Written by’)]Extract any link with a title starting with “Written by.”

How to Extract Common HTML Attributes

XPATHOUTPUT
//@hrefExtract all links
//a[starts-with(@href,’mailto’)]/@hrefExtract link that starts with “mailto” (email address)
//img/@srcExtract all image source URLs
//img[contains(@class,’aligncenter’)]/@srcExtract all image source URLs for images with the class name containing “aligncenter.”
//link[@rel=’alternate’]Extract elements with the rel attribute set to “alternate.”
//@hreflangExtract all hreflang values

Extract Meta Tags (use inner HTML)

XPATHOUTPUT
//meta[@property=’article:published_time’]/@contentExtract the article publish date (commonly-found meta tag on WordPress websites)

Extract Open Graph

XPATHOUTPUT
//meta[@property=’og:type’]/@contentExtract the Open Graph type object
//meta[@property=’og:image’]/@contentExtract the Open Graph featured image URL
//meta[@property=’og:updated_time’]/@contentExtract the Open Graph updated time

Extract Twitter Cards

XPATHOUTPUT
//meta[@name=’twitter:card’]/@contentExtract the Twitter Card type
//meta[@name=’twitter:title’]/@contentExtract the Twitter Card title
//meta[@name=’twitter:site’]/@contentExtract the Twitter Card site object (Twitter handle)

Extract Schema Types

XPATHOUTPUT
//*[@itemtype]/@itemtypeExtract all of the types of schema markup on a page

Extract Breadcrumb Schema

XPATHOUTPUT
//*[contains(@itemtype,’BreadcrumbList’)]/*[@itemprop]/a/@hrefExtract all breadcrumb links
//*[contains(@itemtype,’BreadcrumbList’)]/*[@itemprop][1]/a/@hrefExtract the first breadcrumb link
//*[contains(@itemtype,’BreadcrumbList’)]/*[@itemprop]Extract breadcrumb names (set extraction filter to “Extract Text”)
count(//*[contains(@itemtype,’BreadcrumbList’)]/*[@itemprop])Count the number of breadcrumb list items (set extraction filter to “Function Value”)

Extract Product Schema

XPATHOUTPUT
//*[@itemprop=’name’]/@contentExtract product name
//*[@itemprop=’description’]/@contentExtract product description
//*[@itemprop=’price’]/@contentExtract product price
//*[@itemprop=’priceCurrency’]/@contentExtract product currency
//*[@itemprop=’availability’]/@hrefExtract product availability
//*[@itemprop=’sku’]/@contentExtract product SKU

Extract Review Schema

XPATHOUTPUT
//*[@itemprop=’reviewCount’]Extract review count
//*[@itemprop=’ratingValue’]Extract rating value
//*[@itemprop=’bestRating’]Extract best review rating
//*[@itemprop=’review’]/*[@itemprop=’name’]Extract review name
//*[@itemprop=’review’]/*[@itemprop=’author’]Extract review author
//*[@itemprop=’review’]/*[@itemprop=’datePublished’]/@contentExtract the publish date of reviews
//*[@itemprop=’review’]/*[@itemprop=’reviewBody’]Extract the body content of reviews

Extract Local Business & Organization Schema

XPATHOUTPUT
//*[contains(@itemtype,’Organization’)]/*[@itemprop=’name’]Extract the organization’s name
//*[@itemprop=’address’]/*[@itemprop=’streetAddress’]Extract the street address
//*[@itemprop=’address’]/*[@itemprop=’addressLocality’]Extract the address locality
//*[@itemprop=’address’]/*[@itemprop=’addressRegion’]Extract the address region
//*[@itemprop=’telephone’]Extract the telephone number
//*[@itemprop=’sameAs’]/@hrefExtract the “sameAs” links

Extract Article Schema

XPATHOUTPUT
//*[contains(@itemtype,’Article’)]/*[@itemprop=’headline’]Extract the article headline
//*[@itemprop=’author’]/*[@itemprop=’name’]/@contentExtract author name
//*[@itemprop=’publisher’]/*[@itemprop=’name’]/@contentExtract publisher name
//*[@itemprop=’datePublished’]/@contentExtract publish date
//*[@itemprop=’dateModified’]/@contentExtract modified date

Custom Extraction with Regex

Wildcards

SYNTAXFUNCTION
.Match any 1 character
*Match preceding character 0 or more times
?Match preceding character 0 or 1 time
+Match preceding character 1 or more times
|OR

Anchors

SYNTAXFUNCTION
^The string begins with the succeeding character.
$The string ends with the preceding character.

Groups

SYNTAXFUNCTION
( )Match enclosed characters in exact order
[ ]Match enclosed characters in any order
Match any characters within the specified range

Escape

SYNTAXFUNCTION
\Treat character literally, not as regex.

Regex Custom Extraction

REGEXOUTPUT
[“‘](UA-.*?)[“‘]Extract the Google Analytics tracking ID
[“‘](AW-.*?)[“‘]Extract the Google Ads conversion ID and/or remarketing tag
[“‘](GTM-.*?)[“‘]Extract the Google Tag Manager and/or Google Optimize ID
fbq\([“‘]init[“‘], [“‘](.*?)[“‘]Extract the Facebook Pixel ID
\{ti:[“‘](.*?)[“‘]\}Extract the Bing Ads UET tag
adroll_adv_id = [“‘](.*?)[“‘]Extract the AdRoll Advertiser ID
adroll_pix_id = [“‘](.*?)[“‘]Extract the AdRoll Pixel ID

Extract All Schema Markup and Schema Types

REGEXOUTPUT
[“‘]application/ld\+json[“‘]>(.*?)</script>Extract all of the JSON-LD schema markup
[“‘]@type[“‘]: *[“‘](.*?)[“‘]Extract all of the types of JSON-LD schema markup on a page

Extract Breadcrumb Schema

REGEXOUTPUT
[“‘]item[“‘]: *\{[“‘]@id[“‘]: *[“‘](.*?)[“‘]Extract breadcrumb links
[“‘]item[“‘]: *\{[“‘]@id[“‘]: *[“‘].*?[“‘], *[“‘]name[“‘]: *[“‘](.*?)[“‘]Extract breadcrumb names

Extract Product Schema

REGEXOUTPUT
[“‘]@type[“‘]: *[“‘]Product[“‘].*?[“‘]name[“‘]: *[“‘](.*?)[“‘]Extract product name
[“‘]@type[“‘]: *[“‘]Product[“‘].*?[“‘]description[“‘]: *[“‘](.*?)[“‘]Extract product description
[“‘]@type[“‘]: *[“‘]Product[“‘].*?[“‘]price[“‘]: *[“‘](.*?)[“‘]Extract product price
[“‘]@type[“‘]: *[“‘]Product[“‘].*?[“‘]priceCurrency[“‘]: *[“‘](.*?)[“‘]Extract product currency
[“‘]@type[“‘]: *[“‘]Product[“‘].*?[“‘]availability[“‘]: *[“‘](.*?)[“‘]Extract product availability
[“‘]@type[“‘]: *[“‘]Product[“‘].*?[“‘]sku[“‘]: *[“‘](.*?)[“‘]Extract product SKU

Extract Review Schema

REGEXOUTPUT
[“‘]reviewCount[“‘]: *[“‘](.*?)[“‘]Extract review count
[“‘]ratingValue[“‘]: *[“‘](.*?)[“‘]Extract rating value
[“‘]bestRating[“‘]: *[“‘](.*?)[“‘]Extract best rating

Extract Local Business & Organization Schema

REGEXOUTPUT
[“‘]@type[“‘]: *[“‘]Organization[“‘].*?[“‘]name[“‘]: *[“‘](.*?)[“‘]Extract organization name
[“‘]streetAddress[“‘]: *[“‘](.*?)[“‘]Extract the street address
[“‘]addressLocality[“‘]: *[“‘](.*?)[“‘]Extract the address locality
[“‘]addressRegion[“‘]: *[“‘](.*?)[“‘]Extract the address region
[“‘]telephone[“‘]: *[“‘](.*?)[“‘]Extract the telephone number
[“‘]sameAs[“‘]: *\[(.*?)\]Extract the “sameAs” links

Extract Article or BlogPosting Schema

REGEXOUTPUT
[“‘]headline[“‘]: *[“‘](.*?)[“‘]Extract article headline
[“‘]author[“‘].*?[“‘]name[“‘]: *[“‘](.*?)[“‘]Extract author name
[“‘]publisher[“‘].*?[“‘]name[“‘]: *[“‘](.*?)[“‘]Extract publisher name
[“‘]datePublished[“‘]: *[“‘](.*?)[“‘]Extract publish date
[“‘]dateModified[“‘]: *[“‘](.*?)[“‘]Extract modified date

The possibilities are endless; please let me know if you want any extractions added to this Screaming Frog XPath cheat sheet.

Screaming Frog is a valuable tool for anyone who wants to find the best keywords, do a competitive analysis on their blog or website, or perform any other task that requires crawling and analyzing your site. It’s easy to use and can provide you with a lot of information about your own content and what potential competitors are doing online. If you have questions about how it works or want more in-depth guidance on using the software, please email me. I will be happy to help!