Scrape a catalogue page

Opens a catalogue page, reads the title, price and availability of every tile, and writes them to Excel. The pattern here — one Get Elements per column, joined on position — is the one that survives contact with real pages.

The workflow

StartWebBrowser("https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "newBrowserSession")
WaitForElement("ol.row li:nth-child(1) article.product_pod", state="visible", timeout=30000)

links = GetHtmlElements("article.product_pod h3 a")
prices = GetHtmlElements("article.product_pod p.price_color")
availability = GetHtmlElements("article.product_pod p.instock")

Assert("{{ len(links) == len(prices) }}")
WriteLine("{{ len(links) }} titles on the page")

CreateTable("catalogue", schema={"columns": [{"name": "title", "isKey": True}, {"name": "price"}, {"name": "availability"}, {"name": "url"}]})

for index, link in links:  # _name="Collect one tile" _description="Columns come from three parallel element arrays joined on the tile's position; a length mismatch would already have failed the Assert above"
    TableUpsertRow("catalogue", {"title": "{{ link.title }}", "price": "{{ prices[index].textContent }}", "availability": "{{ trim(availability[index].textContent) }}", "url": "{{ link.href }}"})

export = TableExport("catalogue", "C:/Temp/rinkt-examples/catalogue.xlsx", format="xlsx", sheetName="Mystery")
WriteLine("Wrote {{ export.rows }} rows to {{ export.filePath }}")

The run

Two and a half seconds, 20 rows:

Activity completed [Export Table to File] -> {"filePath": "C:/Temp/rinkt-examples/catalogue.xlsx", "format": "xlsx", "rows": 20}

title	price	availability	url
Sharp Objects	£47.82	In stock	https://books.toscrape.com/catalogue/sharp-objects_997/index.html
In a Dark, Dark Wood	£19.63	In stock	https://books.toscrape.com/catalogue/in-a-dark-dark-wood_963/index.html
The Past Never Ends	£56.50	In stock	https://books.toscrape.com/catalogue/the-past-never-ends_942/index.html

What an element actually is

Get Elements returns objects, not strings. Each one carries:

src  boundingRect  tagName  id  href  title  value
outerHTML  className  textContent  innerHTML  name

Two things follow, and both cost a run to find out:

innerText is not there. It resolves to nothing. The text of an element is textContent.

textContent is what the page shows, truncation included. The tile in this catalogue displays In a Dark, Dark ..., and that is exactly what textContent returns. The full string is in the title attribute, which is why the workflow reads link.title for the name and prices[index].textContent for the price. Check both before choosing.

What it got wrong first

The wait was written against the tiles themselves:

WaitForElement("article.product_pod", state="visible")   # wrong

strict mode violation: locator('article.product_pod') resolved to 20 elements

A wait targets exactly one element. Twenty matches is an error, not a wait that settles on the first. Point it at something singular — here, the first tile inside the list — ol.row li:nth-child(1) article.product_pod. Get Elements on the same page is fine with twenty, because a plural result is what it is for.

The failure is worth reading for a second reason: the error names every element it matched, and the platform's failure envelope records selectorMatchCount: 20, re-queried at the moment it broke. You are told how ambiguous the selector was, not merely that it was.

Worth taking from it

for index, item in collection: gives the loop an index as well as an iterator, which is what joins parallel arrays: prices[index] next to link.

Assert what the join assumes. Assert("{{ len(links) == len(prices) }}") fails the run at the point the assumption breaks, rather than producing a spreadsheet whose price column is off by one from the row the page it came from. The designer flags the input as expecting a boolean and getting a string — an expression is a string until it is evaluated — and corrects the type for the run.