stafford williams

bags: Farfetch pagination loop, CI versioning, log flushing

Mon, 15 Feb 2021 00:00:00 GMT

Follow-up fixes after the logging release, plus build versioning in CI.

Changed

CI passes build version into the Docker image

Fixed

Farfetch infinite loop on empty pagination
FWRD USD price edge cases
Logs lost on exit

bags: Serilog to Postgres, GitHub Actions CI, FWRD USD prices

Sun, 14 Feb 2021 00:00:00 GMT

Production readiness: structured logs stored in the database and a CI pipeline on every push.

Added

Structured Serilog logging to PostgreSQL
GitHub Actions build and test pipeline
Version shown in console output

Changed

Parse errors keep more context

Fixed

FWRD prices shown in USD

bags: Failed FWRD parses saved for debugging

Fri, 29 Jan 2021 00:00:00 GMT

A small debugging aid that keeps the HTML of any FWRD product that fails to parse.

Added

Failed FWRD product HTML saved to payload.txt

bags: Edge-pixel image check limited to LegitBags

Sat, 07 Nov 2020 00:00:00 GMT

The image quality check was too aggressive for some feeds, so it now only runs where clean backgrounds are expected.

Changed

Edge-pixel image check only applies to the LegitBags edit

bags: Prada and Michael Kors, split image commands

Sun, 01 Nov 2020 00:00:00 GMT

Prada and Michael Kors were added, and the image commands were split up so each does one job.

Added

Michael Kors via its JSON API
Bad image blocking
Prada bags with highest-resolution srcset images

Changed

Metadata, Twitter image and S3 upload split into separate commands

bags: Four new retailers, edits for multiple feeds

Wed, 07 Oct 2020 00:00:00 GMT

The new category architecture paid off with four more retailers, and edits let one database drive several themed feeds.

Added

The Iconic dresses
Edits for multiple curated feeds
Per-edit JSON export
Tory Burch, Ferragamo and Rebecca Minkoff
TestMetadataCommand for a single product URL

Changed

More FWRD parsing moved into HtmlHelpers
Better FWRD dresses handling
Shared HtmlHelpers for HTML parsing

Fixed

Product key collisions across retailers

bags: Polymorphic ProductCategory refactor, TestAll command

Fri, 02 Oct 2020 00:00:00 GMT

Retailer-specific behaviour moved out of scattered switch statements and into per-retailer category classes.

Added

TestAllCommand checks every category

Changed

Category name in the nothing-to-tweet log
Category enum replaced by per-retailer ProductCategory classes

Fixed

Outnet shoes URL after the clearance path changed

bags: Tweet hashtags and web gallery component

Mon, 21 Sep 2020 00:00:00 GMT

<h2 id="v0-12"><time datetime="2020-09-21">Sep 21</time> Tweet hashtags and web gallery component <span>v0.12</span></h2>

I integrated the hashtag generator into the tweet workflow. Tweets now include hashtags based on the brand name, making them more discoverable on Twitter. I also fixed a bug in the generator where spaces weren't being stripped properly.

The scraper now extracts product tags from detail pages and stores them in a new database field. I refactored the GetImagesCommand into a more general GetMetadataCommand that returns both images and tags as ProductMetadata. This metadata can be used to add relevant hashtags beyond just the brand name.

I implemented a full reskin of the website view of the feed and most importantly it now works on mobile. To achieve the design I had to override the styling on strikethrough text which is not available natively. The approach I've used to have orange strikethrough is detailed here.

When using material-ui/makeStyles in React (JSS?) you can't set content: '' as it produces an invalid style. Instead wrap in quotes: content: "''".

I built a new Gallery component with a proper image viewer and placeholder loading states. The gallery uses a flipOnIndex utility to handle image navigation, which I test-drove with unit tests. The component uses JSX and has been refactored for better separation of concerns with ProductDetail, ProductImage, and Placeholder components.

bags: Brand hashtags, xUnit test project, ScrapeUrl debugging

Sat, 19 Sep 2020 00:00:00 GMT

Tweets gained brand hashtags, and the project picked up its first tests and a page-source debugging tool.

Added

ScrapeUrlCommand logs a page's source for debugging
Brand hashtag generator
xUnit and Shouldly test project

Changed

Dropped the bundled ChromeDriver package
Solution file for building and testing together

bags: Responsive product grid replaces fullpage scrolling

Sat, 05 Sep 2020 00:00:00 GMT

The web frontend swapped one-product-per-screen scrolling for a grid that's much faster to browse.

Changed

Responsive 2/3/4-column grid replaces fullpage scrolling

bags: Farfetch handbags and shoes, embedded JSON parsing, Outnet heels

Tue, 18 Aug 2020 00:00:00 GMT

Farfetch made four retailers, and scraping moved towards parsing the JSON sites embed in their pages.

Added

Outnet heels category
Farfetch handbags and shoes
GetImageSourcesFromPage for multi-image detail pages

Changed

Category passed into ToEntity and image logic
Outnet parsed from embedded page JSON

Fixed

Duplicate products across category listings

bags: The Outnet coats, pagination, AngleSharp over Selenium

Thu, 13 Aug 2020 00:00:00 GMT

The Outnet joined as a third retailer, and a timing comparison settled on AngleSharp for parsing it.

Added

Outnet scraper pagination
The Outnet coats as a third retailer
Headless Chrome option

Changed

Outnet parsing switched to AngleSharp
Timed Selenium vs AngleSharp extraction comparison

bags: Paginated JSON export, React frontend, LocalStack

Thu, 06 Aug 2020 00:00:00 GMT

Tweeted products got a home on the web: a JSON export feeding a static React site served from S3.

Added

Fullpage scrolling product display with image galleries
Price, savings and retailer link on each product
Frontend S3 bucket config from config.json
React web frontend hosted on S3
Combined ScrapeAndTweet command
LocalStack docker-compose for local S3
Paginated JSON export of products

Changed

Export uploads JSON straight to S3
Reusable S3UploadCommand

Fixed

Twitter image sizing across categories

bags: MediatR commands, S3 image storage, brand split

Sun, 02 Aug 2020 00:00:00 GMT

A big architectural overhaul onto MediatR commands, with processed images now stored in S3 instead of on local disk.

Added

Processed images uploaded to S3 by date
Backfill command for images of existing products
Brand stored separately from product name

Changed

MediatR command/handler architecture
Snake_case database columns
Dapper for raw SQL in content generation

bags: FWRD shoes, dresses and bags, multi-image tweets

Sat, 25 Jul 2020 00:00:00 GMT

The scraper grew from one retailer to four FWRD and Coach feeds, with each feed tweeting from its own pool and richer multi-image posts.

Added

FWRD dresses and bags categories
Up to four images per tweet
Price filter excluding products over $1000
Category selectable via environment variable
FWRD shoes as a second retailer
Category system with per-feed product selection

Changed

Logging through ILogger instead of the console
Category-specific image sizing in a dedicated ImageProcessor
Database operations moved into DatabaseContext
FWRD parsed from its lazy-load API with AngleSharp
Variable-width images keep their aspect ratio
Retailer scraping split into page object models

Fixed

Crash on FWRD products without a sale price

bags: Postgres persistence, Twitter posting, card image processing

Sat, 18 Jul 2020 00:00:00 GMT

Scraped products now land in a database and get tweeted automatically, with images reshaped to suit Twitter cards.

Added

Product images fitted to Twitter card size with ImageSharp
Tweets a sale product with image, discount and prices
PostgreSQL product persistence via EF Core
Random unposted product selection with image download

Changed

Generic host with DI and user secrets config

Fixed

Posted status not saved after tweeting
Crash when there are no new products to tweet

bags: Coach bags Selenium scraper, Docker container

Mon, 06 Jul 2020 00:00:00 GMT

The project started as a C# console app scraping Coach's sale page with Selenium, packaged to run headless in Docker.

Added

Selenium scraper for the Coach Australia bags sale
Anti-detection Chrome setup
Dockerfile for headless runs