How does the RSS.app Generator Work?

The Generator Pipeline

The RSS.app Generator operates as a repeatable pipeline rather than a single action. Each refresh cycle executes the following stages: fetching the page, extracting content, detecting changes, normalizing data, and producing formatted outputs.

Understanding this pipeline explains how the Generator maintains accuracy over time, handles edge cases, and delivers consistent results across different source pages. Each stage has specific responsibilities and failure modes.

Page Fetching

The pipeline begins by requesting the source URL. This is not a simple HTTP GET; the Generator employs multiple strategies to retrieve content reliably:

• Browser rendering. For JavaScript-heavy pages, a headless browser renders the page and waits for dynamic content to load.
• Header management. Appropriate headers ensure the request appears as a standard browser visit, avoiding blocks or degraded responses.
• Timeout handling. Configurable timeouts prevent indefinite waits on slow or unresponsive pages.
• Retry logic. Transient failures trigger automatic retries with backoff intervals.

The output of this stage is the fully rendered HTML of the source page, ready for content extraction.

Content Extraction

With the HTML available, the Generator identifies content items on the page. This involves pattern recognition to find repeating structures:

Container detection. The Generator locates the parent element containing all content items: a list, grid, or table that holds articles, posts, or entries.

Item identification. Within the container, individual items are identified by their structural similarity. Each card, row, or entry becomes a potential feed item.

Field mapping. For each item, the Generator extracts standard feed fields:

• title — The headline or name of the item
• link — The URL to the full content
• description — Summary text or excerpt
• pubDate — Publication or modification timestamp
• image — Associated thumbnail or featured image
• author — Creator attribution when available

The extraction uses heuristics refined across millions of pages. For non-standard layouts, the RSS Builder provides manual control over field selection.

Change Comparison

Detecting new content requires comparing current extraction results against previous runs. The Generator maintains state between refresh cycles:

GUID assignment. Each item receives a globally unique identifier based on its URL and a content hash. This GUID persists across refreshes.

Seen-state tracking. The Generator remembers which GUIDs it has encountered. Items with new GUIDs are marked as new; items with known GUIDs are skipped or updated.

Content updates. If an existing item's content changes (title edited, description modified), the Generator can detect this through hash comparison and update the feed accordingly.

This comparison ensures feed consumers receive new items exactly once, without duplicates or missed updates.

Data Normalization

Raw extracted content requires cleanup before becoming a valid feed. Normalization handles:

Date parsing. Source pages express dates in countless formats—relative ("2 hours ago"), localized ("4 février"), or non-standard layouts. The Generator normalizes these to RFC 822 timestamps required by RSS.

URL resolution. Relative links are converted to absolute URLs. Protocol-relative URLs receive explicit schemes. Malformed URLs are corrected or excluded.

Text cleaning. Excessive whitespace, HTML entities, and encoding issues are resolved. Descriptions are truncated to reasonable lengths if needed.

Image extraction. If images exist but are not directly associated with items, the Generator attempts to match images to their corresponding content.

The result is clean, consistent data ready for format-specific output.

Format-Specific Output

Normalized data is serialized into multiple formats simultaneously:

XML (RSS 2.0)

The standard syndication format. Items become <item> elements with proper channel metadata, publication dates, and GUIDs. The output validates against RSS 2.0 specifications.

JSON

A structured JSON array matching the JSON Feed specification. Each item is an object with typed fields. This format integrates directly with JavaScript applications and REST APIs.

CSV

Tabular export with one row per item. Column headers match feed fields. This format imports directly into spreadsheets and database systems.

All formats share the same underlying data. Choosing a format is purely about consumer compatibility. The information content is identical.

Refresh Cycle Behavior

The pipeline runs on a configurable schedule. Each cycle:

1. Fetches the source page fresh (no caching of source content)
2. Extracts and compares against known state
3. Updates output files only if changes are detected
4. Records metadata about the refresh (timestamp, items found, status)

If no new content exists, outputs remain unchanged. Consumers fetching the feed URL receive cached results until the next successful update. This minimizes unnecessary processing while ensuring freshness when content changes.

Failure Handling and Stability

The Generator implements multiple resilience patterns:

Transient failure recovery. Network timeouts, temporary 5xx errors, and rate limits trigger automatic retries. The Generator distinguishes between recoverable and permanent failures.

Content preservation. If a refresh fails, existing feed content is preserved. Consumers continue receiving the last successful data rather than errors or empty feeds.

Layout change adaptation. Minor source page changes (CSS updates, element reordering) are handled automatically. The extraction heuristics tolerate reasonable variation without breaking.

Alerting. Persistent failures or dramatic content changes can trigger notifications, allowing intervention before consumers are affected.