Skip to main content

Improvement to Daily Sync Detectors for URL/Website Sources | August 2025

Michael Stephenson avatar
Written by Michael Stephenson
Updated yesterday

Released: August 15, 2025

We’ve expanded the Daily Sync feature for URL and Website Knowledge Hub sources to detect a broader range of update signals. With enhanced detection logic, Bolt AI now identifies content changes from more types of metadata, headers, and structured data—reducing missed updates and keeping your knowledge fresher.

Bolt AI now checks for updates using multiple signals, applied in priority order, starting with the most reliable indicators and falling back to others if needed:

  1. HTTP Headers

    • ETag: Reindex if different from the previous value

    • Last-Modified header variations: last-modified, x-last-modified, x-amz-meta-last-modified

    • Content hashes: x-content-hash, content-md5, x-checksum

    • AWS S3 version: x-amz-version-id

  2. Sitemap

    • Fetches sitemap.xml (or sitemaps listed in robots.txt)

    • Checks the lastmod value for the specific page

  3. HTML Meta Tags

    • Modified/Updated dates: article:modified_time, og:updated_time, last-modified, DC.date.modified, dcterms.modified, modified, updated, revised

    • Published/Created dates: article:published_time, og:article:published_time, DC.date, DC.date.created, dcterms.created, date, publish_date, publishdate, pubdate, created

  4. Structured Data

    • JSON-LD: dateModified, datePublished, dateCreated

    • Schema.org microdata: itemprop="dateModified", itemprop="datePublished"

    • HTML5 <time> tags with datetime attributes

  5. RSS Feed Items (for RSS-sourced content)

    • Compares the isoDate field from RSS feed items against the last indexed date

These improvements make Daily Sync more accurate and reliable by capturing updates from a wider variety of signals. This ensures your knowledge hub content reflects the most recent information with fewer missed changes.

Did this answer your question?