How to Monitor Any Website for Changes Automatically

You want to know the moment a competitor changes their pricing, a product comes back in stock, or a regulation page gets updated. Manual checking doesn't scale. That's what automated website change monitoring is for — periodic scraping, comparison, and alerting whenever something meaningful changes. This guide shows you how to build one from scratch using Node.js and Papalily.

Use Cases for Website Change Monitoring

Competitor pricing: Get notified the moment a competitor drops prices or runs a sale
Inventory alerts: Watch an out-of-stock product and get alerted when it returns
Content monitoring: Track regulatory pages, terms of service changes, or press releases
Job postings: Monitor a target company's careers page for new openings
News alerts: Watch a source for articles on specific topics
Domain/IP monitoring: Track changes to infrastructure-related pages
Real estate listings: Get notified when new properties matching your criteria appear

The Core Approach: Extract → Hash → Compare → Alert

The simplest change detection works like this:

Extract the specific data you care about from the page
Hash it to create a compact fingerprint
Compare the new hash to the previous stored hash
If they differ, something changed — send an alert
Store the new hash for next time

The key insight: by extracting only the data you care about (not the entire page), you avoid false positives from ads, timestamps, and irrelevant page elements that change constantly.

Step 1: Build the Core Extractor

const crypto = require('crypto');
const fs = require('fs');

const API_KEY = process.env.PAPALILY_API_KEY;
const STATE_FILE = 'monitor-state.json';

async function extractData(url, prompt) {
  const res = await fetch('https://api.papalily.com/scrape', {
    method: 'POST',
    headers: {
      'x-api-key': API_KEY,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ url, prompt, no_cache: true }),
  });
  const result = await res.json();
  if (!result.success) throw new Error(result.error);
  return result.data;
}

function hashData(data) {
  return crypto
    .createHash('sha256')
    .update(JSON.stringify(data))
    .digest('hex');
}

function loadState() {
  if (!fs.existsSync(STATE_FILE)) return {};
  return JSON.parse(fs.readFileSync(STATE_FILE, 'utf8'));
}

function saveState(state) {
  fs.writeFileSync(STATE_FILE, JSON.stringify(state, null, 2));
}

async function checkForChanges(monitors) {
  const state = loadState();
  const changes = [];

  for (const monitor of monitors) {
    try {
      const data = await extractData(monitor.url, monitor.prompt);
      const hash = hashData(data);
      const key = monitor.url;

      if (state[key] && state[key].hash !== hash) {
        changes.push({
          name: monitor.name,
          url: monitor.url,
          previousData: state[key].data,
          currentData: data,
          detectedAt: new Date().toISOString(),
        });
      }

      state[key] = { hash, data, lastChecked: new Date().toISOString() };
    } catch (err) {
      console.error(`Failed to check ${monitor.name}: ${err.message}`);
    }
  }

  saveState(state);
  return changes;
}

Step 2: Define Your Monitors

const MONITORS = [
  {
    name: 'Competitor Pricing Page',
    url: 'https://competitor.com/pricing',
    prompt: 'Extract all pricing tiers with name, monthly price, annual price, and feature list. Return as JSON.',
  },
  {
    name: 'Sony XM6 — Back in Stock',
    url: 'https://www.sony.com/en/products/headphones/wh-1000xm6',
    prompt: 'Get the product name, current price, and stock status (in stock / out of stock). Return simple JSON.',
  },
  {
    name: 'Target Company Jobs Page',
    url: 'https://jobs.targetcompany.com/engineering',
    prompt: 'Get all open job listings with title and department. Return as array.',
  },
  {
    name: 'Regulatory Update Page',
    url: 'https://www.sec.gov/rules/proposed.shtml',
    prompt: 'Get all proposed rules listed with title, file number, and date. Return as array.',
  },
];

Step 3: Add Slack Alerts

async function sendSlackAlert(change) {
  const webhook = process.env.SLACK_WEBHOOK_URL;
  if (!webhook) return;

  const diff = JSON.stringify(change.currentData, null, 2);
  const truncated = diff.length > 800 ? diff.substring(0, 800) + '...' : diff;

  await fetch(webhook, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      blocks: [
        {
          type: 'section',
          text: {
            type: 'mrkdwn',
            text: `*🚨 Change Detected: ${change.name}*\n` +
                  `URL: ${change.url}\n` +
                  `Detected at: ${change.detectedAt}`,
          },
        },
        {
          type: 'section',
          text: {
            type: 'mrkdwn',
            text: `*New data:*\n\`\`\`${truncated}\`\`\``,
          },
        },
      ],
    }),
  });
}

// Run the full monitor loop
async function runMonitor() {
  console.log(`[${new Date().toISOString()}] Checking ${MONITORS.length} targets...`);
  const changes = await checkForChanges(MONITORS);

  if (changes.length === 0) {
    console.log('No changes detected.');
    return;
  }

  console.log(`${changes.length} change(s) detected!`);
  for (const change of changes) {
    console.log(`Changed: ${change.name}`);
    await sendSlackAlert(change);
  }
}

Step 4: Schedule with node-cron

const cron = require('node-cron');

// Check every 30 minutes
cron.schedule('*/30 * * * *', async () => {
  await runMonitor();
});

// Also run immediately on startup
runMonitor();

console.log('Website change monitor started. Checking every 30 minutes.');

Handling False Positives

Some pages naturally change on every load — timestamps, view counts, session tokens, ad slots. The best way to avoid false positives is to be specific in your prompt:

Instead of: "Get the full page content" (changes every load due to timestamps)
Use: "Get only the product price, stock status, and product name" (stable across loads unless genuinely changed)

You can also add a minimum-change threshold: only alert if the extracted JSON differs by more than a certain number of fields, or only if specific fields (like price or in_stock) change rather than any field.

Scaling to Many Targets

For monitoring 50+ pages on a regular schedule, consider:

Batch API: Group up to 20 monitors per batch call, reducing total API calls and running them in parallel
Stagger checks: Don't hit all targets at once — spread checks throughout the monitoring window to stay within rate limits
SQLite or PostgreSQL: Replace the JSON state file with a proper database for storing history and enabling queries like "show me all changes in the last week"
Separate intervals per monitor: Check pricing pages every 6 hours, inventory pages every hour, regulatory pages once a day

Start Monitoring Any Website Today

Papalily's real-browser AI extraction handles JavaScript, dynamic content, and anti-bot measures. Perfect for change monitoring on modern React and Vue sites. Free tier — 100 requests/month.

Get Free API Key on RapidAPI →

Full docs at papalily.com/docs