🔍 Scrape Explore

Scrape the Explore page trends and content. Collects trending topics, hashtags, and promoted content.

📋 What It Does

This script provides the following capabilities:

1. Automated operation — Runs directly in your browser console on x.com
2. Configurable settings — Customize behavior via the CONFIG object
3. Real-time progress — Shows live status updates with emoji-coded logs
4. Rate limiting — Built-in delays to respect X/Twitter's rate limits
5. Data export — Results exported as JSON/CSV for further analysis

Use cases:

• Scrape the Explore page trends and content. Collects trending topics, hashtags, and promoted content.
• Automate repetitive scrapers tasks on X/Twitter
• Save time with one-click automation — no API keys needed
• Works in any modern browser (Chrome, Firefox, Edge, Safari)

⚠️ Important Notes

Use responsibly! All automation should respect X/Twitter's Terms of Service. Use conservative settings and include breaks between sessions.

• This script runs in the browser DevTools console — not Node.js
• You must be logged in to x.com for the script to work
• Start with low limits and increase gradually
• Include random delays between actions to appear human
• Don't run multiple automation scripts simultaneously

🌐 Browser Console Usage

Steps:

1. Go to x.com/explore
2. Open browser console (F12 → Console tab)
3. Copy and paste the script from scripts/scrapeExplore.js
4. Press Enter to run

// scripts/scrapeExplore.js
// Browser console script to scrape Explore page trends and content
// Paste in DevTools console on x.com/explore
// by nichxbt

(() => {
  const sleep = (ms) => new Promise(r => setTimeout(r, ms));

  const CONFIG = {
    maxTrends: 30,
    scrapeTweets: true,
    maxTweets: 50,
    outputFormat: 'json', // 'json' or 'csv'
  };

  const run = async () => {
    console.log('🔍 XActions Explore Scraper');
    console.log('===========================');

    // Scrape trends
    const trends = [];
    document.querySelectorAll('[data-testid="trend"]').forEach((el, i) => {
      if (i >= CONFIG.maxTrends) return;
      const spans = el.querySelectorAll('[dir="ltr"] span');
      const texts = Array.from(spans).map(s => s.textContent.trim()).filter(Boolean);
      trends.push({
        rank: i + 1,
        name: texts[0] || '',
        category: texts.length > 1 ? texts[1] : '',
        postCount: texts.length > 2 ? texts[2] : '',
      });
    });

    console.log(`\n📊 Found ${trends.length} trends:`);
    trends.forEach(t => {
      console.log(`  ${t.rank}. ${t.name} ${t.postCount ? `(${t.postCount})` : ''}`);
    });

    // Scrape tweets from explore
    const tweets = [];
    if (CONFIG.scrapeTweets) {
      let scrollAttempts = 0;
      
      while (tweets.length < CONFIG.maxTweets && scrollAttempts < 20) {
        document.querySelectorAll('article[data-testid="tweet"]').forEach(tweet => {
          const text = tweet.querySelector('[data-testid="tweetText"]')?.textContent || '';
          const author = tweet.querySelector('[data-testid="User-Name"] a')?.textContent || '';
          const time = tweet.querySelector('time')?.getAttribute('datetime') || '';
          const link = tweet.querySelector('a[href*="/status/"]')?.href || '';
          const likes = tweet.querySelector('[data-testid="like"] span')?.textContent || '0';

          if (link && !tweets.find(t => t.link === link)) {
            tweets.push({ text, author, time, link, likes });
          }
        });

        window.scrollBy(0, 1000);
        await sleep(1500);
        scrollAttempts++;
      }

      console.log(`\n📰 Scraped ${tweets.length} tweets`);
    }

    // Output results
    const result = {
      trends,
      tweets: tweets.slice(0, CONFIG.maxTweets),
      scrapedAt: new Date().toISOString(),
      url: window.location.href,
    };

    if (CONFIG.outputFormat === 'csv') {
      const csvTrends = 'rank,name,category,postCount\n' +
        trends.map(t => `${t.rank},"${t.name}","${t.category}","${t.postCount}"`).join('\n');
      console.log('\n📋 Trends CSV:');
      console.log(csvTrends);
    }

    console.log('\n📦 Full JSON result:');
    console.log(JSON.stringify(result, null, 2));

    // Copy to clipboard
    try {
      await navigator.clipboard.writeText(JSON.stringify(result, null, 2));
      console.log('\n✅ Results copied to clipboard!');
    } catch (e) {
      console.log('\n⚠️ Could not copy to clipboard. Use console output above.');
    }
  };

  run();
})();

⚙️ Configuration

📖 Step-by-Step Tutorial

Step 1: Navigate to the right page

Open your browser and go to x.com/explore. Make sure you're logged in to your X/Twitter account.

Step 2: Open the browser console

• Chrome/Edge: Press F12 or Ctrl+Shift+J (Mac: Cmd+Option+J)
• Firefox: Press F12 or Ctrl+Shift+K
• Safari: Enable Developer menu in Preferences → Advanced, then press Cmd+Option+C

Step 3: Paste the script

Copy the entire script from scripts/scrapeExplore.js and paste it into the console.

Step 4: Customize the CONFIG (optional)

Before running, you can modify the CONFIG object at the top of the script to adjust behavior:

const CONFIG = {
  // Edit these values before running
  // See Configuration table above for all options
};

Step 5: Run and monitor

Press Enter to run the script. Watch the console for real-time progress logs:

• ✅ Green messages = success
• 🔄 Blue messages = in progress
• ⚠️ Yellow messages = warnings
• ❌ Red messages = errors

Step 6: Export results

Most scripts automatically download results as JSON/CSV when complete. Check your Downloads folder.

🖥️ CLI Usage

You can also run this via the XActions CLI:

# Install XActions globally
npm install -g xactions

# Run via CLI
xactions --help

🤖 MCP Server Usage

Use with AI agents (Claude, Cursor, etc.) via the MCP server:

# Start MCP server
npm run mcp

See the MCP Setup Guide for integration with Claude Desktop, Cursor, and other AI tools.

📁 Source Files

🔗 Related Scripts

Author: nich (@nichxbt) — XActions on GitHub