Skip to main content

DataForSEO Integration

Complete guide to DataForSEO API integration, rate limits, cost management, and usage patterns.

API Overview​

RankFabric integrates with multiple DataForSEO API products to provide comprehensive SEO and competitive intelligence data:

API ProductPurposeBase Cost
Backlinks APIReferring domains, backlink analysis, spam scores$0.02-0.10/request
DataForSEO LabsKeyword research, ranked keywords, domain metrics$0.004-0.05/request
Keywords Data APIKeyword suggestions from Google Ads$0.05/request
On-Page APIInstant page content fetching$0.000125/URL
App Data APIMobile app store data (Google Play, Apple)$0.01-0.05/request
Business Data APIBusiness listings searchVariable

File Structure​

packages/api/src/lib/dataforseo/
├── dataforseo-api.js          # App Data API (Google Play, Apple App Store)
├── dataforseo.js              # Keywords API (Labs + Google Ads)
├── dataforseo-backlinks.js    # Backlinks API (referring domains, summaries)
├── dataforseo-spam-score.js   # Bulk spam scores and domain ranks
├── dataforseo-categories.js   # Category taxonomy mapping
├── dataforseo-listings.js     # App listings search
├── dataforseo-business.js     # Business listings search
└── instant-pages-batch.js     # On-Page instant pages (batched)

Authentication​

Credential Storage​

DataForSEO uses HTTP Basic Authentication. Credentials are stored as Cloudflare Worker secrets:

# Set credentials via Wrangler CLI
echo "your-login" | wrangler secret put DATAFORSEO_LOGIN
echo "your-password" | wrangler secret put DATAFORSEO_PASSWORD

Authentication Implementation​

All DataForSEO modules use centralized authentication from dataforseo-api.js:

// Centralized auth function - import this instead of duplicating
export function getDataForSEOAuth(env) {
  if (!env.DATAFORSEO_LOGIN || !env.DATAFORSEO_PASSWORD) {
    throw new Error("DATAFORSEO_LOGIN or DATAFORSEO_PASSWORD missing");
  }
  return `Basic ${btoa(`${env.DATAFORSEO_LOGIN}:${env.DATAFORSEO_PASSWORD}`)}`;
}

// Usage in API calls
const response = await fetch(endpoint, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: getDataForSEOAuth(env),
  },
  body: JSON.stringify(payload),
});

Environment Variables​

Configure in wrangler.toml:

[vars]
DATAFORSEO_LABS_LIMIT = "100"
DATAFORSEO_LABS_MAX_REQUESTS = "1"
DATAFORSEO_DAILY_BUDGET = "50"
DATAFORSEO_LOCATION_CODE = "2840"
DATAFORSEO_LANGUAGE_CODE = "en"

The Backlinks API provides link intelligence data including referring domains, individual backlinks, domain summaries, and spam scores.

Referring Domains​

Get domains linking to a target with full metrics.

Endpoint: POST https://api.dataforseo.com/v3/backlinks/referring_domains/live

Cost: $0.02 base + $0.0001 per result

Request:

const params = {
  target: "spotify.com",
  limit: 1000,
  offset: 0,
  order_by: ["rank,desc"],
  include_subdomains: true,
  exclude_internal_backlinks: true,
  include_indirect_links: true,
  backlinks_status_type: "live",
  internal_list_limit: 10,
  rank_scale: "one_hundred",
};

const response = await backlinksRequest("/referring_domains/live", [params], env);

Response Structure:

{
  "tasks": [{
    "status_code": 20000,
    "cost": 0.12,
    "result": [{
      "target": "spotify.com",
      "total_count": 5847293,
      "items_count": 1000,
      "items": [{
        "domain": "example.com",
        "rank": 85,
        "backlinks": 47,
        "backlinks_spam_score": 12,
        "referring_domains": 1250,
        "broken_backlinks": 2,
        "first_seen": "2019-02-16 14:22:37 +00:00",
        "lost_date": null,
        "referring_links_tld": { "com": 35, "org": 8, "net": 4 },
        "referring_links_types": { "anchor": 40, "image": 5, "redirect": 2 },
        "referring_links_platform_types": { "blogs": 15, "news": 10 },
        "referring_links_semantic_locations": { "main": 30, "footer": 10 },
        "referring_links_countries": { "US": 25, "GB": 12, "DE": 5 }
      }]
    }]
  }]
}

Usage:

import { getReferringDomains, fetchAndStoreReferringDomains } from "./dataforseo-backlinks.js";

// Fetch only
const result = await getReferringDomains("spotify.com", { limit: 1000 }, env);

// Fetch and store in database
const stored = await fetchAndStoreReferringDomains("spotify.com", {
  limit: 1000,
  dataTier: "global_sample",
}, env);
// Returns: { total_count, fetched, inserted, updated, cost }

Domain Summary​

Aggregate backlink statistics without individual domain details.

Endpoint: POST https://api.dataforseo.com/v3/backlinks/summary/live

Cost: $0.02 per request (cheap aggregate endpoint)

Request:

const params = {
  target: "spotify.com",
  include_subdomains: true,
  exclude_internal_backlinks: true,
  backlinks_status_type: "live",
};

Response:

{
  "tasks": [{
    "cost": 0.02,
    "result": [{
      "target": "spotify.com",
      "rank": 92,
      "backlinks": 847293847,
      "referring_domains": 5847293,
      "referring_domains_nofollow": 1234567,
      "referring_main_domains": 4521000,
      "referring_ips": 892345,
      "referring_subnets": 234567,
      "backlinks_spam_score": 8,
      "broken_backlinks": 12847,
      "broken_pages": 4521,
      "referring_links_types": { "anchor": 70, "image": 20, "redirect": 10 },
      "referring_links_platform_types": {
        "news": 150000,
        "blogs": 280000,
        "ecommerce": 45000,
        "message-boards": 32000,
        "social": 89000
      }
    }]
  }]
}

Usage:

import { getDomainSummary, fetchAndStoreDomainSummary } from "./dataforseo-backlinks.js";

// Fetch and store with weekly snapshots
const result = await fetchAndStoreDomainSummary("spotify.com", {}, env);
// Stores in domain_summaries table with year_week for historical tracking

Get actual URLs linking to the target.

Endpoint: POST https://api.dataforseo.com/v3/backlinks/backlinks/live

Cost: $0.02 base + $0.0001 per result

Request:

const params = {
  target: "spotify.com",
  limit: 1000,
  offset: 0,
  order_by: ["rank,desc"],
  include_subdomains: true,
  exclude_internal_backlinks: true,
  backlinks_status_type: "live",
  mode: "as_is", // as_is, one_per_domain, one_per_anchor
};

Response includes:

  • url_from - Source URL of the backlink
  • url_to - Target URL being linked to
  • anchor - Anchor text
  • domain_from_rank - Source domain authority
  • page_from_rank - Source page authority
  • backlink_spam_score - Spam score (0-100)
  • semantic_location - Where link appears (main, footer, sidebar)
  • link_attributes - nofollow, ugc, sponsored
  • is_broken, is_lost - Link status

Domain Pages Summary​

Get backlink stats for pages on YOUR domain (not linking domains).

Endpoint: POST https://api.dataforseo.com/v3/backlinks/domain_pages_summary/live

Cost: $0.02 base + $0.0001 per result

Usage:

import { getDomainPagesSummary, fetchAndStoreDomainPagesSummary } from "./dataforseo-backlinks.js";

// Get backlink stats for top pages on your domain
const result = await getDomainPagesSummary("notion.so", { limit: 500 }, env);
// Returns pages sorted by backlinks count with spam scores and link distributions

Bulk Spam Scores​

Fetch spam scores for up to 1000 targets in one request.

Endpoint: POST https://api.dataforseo.com/v3/backlinks/bulk_spam_score/live

Cost: ~$0.02 per 1000 targets

Request:

const targets = ["forbes.com", "spamsite.xyz", "example.com"];

const response = await fetch("https://api.dataforseo.com/v3/backlinks/bulk_spam_score/live", {
  method: "POST",
  headers: {
    Authorization: `Basic ${authB64}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify([{ targets }]),
});

Response:

{
  "tasks": [{
    "cost": 0.02,
    "result": [{
      "items": [
        { "target": "forbes.com", "spam_score": 2 },
        { "target": "spamsite.xyz", "spam_score": 87 },
        { "target": "example.com", "spam_score": 5 }
      ]
    }]
  }]
}

Spam Score Tiers:

ScoreTierDescription
0-19cleanLow risk, high quality
20-39low_riskAcceptable
40-59moderateMonitor
60-79high_riskCaution advised
80-100toxicDisavow candidate

Bulk Domain Ranks​

Fetch domain authority ranks for up to 1000 domains.

Endpoint: POST https://api.dataforseo.com/v3/backlinks/bulk_ranks/live

Cost: ~$0.02 per 1000 targets

import { fetchBulkRanks, enrichDomainWithRank } from "./dataforseo-spam-score.js";

// Batch fetch
const { results, cost } = await fetchBulkRanks(["google.com", "example.com"], env);
// results = { "google.com": 100, "example.com": 45 }

// Fetch and update database with quality_tier
await enrichDomainWithRank(domainId, "example.com", env);

Keywords API​

Ranked Keywords​

Get keywords a domain actually ranks for with positions and URLs.

Endpoint: POST https://api.dataforseo.com/v3/dataforseo_labs/google/ranked_keywords/live

Cost: $0.05 per request

Request:

const payload = [{
  target: "notion.so",
  location_code: 2840,
  language_code: "en",
  limit: 100,
  offset: 0,
  order_by: ["keyword_data.keyword_info.search_volume,desc"],
  load_rank_absolute: true,
  ignore_synonyms: false,
  include_clickstream_data: false,
}];

Response:

{
  "tasks": [{
    "cost": 0.05,
    "result": [{
      "total_count": 847293,
      "items": [{
        "keyword_data": {
          "keyword": "notion templates",
          "keyword_info": {
            "search_volume": 27100,
            "cpc": 2.45,
            "competition": 0.65,
            "monthly_searches": [
              { "month": 11, "year": 2025, "search_volume": 27100 }
            ]
          },
          "search_intent_info": {
            "main_intent": "commercial"
          },
          "keyword_properties": {
            "keyword_difficulty": 72
          }
        },
        "ranked_serp_element": {
          "url": "https://notion.so/templates",
          "serp_item": {
            "rank_absolute": 3,
            "rank_group": 2,
            "is_new": false,
            "is_up": true
          }
        }
      }]
    }]
  }]
}

Usage:

import { fetchRankedKeywords, fetchAllRankedKeywords } from "./dataforseo.js";

// Single page
const result = await fetchRankedKeywords("notion.so", env, { limit: 100 });

// Paginated fetch (up to max_items)
const all = await fetchAllRankedKeywords("notion.so", env, {
  max_items: 1000,
  batch_size: 100,
});
// Returns: { items, total_count, fetched, cost }

Keywords for Site​

Get keyword opportunities for a domain.

Endpoint: POST https://api.dataforseo.com/v3/dataforseo_labs/google/keywords_for_site/live

Cost: ~$0.004 per 100 keywords

import { fetchKeywordsForSite } from "./dataforseo.js";

const keywords = await fetchKeywordsForSite("notion.so", env, {
  limit: 100,
  location_code: 2840,
  language_code: "en",
});

Keyword Suggestions (Labs)​

Get related keywords with full metrics.

Endpoint: POST https://api.dataforseo.com/v3/dataforseo_labs/google/keyword_suggestions/live

Cost: ~$0.004 per 100 keywords

import { fetchKeywordSuggestionsLabs } from "./dataforseo.js";

const result = await fetchKeywordSuggestionsLabs({
  keyword: "project management",
  location_code: 2840,
  language_code: "en",
  limit: 100,
  include_seed_keyword: true,
  include_serp_info: false,
  order_by: ["keyword_info.search_volume,desc"],
}, env);

// Returns: { suggestions, meta, raw }

Keyword Suggestions (Google Ads)​

Get related keywords from Google Ads API.

Endpoint: POST https://api.dataforseo.com/v3/keywords_data/google_ads/keywords_for_keywords/live

Cost: ~$0.05 per request

import { fetchKeywordSuggestions } from "./dataforseo.js";

const suggestions = await fetchKeywordSuggestions("seo tools", env, {
  limit: 100,
  location_code: 2840,
  include_seed_keyword: true,
});

Categories for Domain​

Get DataForSEO category IDs for a domain based on its keyword rankings.

Endpoint: POST https://api.dataforseo.com/v3/dataforseo_labs/google/categories_for_domain/live

import { fetchCategoriesForDomain } from "./dataforseo.js";

const categories = await fetchCategoriesForDomain("notion.so", env, {
  limit: 100,
  include_subcategories: false,
});
// Returns categories with organic/paid ETV and keyword counts

Domain Metrics by Category​

Get top domains for a specific category with traffic metrics.

Endpoint: POST https://api.dataforseo.com/v3/dataforseo_labs/google/domain_metrics_by_categories/live

import { fetchDomainMetricsByCategory } from "./dataforseo.js";

const domains = await fetchDomainMetricsByCategory(12045, env, {
  limit: 100,
  first_date: "2025-09-01",
  second_date: "2025-11-01",
});
// Returns domains with organic ETV, keyword counts, and period-over-period changes

On-Page API (Instant Pages)​

Fetch rendered HTML content from URLs with metadata extraction.

Rate Limits​

  • 2,000 requests/minute
  • 20 tasks (URLs) per request
  • Max 5 same-domain URLs per batch
  • 30 concurrent requests

Theoretical throughput: 40,000 URLs/minute

Costs​

  • Cost per URL: $0.000125

Batched Requests​

import { fetchInstantPagesBatch, fetchInstantPagesSingle } from "./instant-pages-batch.js";

// Batch fetch (recommended)
const results = await fetchInstantPagesBatch(
  ["https://example.com/page1", "https://example.com/page2"],
  env,
  {
    enable_javascript: true,
    enable_browser_rendering: false,
    load_resources: false,
    store_raw_html: false,
  }
);
// Returns: Map<url, result>

// Single URL convenience wrapper
const result = await fetchInstantPagesSingle("https://example.com", env);

Request Payload:

[{
  "url": "https://example.com/page",
  "check_spell": false,
  "disable_cookie_popup": true,
  "return_despite_timeout": true,
  "load_resources": false,
  "enable_javascript": true,
  "enable_browser_rendering": false,
  "store_raw_html": false
}]

Response Structure:

{
  success: true,
  url: "https://example.com/page",
  status_code: 200,
  content: {
    title: "Page Title",
    description: "Meta description",
    h1: ["Main Heading"],
    h2: ["Subheading 1", "Subheading 2"],
    h3: [],
    plain_text: "Full text content...",
    word_count: 1500,
    internal_links_count: 25,
    external_links_count: 8,
  },
  raw: {
    canonical: "https://example.com/page",
    og_tags: { "og:title": "...", "og:image": "..." },
    scripts: ["https://cdn.example.com/app.js"],
    links: { internal: [...], external: [...] },
    content_type: "text/html",
    charset: "utf-8",
    raw_html: null, // Only if store_raw_html: true
  },
  cost: 0.000125,
}

App Data API​

Google Play App List​

Get apps from Google Play charts by category.

Endpoint: POST https://api.dataforseo.com/v3/app_data/google/app_list/task_post

import { getGooglePlayCategoryApps } from "./dataforseo-api.js";

const task = await getGooglePlayCategoryApps("PRODUCTIVITY", {
  appCollection: "topselling_free", // topselling_free, topselling_paid, topgrossing
  locationCode: 2840,
  depth: 100,
  priority: 1, // 1 = normal (45 min), 2 = high (1 min)
}, env);
// Returns: { task_id, cost, status_code }

Apple App Store​

import { getAppleCategoryApps, getAppleAppInfo } from "./dataforseo-api.js";

// Category list
const task = await getAppleCategoryApps("6007", { // Productivity category
  appCollection: "top_free_ios",
  depth: 100,
}, env);

// Individual app info
const appTask = await getAppleAppInfo("284882215", env); // Facebook app ID

App Listings Search (Live)​

Instant results for app searches by category.

import { searchGooglePlayListings, searchAppleListings } from "./dataforseo-listings.js";

// Google Play
const result = await searchGooglePlayListings("Productivity", {
  limit: 100,
  appCollection: "topselling_free",
}, env);

// Apple
const result = await searchAppleListings(["Productivity"], {
  limit: 100,
  locationCode: 2840,
}, env);

Rate Limiting​

Account Limits​

Limit TypeValue
Requests per minute2,000
Concurrent requests30
Tasks per requestVaries by endpoint (1-1000)
Daily requestsUnlimited (cost-based)

Handling Rate Limits​

// Rate limit error response
{
  "status_code": 40901,
  "status_message": "Rate limit exceeded"
}

// Retry with exponential backoff
async function retryWithBackoff(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (err) {
      if (err.message.includes("Rate limit") && i < maxRetries - 1) {
        const delay = Math.pow(2, i) * 1000 + Math.random() * 1000;
        await new Promise(res => setTimeout(res, delay));
        continue;
      }
      throw err;
    }
  }
}

Cost Management​

Per-Call Costs​

EndpointBase CostPer-Result Cost
Backlinks - Referring Domains$0.02$0.0001/domain
Backlinks - Summary$0.02-
Backlinks - Backlinks$0.02$0.0001/link
Backlinks - Bulk Spam Score$0.02Per 1000 targets
Backlinks - Bulk Ranks$0.02Per 1000 targets
Labs - Ranked Keywords$0.05-
Labs - Keywords for Site~$0.004Per 100 keywords
Labs - Keyword Suggestions~$0.004Per 100 keywords
Keywords Data - Keywords for Keywords$0.05-
On-Page - Instant Pages$0.000125Per URL
App Data - App List$0.01-0.05Varies

Cost Tracking​

All API calls track costs via centralized cost tracker:

import { trackCost, COST_SERVICES } from "../utils/cost-tracker.js";

// Track after successful API call
if (task.cost > 0) {
  await trackCost(env, {
    service: COST_SERVICES.DATAFORSEO_BACKLINKS,
    cost_usd: task.cost,
    endpoint: "/referring_domains/live",
    items_returned: result.items_count || 0,
    success: true,
  });
}

Budget Enforcement​

// Check daily budget before expensive operations
const spent = await env.DFS_BUDGETS.get('budget:daily');
const limit = parseFloat(env.DATAFORSEO_DAILY_BUDGET || 50);

if (parseFloat(spent || 0) >= limit) {
  throw new Error('Daily DataForSEO budget exceeded');
}

Cost Optimization Best Practices​

  1. Use batch endpoints - 100 keywords in 1 request = $0.004 vs 100 separate requests = $0.40

  2. Use bulk endpoints - Bulk spam scores/ranks: 1000 targets for $0.02 vs individual lookups

  3. Use summary endpoints - Domain summary ($0.02) vs full referring domains ($0.02 + $0.0001/result)

  4. Cache aggressively - Category data rarely changes, keyword data changes monthly

  5. Set daily budgets - Prevent runaway costs from bugs or loops

Error Handling​

Status Codes​

CodeMeaningAction
20000SuccessProcess results
20100Task in queuePoll for results
40001Invalid parametersFix request payload
40101Authentication failedCheck credentials
40201Rate limit exceededImplement backoff
40204Insufficient balanceAdd credits
50000Internal server errorRetry with backoff

Error Response Example​

{
  "status_code": 40101,
  "status_message": "Incorrect login or password",
  "tasks": null
}

Error Handling Pattern​

async function makeDataForSEORequest(endpoint, payload, env) {
  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      Authorization: getDataForSEOAuth(env),
      "Content-Type": "application/json",
    },
    body: JSON.stringify(payload),
  });

  if (!response.ok) {
    const errorText = await response.text();
    const err = new Error(
      `DataForSEO API error ${response.status}: ${errorText.slice(0, 200)}`
    );
    
    // Set appropriate error status for upstream handling
    if (response.status === 429) {
      err.status = 429; // Rate limit - retry
    } else if (response.status >= 400 && response.status < 500) {
      err.status = 400; // Bad request - don't retry
    } else {
      err.status = 502; // Server error - retry
    }
    throw err;
  }

  const data = await response.json();
  
  if (data.status_code !== 20000) {
    throw new Error(`DataForSEO error: ${data.status_message} (code: ${data.status_code})`);
  }

  const task = data.tasks?.[0];
  if (!task || task.status_code !== 20000) {
    throw new Error(`DataForSEO task error: ${task?.status_message || "No task returned"}`);
  }

  return task;
}

Retry Strategy​

Error TypeRetry?Strategy
Network errorsYesExponential backoff
5xx server errorsYesExponential backoff
Rate limit (429/40201)YesLonger backoff (5-30s)
401 authenticationNoCheck credentials
400 bad requestNoFix request
402 insufficient fundsNoAdd credits

Caching​

Cache Strategy by Data Type​

Data TypeTTLCache Key Pattern
Category taxonomy30 daysdfs:category:{id}
Domain summary7 daysdfs:summary:{domain}:{yearWeek}
Keyword suggestions7 daysdfs:suggestions:{keyword}:{location}
Ranked keywords1 daydfs:ranked:{domain}:{location}
Spam scores7 daysdfs:spam:{domain}
Instant pages1 daydfs:page:{urlHash}

KV Cache Implementation​

// Check cache before API call
const cacheKey = `dfs:suggestions:${keyword}:${locationCode}`;
const cached = await env.DFS_CACHE.get(cacheKey);

if (cached) {
  return JSON.parse(cached);
}

// Fetch from DataForSEO
const result = await fetchFromDataForSEO(...);

// Cache result with TTL
await env.DFS_CACHE.put(cacheKey, JSON.stringify(result), {
  expirationTtl: 86400 * 7, // 7 days
});

return result;

Category Taxonomy Caching​

DataForSEO categories are stored in KV with local JSON fallback:

import { getCategoryPath, getCategoryPaths } from "./dataforseo-categories.js";

// Single category lookup (uses in-memory cache -> KV -> local JSON)
const path = await getCategoryPath("12045", env);
// Returns: "Internet & Telecom > Web Services > SEO & SEM"

// Batch lookup
const paths = await getCategoryPaths(["12045", "12046"], env);
// Returns: { "12045": "...", "12046": "..." }

Location and Language Codes​

Common Location Codes​

CodeLocation
2840United States
2826United Kingdom
2124Canada
2036Australia
2276Germany
2250France
2392Japan
2076Brazil

Common Language Codes​

CodeLanguage
enEnglish
esSpanish
frFrench
deGerman
ptPortuguese
jaJapanese
zhChinese

Full lists: Location Codes | Language Codes

Database Integration​

Upsert Patterns​

RankFabric uses batched upserts to efficiently store DataForSEO data:

// Batched upsert for referring domains (50 per batch)
export async function upsertReferringDomains(items, targetDomain, env, options = {}) {
  const statements = items.map((item) => {
    const mapped = mapReferringDomain(item, targetDomain, options);
    return env.DB.prepare(`
      INSERT INTO referring_domains (...) VALUES (...)
      ON CONFLICT(target_domain, referring_domain) DO UPDATE SET
        rank = excluded.rank,
        backlinks_count = excluded.backlinks_count,
        ...
    `).bind(...);
  });

  // Execute in batches of 50 (D1 limit)
  const BATCH_SIZE = 50;
  for (let i = 0; i < statements.length; i += BATCH_SIZE) {
    await env.DB.batch(statements.slice(i, i + BATCH_SIZE));
  }
}

Weekly Snapshots​

Domain summaries are stored with year_week for historical tracking:

export function getCurrentYearWeek() {
  const now = new Date();
  const year = now.getFullYear();
  const oneJan = new Date(year, 0, 1);
  const week = Math.ceil(((now - oneJan) / 86400000 + oneJan.getDay() + 1) / 7);
  return year * 100 + week; // e.g., 202547
}

See Also​