Pagination and Cursors

The whole model

All paginated endpoints return next_cursor followed immediately by has_next_page. Pass the returned cursor unchanged into the endpoint's documented pagination parameter. Followers and following use cursor; Posts and Reels use next_cursor. When there is no further page, next_cursor is null and has_next_page is false.

curl "https://api.socialscrape.dev/v1/instagram/followers?handle=natgeo" \
  -H "x-api-key: $INSCRAPE_KEY"

# data.next_cursor: "QVFEVWx6..."

curl "https://api.socialscrape.dev/v1/instagram/followers?handle=natgeo&cursor=QVFEVWx6..." \
  -H "x-api-key: $INSCRAPE_KEY"

Stop on the cursor, not on a page count

The number of items per page is decided upstream and is not a stable contract, so a loop that runs for a fixed number of pages either stops early and silently loses data or runs past the end and wastes requests on empty responses. Successful empty first requests are charged at the endpoint rate. Exception: a successful cursor request returning zero items costs 0 on posts, reels-posts, reposts-posts, tagged-posts, comments, popular and hashtag. Their first requests still cost the endpoint rate even when empty. Loop while the endpoint's returned cursor is truthy. If you need a spend ceiling, cap the loop on items collected or credits spent, both of which you can read directly from the response, rather than on pages.

A full drain in Node

The list key differs per endpoint and each endpoint reference page names its own, so pass it in rather than guessing. Everything else is identical across the nine list endpoints.

async function drain(path, params, listKey, maxItems = Infinity) {
  const items = [];
  let cursor;

  do {
    const url = new URL(path, "https://api.socialscrape.dev");
    for (const [k, v] of Object.entries(params)) url.searchParams.set(k, String(v));
    if (cursor) url.searchParams.set("cursor", cursor);

    const res = await fetch(url, {
      headers: { "x-api-key": process.env.INSCRAPE_KEY },
    });
    const body = await res.json();
    if (!res.ok) throw new Error(`${body.error.code}: ${body.error.message}`);

    items.push(...body.data[listKey]);
    cursor = body.data.next_cursor;
  } while (cursor && items.length < maxItems);

  return items;
}

const posts = await drain("/v1/instagram/posts", { handle: "natgeo", limit: 50 }, "posts");

The same drain in Python

Yield instead of accumulating once the list is large enough that holding it in memory matters, which a follower list of any size will be.

import os
import requests

BASE = "https://api.socialscrape.dev"
HEADERS = {"x-api-key": os.environ["INSCRAPE_KEY"]}


def drain(path, params, list_key):
    cursor = None
    while True:
        query = dict(params)
        if cursor:
            query["cursor"] = cursor

        res = requests.get(f"{BASE}{path}", params=query, headers=HEADERS, timeout=30)
        body = res.json()
        if not res.ok:
            raise RuntimeError(f"{body['error']['code']}: {body['error']['message']}")

        yield from body["data"][list_key]

        cursor = body["data"].get("next_cursor")
        if not cursor:
            return


for post in drain("/v1/instagram/posts", {"handle": "natgeo", "limit": 50}, "posts"):
    print(post["shortcode"])

Each page is priced separately

A page is a request. Successful pages cost the endpoint rate. Exception: a successful cursor request returning zero items costs 0 on posts, reels-posts, reposts-posts, tagged-posts, comments, popular and hashtag. Their first requests still cost the endpoint rate even when empty. Store the last cursor you processed if you need to resume a drain, but treat it as short-lived request state rather than a permanent page number.

Cursors are opaque and not durable

Treat the cursor as an opaque string. Do not parse it, do not construct one, and do not assume one taken from a run last week still resolves. The gateway does not inspect a cursor, it forwards it, so a stale one fails at the upstream fetch rather than being rejected on the way in, and like every other failure it is not charged. For a long-running incremental job, key your state on something stable in the data such as a shortcode or a timestamp, and use the cursor only within a single drain.