MockGPT Web: trusted sources and local summaries

Get body text from a real web page or a trusted load, then hand it to input-dependent generators.

@jangts/agent-core/mock/web is a Node-only entry. sources.web({url?,profile?,headers?,timeoutMs?,maxBytes?,maxChars?,maxRedirects?,fallback?,load?}) returns a source for text generators. By default it takes the single URL from the prompt; ordinary generators do not go online. The defaults are a 20000 ms total deadline, 2 MiB, 30000 code points, and 3 redirects. Scripts and templates are stripped first and Readability then extracts the body; it does not execute page JS, log in, or read PDFs.

The default fetch validates protocol, DNS address, and redirects hop by hop, pins the actual connection address, and rejects local/private/reserved targets; isPublicAddress checks only a single address and is not a complete URL authorizer. Size is limited both before and after decompression. headers cannot override Host, Cookie, or Authorization, and model credentials are not forwarded.

load(url,signal,maxBytes) is a trusted configuration port returning {bytes,url,contentType}; whoever injects it owns the target address and transport security, and it must not accept a function supplied arbitrarily by a remote request. Read failures throw by default, and an explicit fallback only catches web-read failures that are allowed to degrade and then passes the text to the generator as source text; cancellation, invalid URLs, and target-address rejections are not turned into fallback success. Logged fields such as sourceError, sourceFallback, webStage, and receivedBytes distinguish failures, replacement text, and the actual body text, so a verification or login page is not reported as a successful read. Live web changes are not fixed by seed.

Progressive examples for the same business case

A trusted load supplies offline body text

Beginner · Node.js

TypeScript

import { MockGPT, generators } from '@jangts/agent-core/mock';
import { sources, isPublicAddress } from '@jangts/agent-core/mock/web';
const source = sources.web({ url: 'https://example.com/article', async load(url, signal, maxBytes) {
  signal.throwIfAborted();
  const bytes = new TextEncoder().encode('服务维护完成。健康检查通过。');
  if (bytes.byteLength > maxBytes) throw new Error('来源超过预算');
  return { bytes, url: url.href, contentType: 'text/plain; charset=utf-8' };
} });
const mock = new MockGPT({ rules: [{ generator: generators.summary({ source }) }] });
console.log(await mock.generate({ model: 'demo', messages: [{ role: 'user', content: '概括正文' }] }));
console.log(isPublicAddress('127.0.0.1')); // false

JavaScript

import { MockGPT, generators } from '@jangts/agent-core/mock';
import { sources, isPublicAddress } from '@jangts/agent-core/mock/web';
const source = sources.web({ url: 'https://example.com/article', async load(url, signal, maxBytes) {
        signal.throwIfAborted();
        const bytes = new TextEncoder().encode('服务维护完成。健康检查通过。');
        if (bytes.byteLength > maxBytes)
            throw new Error('来源超过预算');
        return { bytes, url: url.href, contentType: 'text/plain; charset=utf-8' };
    } });
const mock = new MockGPT({ rules: [{ generator: generators.summary({ source }) }] });
console.log(await mock.generate({ model: 'demo', messages: [{ role: 'user', content: '概括正文' }] }));
console.log(isPublicAddress('127.0.0.1')); // false

Explicitly enable a web source and a failure replacement text

Configuration · Node.js

TypeScript

import { MockGPT, generators } from '@jangts/agent-core/mock';
import { sources } from '@jangts/agent-core/mock/web';
const mock = new MockGPT({ rules: [{ generator: generators.summary({ source: sources.web({
  url: 'https://example.com/article', timeoutMs: 5000,
  fallback: '网页不可读取，请提供正文。',
}) }) }] });
console.log(await mock.generate({ model: 'demo', messages: [{ role: 'user', content: '总结文章' }] }));
console.log(mock.records); // 检查 sourceFallback/sourceError，不把替代文本当原文

JavaScript

import { MockGPT, generators } from '@jangts/agent-core/mock';
import { sources } from '@jangts/agent-core/mock/web';
const mock = new MockGPT({ rules: [{ generator: generators.summary({ source: sources.web({
                    url: 'https://example.com/article', timeoutMs: 5000,
                    fallback: '网页不可读取，请提供正文。',
                }) }) }] });
console.log(await mock.generate({ model: 'demo', messages: [{ role: 'user', content: '总结文章' }] }));
console.log(mock.records); // 检查 sourceFallback/sourceError，不把替代文本当原文