TextBlocks: structure-aware splitting of long text

Split long text into bounded structural blocks while keeping the original text and offsets for paging or indexing.

split(text, { targetChars }) only processes decoded plain text. Headings and horizontal rules start a new block, blank lines end the current one, and other lines break at line boundaries once targetChars would be exceeded; a single over-long line stays one block so later paging can fragment it without turning one line into unrelated sections. Blocks reassemble into the original text exactly, preserving whitespace and CRLF; number starts at 1 and start/end are UTF-16 offsets in the source.

targetChars must be a positive integer, and empty text returns an empty array. The Kit does not read files, page, call models or touch storage; paging windows, retrieval, summaries and prompts remain App decisions. Boundaries follow structure rather than a fixed character window, so report coverage with start/end.

Progressive examples for the same business case

First split a short document with two sections

Beginner · Node.js

The example only demonstrates the splitting contract; paging windows, context budgets and prompts still belong to the App.

TypeScript

import { TextBlocks } from '@jangts/agent-core/kits/text-blocks';
const text = '# 概览\n\n' + '说明内容\n'.repeat(4) + '\n## 细节\n\n' + '更多内容\n'.repeat(4);
const blocks = TextBlocks.split(text, { targetChars: 40 });
console.log(blocks.map(block => [block.number, block.start, block.end]));
console.log(blocks.map(block => block.text).join('') === text);

JavaScript

import { TextBlocks } from '@jangts/agent-core/kits/text-blocks';
const text = '# 概览\n\n' + '说明内容\n'.repeat(4) + '\n## 细节\n\n' + '更多内容\n'.repeat(4);
const blocks = TextBlocks.split(text, { targetChars: 40 });
console.log(blocks.map(block => [block.number, block.start, block.end]));
console.log(blocks.map(block => block.text).join('') === text);

Page through blocks and record coverage

Configuration · Node.js

Block boundaries are not a fixed character window; record each block start/end instead of inferring character positions.

TypeScript

import { TextBlocks, type TextBlock } from '@jangts/agent-core/kits/text-blocks';
const text = '# 报告\n\n' + '记录内容\n'.repeat(200);
const blocks = TextBlocks.split(text, { targetChars: 4000 });
const page = (offset: number, limit: number): { blocks: TextBlock[]; nextOffset?: number } => ({
  blocks: blocks.slice(offset, offset + limit),
  nextOffset: offset + limit < blocks.length ? offset + limit : undefined,
});
const first = page(0, 2);
console.log(first.blocks.map(block => block.number), first.nextOffset);

JavaScript

import { TextBlocks } from '@jangts/agent-core/kits/text-blocks';
const text = '# 报告\n\n' + '记录内容\n'.repeat(200);
const blocks = TextBlocks.split(text, { targetChars: 4000 });
const page = (offset, limit) => ({
    blocks: blocks.slice(offset, offset + limit),
    nextOffset: offset + limit < blocks.length ? offset + limit : undefined,
});
const first = page(0, 2);
console.log(first.blocks.map(block => block.number), first.nextOffset);

Policies & implementations

• Structural boundaries, exact reassembly, and source offsets