sentencepiece

62 stars 2 forks

"Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization."

Also in: javascript ci cd networking

Third-Party Agent Skill: Review the code before installing. Agent skills execute in your AI assistant's environment and can access your files. Learn more about security

Installation for Agentic Skill

View all platforms →

Claude Code (CLI) Fast

skilz install zechenzhangAGI/AI-research-SKILLs/sentencepiece

OpenCode (CLI) Fast

skilz install zechenzhangAGI/AI-research-SKILLs/sentencepiece --agent opencode

OpenAI Codex (CLI) Native

skilz install zechenzhangAGI/AI-research-SKILLs/sentencepiece --agent codex

Gemini CLI (Project) Project

skilz install zechenzhangAGI/AI-research-SKILLs/sentencepiece --agent gemini

First time? Install Skilz: pip install skilz

Works with 22+ AI coding assistants

Cursor, Aider, Copilot, Windsurf, Qwen, Kimi, and more...

View All Agents

For Claude Desktop Easy

Download Agent Skill ZIP

Extract and copy to ~/.claude/skills/ then restart Claude Desktop

Manual Installation

1. Clone the repository:

git clone https://github.com/zechenzhangAGI/AI-research-SKILLs

2. Copy the agent skill directory:

cp -r AI-research-SKILLs/02-tokenization/sentencepiece ~/.claude/skills/

View on GitHub

Need detailed installation help? Check our platform-specific guides:

Claude Desktop Guide Claude Code Guide Troubleshooting

Related Agentic Skills

mastering-github-cli

by SpillwaveSolutions

GitHub CLI (gh) command reference for repository search, code discovery, CI/CD monitoring, workflow authoring, and automation. Triggers on "gh" com...

100

TECHgithub

Marketplace

mastering-github-cli

by SpillwaveSolutions

GitHub CLI (gh) command reference for repository search, code discovery, CI/CD monitoring, workflow authoring, and automation. Triggers on "gh" com...

100

TECHgithub

Marketplace