huggingface-tokenizers

Name: huggingface-tokenizers
Rating: 1.4 (1 reviews)
Author: zechenzhangAGI

27.2

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Marketplace

Third-Party Agent Skill: Review the code before installing. Agent skills execute in your AI assistant's environment and can access your files. Learn more about security

Installation for Agentic Skill

View all platforms →

Claude Code (CLI) Fast

skilz install zechenzhangAGI/AI-research-SKILLs/huggingface-tokenizers

OpenCode (CLI) Fast

skilz install zechenzhangAGI/AI-research-SKILLs/huggingface-tokenizers --agent opencode

OpenAI Codex (CLI) Native

skilz install zechenzhangAGI/AI-research-SKILLs/huggingface-tokenizers --agent codex

Gemini CLI (Project) Project

skilz install zechenzhangAGI/AI-research-SKILLs/huggingface-tokenizers --agent gemini

First time? Install Skilz: pip install skilz

Works with 14 AI coding assistants

Cursor, Aider, Copilot, Windsurf, Qwen, Kimi, and more...

View All Agents

For Claude Desktop Easy

Download Agent Skill ZIP

Extract and copy to ~/.claude/skills/ then restart Claude Desktop

Manual Installation

1. Clone the repository:

git clone https://github.com/zechenzhangAGI/AI-research-SKILLs

2. Copy the agent skill directory:

cp -r AI-research-SKILLs/02-tokenization/huggingface-tokenizers ~/.claude/skills/

View on GitHub

Need detailed installation help? Check our platform-specific guides:

Claude Desktop Guide Claude Code Guide Troubleshooting

Related Agentic Skills

deepspeed

by zechenzhangAGI

Expert guidance for distributed training with DeepSpeed - ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, sparse attention

general› learning models model

Marketplace

model-trainer

by kntism

This skill should be used when users want to train or fine-tune language models using TRL (Transformer Reinforcement Learning) on Hugging Face Jobs in...

general› learning models model

CommandsMarketplace

hugging-face-evaluation-manager

by kntism

Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial A...

general› learning models model

CommandsMarketplace

PRFAQ Writer

by thinkbigleaders

**Description**: Write compelling PRFAQ (Press Release + FAQ) documents using Amazon's Working Backwards methodology to develop customer-centric innov...

general› learning models model

Agentic Skill Details

Owner: zechenzhangAGI (GitHub)
Repository: AI-research-SKILLs
Type: Non-Technical
Meta-Domain: general
Primary Domain: general
Sub-Domain: learning models model
Market Score: 27.2

Agentic Skill Grades →

Browse Category

More general Agentic Skills

Report Security Issue

Found a security vulnerability in this agent skill?