Educational Implementation · C++

byte-pair-encoding

naive implementation of the byte pair encoding algorithm used for tokenization in LLMs

Vraj Patel@vrajpat3ll · Atlas 61.7 · #14= builder

What it is

This repository provides a self-contained C++ implementation of the Byte Pair Encoding (BPE) tokenization algorithm, including a driver program that trains BPE on a text corpus and outputs the merged pairs. A companion Python script automates repeated compilation and execution for demonstration.

Scored atd0c50224 Nov 2026Changes since

Merit describes the code at this commit. Later pushes count once the owner rescans. How merit is scored

Where it counts

NLP & LLMs90%
Machine Learning85%
Data Science20%

How much of the project is about each field, as scoring counts it. It ranks in the field it is most about.

Around it on the NLP & LLMs board

Whole board

RankProjectBuilderFieldMerit

60doc-assistant-hfML Application · Jupyter NotebookThe repository provides a document intelligence notebook that fine-tunes a BART summarizer with QLoRA, runs extractive QA using a Longformer model with custom span decoding, and implements a TextRank-based extractive summarizer, all wrapped in a Gradio interface.53.7/100ImplementationPop AlexandruNLP & LLMs
61promptlintAI Developer Tool · Pythonpromptlint is a static analysis linter for LLM prompts that detects common anti-patterns such as prompt injection risks, missing token limits, and hardcoded secrets.53.6/100ImplementationPop AlexandruNLP & LLMs
62byte-pair-encodingEducational Implementation · C++53.3/100ImplementationVraj PatelNLP & LLMs
63llm-trainerML Application · JavaScriptllm-trainer is a full-stack application that provides a graphical interface for browsing Hugging Face datasets and fine-tuning local LLMs. It consists of a React frontend, an Express backend with WebSocket support, and a Python training worker that uses the Hugging Face…53.0/100ImplementationMaster MindNLP & LLMs
64, shared rankdisposition-resultsBenchmarking Project · PythonThis repository hosts and visualizes results from the Disposition benchmark, providing tools to validate submissions and generate comparative charts of language models across seven dispositional traits.52.6/100Implementationjerry howellNLP & LLMs

What it is

Where it counts

Around it on the NLP & LLMs board

More from Vraj Patel