Educational Implementation · Python

tokenizer-hi

Tokenizer for Hindi language.

Vraj Patel@vrajpat3ll · Atlas 61.7 · #14= builder

What it is

tokenizer-hi is a course-assignment repository implementing a byte-level BPE tokenizer for Hindi from the Python standard library, with corpus building, training, encode/decode, dataset fetching via kagglehub, and pickle serialization of the trained vocab and merges.

Scored ata4962a14 Nov 2026Changes since

Merit describes the code at this commit. Later pushes count once the owner rescans. How merit is scored

Where it counts

NLP & LLMs85%
Machine Learning35%
Data Science10%
MLOps10%

How much of the project is about each field, as scoring counts it. It ranks in the field it is most about.

Around it on the NLP & LLMs board

Whole board

RankProjectBuilderFieldMerit

70, shared rankbol-tools-v2AI Developer Tool · PythonThe repository provides a suite of tools for probing and analyzing the internal representations of pre-trained transformer language models to investigate the Octopus Theory (whether models can understand language without generating it).50.4/100ImplementationRickNLP & LLMs
70, shared rankchk-livive-netML Application · JavaScriptThe repository provides a browser-based checkers game with alpha-beta Minimax AI and WebGPU-powered LLM personas, accompanied by a Godot 4 3D frontend and automated testing infrastructure.50.4/100ImplementationJarad PeckenpaughNLP & LLMs
72tokenizer-hiEducational Implementation · Python50.2/100ImplementationVraj PatelNLP & LLMs
73ollama-webuiML Application · SvelteOllama Web UI is a full-stack web application that provides a user-friendly interface for interacting with large language models via Ollama or OpenAI-compatible APIs, featuring capabilities such as chat, retrieval-augmented generation (RAG), model management, and document…50.0/100ImplementationPetr SovadinaNLP & LLMs
74gixML Application · PythonGiX is a command-line tool that automates git commits and pushes using manual, interval-based, or voice-activated modes, with optional LLM-generated commit messages.49.5/100ImplementationVraj PatelNLP & LLMs

What it is

Where it counts

Around it on the NLP & LLMs board

More from Vraj Patel