Skip to content
#

quality-filtering

Star

Here are 4 public repositories matching this topic...

Language: All
Filter by language
All 4 Python 2 C++ 1 Jupyter Notebook 1

JochiRaider / sievio

Star 2

Sievio is a Python toolkit that streams GitHub, local repositories, and other text/code sources into clean JSONL corpora for LLM pre-training, fine-tuning, or RAG. It includes structure-aware chunking, robust Unicode decoding, pluggable quality and safety screening, and optional dataset card generation and deduplication support.

python data-deduplication dataset-creation data-pipelines repository-mining jsonl github-repos rag text-preprocessing quality-filtering code-mining llm llm-training llm-datasets
  • Updated Dec 15, 2025
  • Python

Improve this page

Add a description, image, and links to the quality-filtering topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the quality-filtering topic, visit your repo's landing page and select "manage topics."

Learn more