alireza/vosk-datacleaner
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

shards

Browse Source
This commit is contained in:
Alireza
2025-08-02 12:26:00 +03:30
parent 3cc3bf0d3e
commit 474245fe83
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
vosk/test_files/batch_confirm_hf.py
View File

@@ -95,7 +95,7 @@ if confirmed:
confirmed_dataset = confirmed_dataset.map(extract_minimal, remove_columns=confirmed_dataset.column_names) confirmed_dataset = confirmed_dataset.map(extract_minimal, remove_columns=confirmed_dataset.column_names)
# Sharding parameters # Sharding parameters
num_shards = min(1, len(confirmed)) # Don't create more shards than samples num_shards = min(25, len(confirmed)) # Don't create more shards than samples
shard_size = len(confirmed_dataset) // num_shards + 1 shard_size = len(confirmed_dataset) // num_shards + 1
# Write each shard separately # Write each shard separately