Progress Tracking & Resumable Batches

Large batches fail partway through — rate limits, network blips, machine restarts. A batch that cannot resume from where it stopped wastes every successful call when restarted. Resumable batches require a progress record: a file that tracks which items have been processed successfully. Before processing each item, check if it is already in the progress file. After successful processing, append it. On restart, the batch skips completed items and continues from the first unprocessed one.

Progress tracking also enables monitoring. A simple progress file lets you compute percentage complete, estimate time remaining, and detect stalls. For long-running batches (hundreds of files), display a progress bar or periodic status updates. The combination of resumability and visibility transforms a fragile, opaque batch into a production-grade pipeline that operators can monitor and trust.

#!/bin/bash
# Resumable batch with progress tracking

PROGRESS_FILE=".batch-progress"
OUT_DIR="results"
touch "$PROGRESS_FILE"
mkdir -p "$OUT_DIR"

# Build file list
mapfile -t FILES < <(find src/ -name "*.ts" | sort)
TOTAL=${#FILES[@]}
DONE=$(wc -l < "$PROGRESS_FILE")

echo "Batch: $TOTAL total, $DONE already done, $((TOTAL - DONE)) remaining"

for file in "${FILES[@]}"; do
  # Skip if already processed
  if grep -qF "$file" "$PROGRESS_FILE"; then
    continue
  fi

  name=$(basename "$file" .ts)
  echo "[$((DONE + 1))/$TOTAL] $name..."

  if gemini -p "Analyze $file for code quality issues" --output-format json \
    > "$OUT_DIR/$name.json" 2>/dev/null; then
    echo "$file" >> "$PROGRESS_FILE"
    ((DONE++))
  else
    echo "  FAILED: $name (will retry on next run)"
    sleep 5  # Back off on failure
  fi
done

echo "\nComplete: $DONE/$TOTAL processed"