Performance

gnaw is designed for high-performance search across large codebases and datasets.

Performance Characteristics

Multi-threaded Processing

gnaw leverages all available CPU cores for parallel processing:

• Parallel File Processing: Multiple files processed simultaneously
• Thread Pool Management: Optimized thread allocation and management
• Load Balancing: Intelligent work distribution across cores

Memory Optimization

Results streamed as they're found, reducing memory usage Smart buffering strategies for different file sizes Minimal memory allocation and efficient cleanup Intelligent caching for repeated searches

Performance Tuning

File Size Optimization

gnaw automatically optimizes for different file sizes:

# gnaw.toml
[tuning]
tiny   = "64 KiB"    # Small files: minimal overhead
small  = "8.6 MiB"   # Medium files: balanced approach
medium = "256 MiB"   # Large files: streaming optimization
large  = "2 GiB"     # Very large files: chunked processing

Regex Engine Configuration

Optimize regex performance for different pattern complexities:

# gnaw.toml
[regex.simple]
size_limit_mb = 200      # Maximum memory for simple patterns
dfa_size_limit_mb = 200  # DFA size limit
nest_limit = 250         # Maximum nesting depth

[regex.complex]
size_limit_mb = 100      # Reduced limits for complex patterns
dfa_size_limit_mb = 100
nest_limit = 200

I/O Optimization

# gnaw.toml
# Size of each I/O chunk (in bytes)
io_chunk_size_bytes = 8388608  # 8 MB

# Number of lines to buffer before streaming
chunk_size = 100

Benchmarking Results

vs grep Performance

gnaw consistently outperforms grep on large files:

| File Size | grep Time | gnaw Time | Speedup | |-----------|-----------|-----------|---------| | 100 MB | 2.3s | 0.8s | 2.9x | | 1 GB | 23.1s | 7.2s | 3.2x | | 10 GB | 4m 12s | 1m 8s | 3.7x |

Memory Usage

gnaw uses significantly less memory than grep:

| File Size | grep Memory | gnaw Memory | Reduction | |-----------|-------------|------------|-----------| | 100 MB | 45 MB | 12 MB | 73% | | 1 GB | 180 MB | 35 MB | 81% | | 10 GB | 1.2 GB | 120 MB | 90% |

Performance Monitoring

Built-in Profiling

Enable performance monitoring:

# Enable debug logging
RUST_LOG=debug gnaw "pattern" file.txt

# Profile with flamegraph
cargo flamegraph --release --bin gnaw -- "pattern" file.txt

Custom Benchmarking

Create performance tests:

# Generate test data
cargo run --bin generate_test_logs

# Run benchmarks
./scripts/bench_compare.sh

# View results
cd dashboard
streamlit run streamlit_app.py

Optimization Strategies

For Large Files

1. Use Streaming: Enable --stream for real-time results
2. Chunk Processing: Adjust io_chunk_size_bytes for your hardware
3. Memory Limits: Set appropriate regex engine limits

For Large Codebases

1. Build Index: Use gnaw agent index build for semantic search
2. Filter Results: Use --type and --dir flags to narrow scope
3. Parallel Processing: Ensure sufficient CPU cores are available

For CI/CD Integration

1. JSON Output: Use --json for programmatic consumption
2. Exit Codes: Leverage exit codes for automation
3. Resource Limits: Set appropriate memory and time limits

Hardware Recommendations

More cores = better performance. gnaw scales well with CPU core count. Minimum 4GB RAM recommended. 8GB+ for large codebases. SSD recommended for better I/O performance, especially with large files.

Troubleshooting Performance

- Check if ignore rules are too restrictive - Verify file permissions - Consider using `--no-ignore` for debugging - Reduce `io_chunk_size_bytes` - Lower regex engine limits - Use streaming output - Check thread count with `htop` - Verify multi-threading is enabled - Consider system load For best performance, use traditional search for simple patterns and AI search for complex semantic queries.