examples / monitoring

Monitoring

Real-time monitoring examples using gnaw for system health, application logs, and performance metrics.

Log Monitoring

Real-time Error Monitoring

Monitor application logs for errors in real-time:

#!/bin/bash
# monitor-errors.sh
LOG_FILE="/var/log/app.log"
ALERT_THRESHOLD=10
ALERT_EMAIL="admin@example.com"

# Monitor for errors
tail -f "$LOG_FILE" | gnaw --stream "ERROR" | while read line; do
    echo "$(date): $line" >> /var/log/errors.log
    
    # Count errors in the last hour
    ERROR_COUNT=$(gnaw --raw -c "ERROR" /var/log/errors.log)
    
    if [ "$ERROR_COUNT" -gt "$ALERT_THRESHOLD" ]; then
        echo "High error count: $ERROR_COUNT" | mail -s "High Error Count" "$ALERT_EMAIL"
    fi
done

Performance Monitoring

Monitor application performance:

#!/bin/bash
# monitor-performance.sh
LOG_FILE="/var/log/app.log"

# Monitor for slow queries
tail -f "$LOG_FILE" | gnaw --stream "slow query" | while read line; do
    echo "$(date): $line" >> /var/log/slow-queries.log
    
    # Alert if too many slow queries
    SLOW_COUNT=$(gnaw --raw -c "slow query" /var/log/slow-queries.log)
    if [ "$SLOW_COUNT" -gt 5 ]; then
        echo "High slow query count: $SLOW_COUNT" | mail -s "Performance Alert" admin@example.com
    fi
done

Security Monitoring

Monitor for security issues:

#!/bin/bash
# monitor-security.sh
LOG_FILE="/var/log/auth.log"

# Monitor for failed login attempts
tail -f "$LOG_FILE" | gnaw --stream "authentication failed\|login failed" | while read line; do
    echo "$(date): $line" >> /var/log/security.log
    
    # Count failed attempts in the last hour
    FAILED_COUNT=$(gnaw --raw -c "authentication failed\|login failed" /var/log/security.log)
    
    if [ "$FAILED_COUNT" -gt 10 ]; then
        echo "High failed login count: $FAILED_COUNT" | mail -s "Security Alert" security@example.com
    fi
done

System Monitoring

Resource Monitoring

Monitor system resources:

#!/bin/bash
# monitor-resources.sh

# Check disk usage
DISK_USAGE=$(df -h | gnaw -E "[0-9]{2,}%" | wc -l)
if [ "$DISK_USAGE" -gt 0 ]; then
    echo "High disk usage detected" | mail -s "Disk Usage Alert" admin@example.com
fi

# Check memory usage
MEMORY_USAGE=$(free -h | gnaw -E "[0-9]+G" | wc -l)
if [ "$MEMORY_USAGE" -gt 0 ]; then
    echo "High memory usage detected" | mail -s "Memory Usage Alert" admin@example.com
fi

# Check CPU usage
CPU_USAGE=$(top -bn1 | gnaw -E "load average.*[0-9]+\.[0-9]+" | wc -l)
if [ "$CPU_USAGE" -gt 0 ]; then
    echo "High CPU usage detected" | mail -s "CPU Usage Alert" admin@example.com
fi

Service Monitoring

Monitor system services:

#!/bin/bash
# monitor-services.sh

# Check service status
SERVICE_STATUS=$(systemctl status nginx | gnaw -i "active\|failed")
if [ "$SERVICE_STATUS" = "failed" ]; then
    echo "Service failed: nginx" | mail -s "Service Alert" admin@example.com
fi

# Check service logs
SERVICE_LOGS=$(journalctl -u nginx | gnaw -i "error\|warning")
if [ -n "$SERVICE_LOGS" ]; then
    echo "Service logs contain errors: $SERVICE_LOGS" | mail -s "Service Log Alert" admin@example.com
fi

Application Monitoring

API Monitoring

Monitor API endpoints:

#!/bin/bash
# monitor-api.sh
API_LOG="/var/log/api.log"

# Monitor for API errors
tail -f "$API_LOG" | gnaw --stream "HTTP/[0-9]+.*[4-5][0-9][0-9]" | while read line; do
    echo "$(date): $line" >> /var/log/api-errors.log
    
    # Count errors in the last hour
    ERROR_COUNT=$(gnaw --raw -c "HTTP/[0-9]+.*[4-5][0-9][0-9]" /var/log/api-errors.log)
    
    if [ "$ERROR_COUNT" -gt 20 ]; then
        echo "High API error count: $ERROR_COUNT" | mail -s "API Error Alert" admin@example.com
    fi
done

Database Monitoring

Monitor database performance:

#!/bin/bash
# monitor-database.sh
DB_LOG="/var/log/database.log"

# Monitor for slow queries
tail -f "$DB_LOG" | gnaw --stream "slow query\|timeout" | while read line; do
    echo "$(date): $line" >> /var/log/slow-queries.log
    
    # Count slow queries in the last hour
    SLOW_COUNT=$(gnaw --raw -c "slow query\|timeout" /var/log/slow-queries.log)
    
    if [ "$SLOW_COUNT" -gt 5 ]; then
        echo "High slow query count: $SLOW_COUNT" | mail -s "Database Performance Alert" admin@example.com
    fi
done

Network Monitoring

Network Traffic

Monitor network traffic:

#!/bin/bash
# monitor-network.sh
NETWORK_LOG="/var/log/network.log"

# Monitor for network issues
tail -f "$NETWORK_LOG" | gnaw --stream "connection failed\|timeout" | while read line; do
    echo "$(date): $line" >> /var/log/network-issues.log
    
    # Count network issues in the last hour
    ISSUE_COUNT=$(gnaw --raw -c "connection failed\|timeout" /var/log/network-issues.log)
    
    if [ "$ISSUE_COUNT" -gt 10 ]; then
        echo "High network issue count: $ISSUE_COUNT" | mail -s "Network Alert" admin@example.com
    fi
done

Bandwidth Monitoring

Monitor bandwidth usage:

#!/bin/bash
# monitor-bandwidth.sh
BANDWIDTH_LOG="/var/log/bandwidth.log"

# Monitor for high bandwidth usage
tail -f "$BANDWIDTH_LOG" | gnaw --stream "bytes.*[0-9]{6,}" | while read line; do
    echo "$(date): $line" >> /var/log/high-bandwidth.log
    
    # Count high bandwidth usage in the last hour
    HIGH_COUNT=$(gnaw --raw -c "bytes.*[0-9]{6,}" /var/log/high-bandwidth.log)
    
    if [ "$HIGH_COUNT" -gt 5 ]; then
        echo "High bandwidth usage detected: $HIGH_COUNT" | mail -s "Bandwidth Alert" admin@example.com
    fi
done

Integration Examples

Slack Integration

Send alerts to Slack:

#!/bin/bash
# slack-alerts.sh
WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

# Monitor for errors and send to Slack
tail -f /var/log/app.log | gnaw --stream "ERROR" | while read line; do
    curl -X POST -H "Content-Type: application/json" \
         -d "{\"text\":\"🚨 Error detected: $line\"}" \
         "$WEBHOOK_URL"
done

Discord Integration

Send alerts to Discord:

#!/bin/bash
# discord-alerts.sh
WEBHOOK_URL="https://discord.com/api/webhooks/YOUR/WEBHOOK/URL"

# Monitor for critical errors and send to Discord
tail -f /var/log/app.log | gnaw --stream "FATAL\|CRITICAL" | while read line; do
    curl -X POST -H "Content-Type: application/json" \
         -d "{\"content\":\"🚨 Critical error: $line\"}" \
         "$WEBHOOK_URL"
done

Email Integration

Send alerts via email:

#!/bin/bash
# email-alerts.sh
ALERT_EMAIL="admin@example.com"

# Monitor for errors and send email alerts
tail -f /var/log/app.log | gnaw --stream "ERROR" | while read line; do
    echo "Error detected: $line" | mail -s "Error Alert" "$ALERT_EMAIL"
done

Dashboard Integration

Prometheus Metrics

Export metrics to Prometheus:

#!/bin/bash
# prometheus-metrics.sh
METRICS_FILE="/tmp/metrics"

# Generate metrics
ERROR_COUNT=$(gnaw --raw -c "ERROR" /var/log/app.log)
WARN_COUNT=$(gnaw --raw -c "WARN" /var/log/app.log)

# Write to metrics file
echo "# HELP app_errors_total Total number of errors" >> "$METRICS_FILE"
echo "# TYPE app_errors_total counter" >> "$METRICS_FILE"
echo "app_errors_total $ERROR_COUNT" >> "$METRICS_FILE"

echo "# HELP app_warnings_total Total number of warnings" >> "$METRICS_FILE"
echo "# TYPE app_warnings_total counter" >> "$METRICS_FILE"
echo "app_warnings_total $WARN_COUNT" >> "$METRICS_FILE"

Grafana Integration

Create Grafana dashboards:

{
  "dashboard": {
    "title": "Application Monitoring",
    "panels": [
      {
        "title": "Error Count",
        "type": "stat",
        "targets": [
          {
            "expr": "app_errors_total",
            "legendFormat": "Errors"
          }
        ]
      },
      {
        "title": "Warning Count",
        "type": "stat",
        "targets": [
          {
            "expr": "app_warnings_total",
            "legendFormat": "Warnings"
          }
        ]
      }
    ]
  }
}

Best Practices

- Use streaming for real-time monitoring - Set appropriate thresholds - Monitor resource usage - Optimize for your environment - Implement error handling - Use retry mechanisms - Monitor monitoring systems - Document failures - Use secure communication - Validate input data - Monitor for abuse - Implement access controls

Troubleshooting

- Check system resources - Monitor log file sizes - Optimize patterns - Consider parallel processing - Verify webhook URLs - Check authentication - Monitor network connectivity - Test locally first - Check alert thresholds - Verify notification channels - Monitor alert delivery - Test alert systems Start with basic monitoring and gradually add more sophisticated analysis as your needs grow. Monitor the monitoring systems themselves to ensure reliability.