Skip to Content
OperateNode Operations Guide

SHE Node Operations

This guide covers the detailed operational aspects of running a SHE node, including configuration management, maintenance procedures, and best practices for stable and performant operations.

Configuration Management

Directory Structure

The SHE node configuration is stored in ~/.she/config/:

~/.she/config/
├── app.toml          # Application configuration (gas fees, API settings, pruning, etc.)
├── client.toml       # CLI and client-related settings
├── config.toml       # Core Tendermint settings (network, consensus, and RPC)
├── genesis.json      # Chain genesis file, defines initial state
├── node_key.json     # Unique node identity key for peer-to-peer (P2P) networking
└── priv_validator_key.json  # Validator private signing key (if running as a validator)

Essential Configuration Parameters

Network Settings (config.toml)

[p2p]
# Public IP for other nodes to reach you
external_address = "your-public-ip:26656"
# Local address to listen for incoming P2P connections
laddr = "tcp://0.0.0.0:26656"
# Number of peers allowed
max_num_inbound_peers = 40
max_num_outbound_peers = 20
# Network bandwidth limits to prevent congestion
send_rate = 20480000  # 20MB/s
recv_rate = 20480000  # 20MB/s
 
[rpc]
# RPC listen address
laddr = "tcp://0.0.0.0:26657"
# Maximum number of simultaneous connections
max_open_connections = 900
# Transaction confirmation timeout
timeout_broadcast_tx_commit = "10s"

Application Settings (app.toml)

# Minimum gas prices [to prevent spam transactions]
minimum-gas-prices = "0.01ushe"
 
[api]
# Enable the API server
enable = true
max-open-connections = 1000
 
[state-commit]
# Use SheDB for improved performance
sc-enable = true
 
[state-store]
# Enable state store for historical queries
ss-enable = true
# Retain 100,000 blocks for queryability
# 0 = "keep all"
ss-keep-recent = 100000

Database Management

Database Types

SHE supports two database backends:

  1. SheDB (Recommended)

    • Optimized for performance and sync times
    • Reduces resource usage
    • Best for all nodes

  2. Legacy IAVL DB

    • Standard Cosmos SDK database
    • More widely tested

SHE-DB Configuration

[state-commit]
sc-enable = true
sc-async-commit-buffer = 100
sc-keep-recent = 1  # Keep only the most recent state for performance
sc-snapshot-interval = 10000  # Take state snapshots every 10,000 blocks
 
[state-store]
ss-enable = true
ss-backend = "pebbledb"  # Default, required
ss-async-write-buffer = 100
ss-keep-recent = 100000  # Keep last 100,000 blocks
ss-prune-interval = 600  # Cleanup interval for pruning

Setting very small [more frequent] pruning intervals may cause issues with automated snapshotting in the event those events collide. Too large [less frequent] pruning intervals means it will take a longer overall time to prune which may cause missed blocks and excessive resync time.

Database Maintenance

The database is typically stable and can be left alone, although some attention may be required:

# Check database size
du -sh ~/.she/data/
 
# Confirm correct file paths as per your own setup before proceeding
# Before performing a database wipe, ensure you back up essential files:
# - `node_key.json`: Preserves node identity to maintain peer connections.
# - `priv_validator_key.json`: Critical for validators; losing this key results in double signing risks.
# - `config.toml` & `app.toml`: Retains node-specific configuration.
# - `genesis.json`: Required for correct chain initialization.
 
# Compact database to optimize storage (only if needed)
find ~/.she/data/ -mindepth 1 ! -name 'priv_validator_state.json' -delete && rm -rf ~/.she/wasm
 
 
# Backup database
cp -r ~/.she/data/ ~/she-backup-$(date +%Y%m%d)/

Service Management

Systemd Commands

# Check service status
systemctl status shed
 
# Start service
systemctl start shed
 
# Stop service
systemctl stop shed
 
# Restart service
systemctl restart shed
 
# View logs in real time
journalctl -fu shed -o cat

Log Management

Prevent logs from consuming excessive disk space by enabling rotation:

sudo tee /etc/logrotate.d/she > /dev/null << EOF
/var/log/she/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 she she
    sharedscripts
    postrotate
        systemctl reload shed
    endscript
}
EOF

Update Procedures

Minor Updates

For minor updates that are non-consensus-breaking:

# Stop node
sudo systemctl stop shed
 
# Update binary
cd she-chain
git fetch --all
git checkout [new-version]
make install
 
# Restart node
sudo systemctl restart shed

Major Updates

For major upgrades that introduce state-breaking changes:

  1. Wait for the designated upgrade block height [this can be seen in the upgrade proposal under ‘plan’]
  2. The node will halt automatically.
  3. Update/replace the binary
  4. Restart the node.
# After node halts
cd she-chain
git pull
git checkout [new-version]
make install
sudo systemctl restart shed

Tip - build the upgrade before the halt-height so you can quickly replace it with minimal downtime

Performance Optimization

Performance optimizations can yield different results depending on your system’s hardware, workload, and network conditions. Before implementing any changes, research and test them in a controlled environment to ensure they align with your specific configuration and requirements. Always back up important data before making modifications.

Memory Management (sysctl tuning)

Optimizing memory management settings can help improve performance and stability, particularly for high-load nodes. These settings control swap usage and the handling of dirty (unwritten) pages in RAM.

vm.swappiness = 1  # Reduce swapping to disk, ensuring RAM is used efficiently before relying on slower swap space
vm.dirty_background_ratio = 3  # The percentage of system memory that can be filled with "dirty" pages before background writing begins
vm.dirty_ratio = 10  # The maximum percentage of system memory that can be filled with "dirty" pages before a full flush is triggered
vm.dirty_expire_centisecs = 300  # Time (in hundredths of a second) before dirty data is written to disk
vm.dirty_writeback_centisecs = 100  # Frequency (in hundredths of a second) at which the system writes "dirty" pages to disk

Network Stack Optimization

Tuning the network stack can enhance packet processing efficiency and throughput, particularly for nodes handling a large number of peers and high transaction volume.

net.core.somaxconn = 32768  # max connections in queued
net.core.netdev_max_backlog = 32768  # packets allowed in queue before dropping incoming packets
net.ipv4.tcp_max_syn_backlog = 16384  # outstanding SYN requests (half-open connections) allowed before dropping new connections
net.core.rmem_max = 16777216  # receive buffer size for network sockets
net.core.wmem_max = 16777216  # send buffer size for network sockets

Storage Optimization

Optimizing storage settings can significantly reduce write latency and improve database performance, especially for nodes using NVMe SSDs.

echo "none" > /sys/block/nvme0n1/queue/scheduler  # disable I/O scheduler to reduce latency
blockdev --setra 4096 /dev/nvme0n1  # readahead value to optimize sequential reads

Backup and Recovery

Regular Backups

Automate backups to avoid data loss:

#!/bin/bash
BACKUP_DIR="/backup/she"
DATE=$(date +%Y%m%d)
 
# Stop service
systemctl stop shed
 
# Create backup
tar czf $BACKUP_DIR/she-backup-$DATE.tar.gz ~/.she/
 
# Restart service
systemctl start shed

Recovery Procedure

Restoring from backup in case of corruption or accidental deletion:

# Stop node
systemctl stop shed
 
# Remove corrupted data
rm -rf ~/.she/data/
 
# Restore from backup
tar xzf she-backup-[date].tar.gz -C /
 
# Restart node
systemctl start shed

Troubleshooting

Common Issues and Fixes

  1. Sync Problems

    • Check available disk space (df -h)
    • Ensure proper peer connections (shed net info)
    • Verify firewall settings (port 26656 open)

  2. Performance Issues

    • Monitor system resources (htop or iotop)
    • Check disk I/O performance (iostat)
    • Analyze network traffic (iftop)

  3. Database Issues

    • Run database integrity checks using:

      shed debug dump-db | grep -i error

      If errors are detected, consider restoring from a recent backup.

    • Consider pruning excessive historical data by adjusting ss-keep-recent in app.toml or running:

      shed unsafe-reset-all --home=$HOME/.she --keep-addr-book

      Alternatively, manually remove old state snapshots to free up space:

      rm -rf ~/.she/data/snapshots/*

Diagnostic Commands

# Check sync status
shed status | jq .SyncInfo
 
# View peer count
shed net info | jq '.n_peers'
 
# Check validator status
shed query staking validator $(shed keys show -a $VALIDATOR_KEY)
 
# View recent blocks
shed query blocks recent

Security Considerations

  • Use firewalls and rate-limiting to prevent attacks
  • Keep your system and node software updated
  • Secure SSH access with key-based authentication
  • Protect validator keys with offline storage or hardware security modules (HSMs)

For more in-depth system and configuration guidelines, refer to the [Advanced Configuration and Monitoring Guide](./advanced-config-monitoring).

Last updated on
HomeValidator Operations Guide