Sentinel

A lightweight process orchestrator CLI

Features

Start and manage background processes with automatic logging
Real-time CPU, memory, and uptime monitoring
Automatic restart on process crash or exit
Process groups for organizing related processes
Size-based log rotation for stdout/stderr logs
Optional HTTP/TCP process health checks
Run processes as specific system users
Port allocation and management
Metrics export in JSON and table formats
Remote process management via SSH
Persistent state across sessions

Installation

For more installation options, see the installation guide in the docs.

Nix

1nix profile add github:4ster-light/sentinel

If you are on NixOS add the flake to your system configuration.

Python (Uv)

1uv tool install git+https://github.com/4ster-light/sentinel

Alternatively using pip:

1pip install git+https://github.com/4ster-light/sentinel

Commands Overview

Command	Description
`sentinel run`	Start a background process
`sentinel list`	List all managed processes
`sentinel status`	Show detailed status of a process
`sentinel stop`	Stop a running process
`sentinel restart`	Restart a process
`sentinel logs`	View process logs
`sentinel clean`	Remove dead processes from state
`sentinel stopall`	Stop all managed processes
`sentinel startall`	Start all stopped processes
`sentinel restartall`	Restart all managed processes
`sentinel daemon`	Manage the restart monitor daemon
`sentinel group`	Manage process groups
`sentinel port`	Manage port allocations
`sentinel startup`	Generate startup scripts
`sentinel metrics`	Export process metrics
`sentinel remote`	Manage remote hosts via SSH

Run sentinel run --help to see process runtime options including --user, --startup-timeout, --instances, --nice, and --ionice.

Process Management

Starting a Process

Start a background process with sentinel run:

1# Basic usage2sentinel run "python server.py"34# Give the process a name for easier reference5sentinel run "python server.py" --name myserver67# Start multiple instances with derived names8sentinel run "python server.py" --name web --instances 3910# Enable auto-restart on crash or exit11sentinel run "npm start" --name frontend --restart1213# Add process to a group14sentinel run "python worker.py" --name worker1 --group workers1516# Use environment variables from a file17sentinel run "node app.js" --name app --env-file .env1819# Run as a specific system user (name or uid)20sentinel run "python worker.py" --name worker --user deploy2122# Add an HTTP health check (auto-restart still requires --restart)23sentinel run "python api.py" --name api --restart --health-http http://127.0.0.1:8000/health2425# Add a TCP health check with custom thresholds26sentinel run "python worker.py" --name worker --restart --health-tcp 127.0.0.1:9000 --health-interval 10 --health-failures 2

When you use --restart without the daemon running, you will see a warning:

1Started myserver (id: 1, pid: 12345)2Warning: Restart flag set but daemon is not running. Restarts will only happen3when you run other sentinel commands.4  Run 'sentinel daemon start' for continuous monitoring.

Startup Script Generation

Generate a minimal systemd unit for a Sentinel-managed command:

1sentinel startup systemd --name web --cwd /srv/web --restart python app.py

Output:

1[Unit]2Description=Sentinel process: web3After=network.target45[Service]6Type=simple7WorkingDirectory=/srv/web8ExecStart=/usr/bin/env python app.py9Restart=always1011[Install]12WantedBy=multi-user.target

Listing Processes

View all managed processes:

1sentinel list

Output shows a table with ID, name, PID, status, CPU usage, memory usage, uptime, restart flag, group, and command:

1+----+----------+-------+---------+------+-------+--------+---------+---------+-------+-----------+2| ID | NAME     |   PID | STATUS  |  CPU |   MEM | UPTIME | RESTART | USER    | GROUP | COMMAND   |3+----+----------+-------+---------+------+-------+--------+---------+---------+-------+-----------+4|  1 | myserver | 12345 | running | 2.1% | 45 MB |   5m 3s|    -    |    -    |   -   | python ...|5|  2 | frontend | 12346 | running | 0.5% | 120MB |   2m 1s|    X    | deploy  |   -   | npm start |6+----+----------+-------+---------+------+-------+--------+---------+---------+-------+-----------+

Process Status

Get detailed information about a specific process:

1# By name2sentinel status myserver34# By ID5sentinel status 1

Output:

1myserver (id: 1)2  PID:       123453  Status:    running4  CPU:       2.1%5  Memory:    45.2MB6  Uptime:    5m 32s7  Restart:   no8  User:      default9  Group:     none10  CWD:       /home/user/project11  Command:   python server.py12  Stdout:    /home/user/.sentinel/logs/myserver.stdout.log13  Stderr:    /home/user/.sentinel/logs/myserver.stderr.log

Stopping Processes

Stop a running process:

1# By name2sentinel stop myserver34# By ID5sentinel stop 167# Force kill (SIGKILL instead of SIGTERM)8sentinel stop myserver --force

Stopping signals the process's whole process group, so children of compound commands (npm run build && node server.js, background jobs) die with the leader.

Restarting Processes

Restart a process (stops and starts it again):

1sentinel restart myserver

Viewing Logs

View stdout and stderr logs for a process:

1# Show last 50 lines (default)2sentinel logs myserver34# Show last 100 lines5sentinel logs myserver --lines 10067# Follow log output in real-time8sentinel logs myserver --follow910# Show only stdout11sentinel logs myserver --stream stdout1213# Show only stderr14sentinel logs myserver --stream stderr1516# Clear logs17sentinel logs myserver --clear

Sentinel rotates process logs automatically when they exceed 10MB, keeping up to 3 backup files per stream (.1, .2, .3).

Bulk Operations

Control all processes at once:

1# Stop all processes2sentinel stopall34# Force stop all processes5sentinel stopall --force67# Start all stopped processes8sentinel startall910# Restart all processes11sentinel restartall

Cleaning Up

Remove dead processes from the state file:

1sentinel clean

This removes processes whose PIDs no longer exist from the state, without affecting running processes.

Auto-Restart

Sentinel can automatically restart processes that crash or exit. There are two ways this works:

Lazy Restart (Default)

When you run certain CLI commands (list, status), Sentinel checks for dead processes and restarts those with the restart flag enabled. This happens automatically without any extra setup.

Example:

1# Start a process with restart flag2sentinel run "python worker.py" --name worker --restart34# If the process crashes, the next time you run list, it will be restarted5sentinel list6# Output: Auto-restarted worker (old_pid: 12345, new_pid: 12350)

Daemon Mode (Continuous Monitoring)

For continuous monitoring without needing to run CLI commands, start the daemon:

1# Start the daemon2sentinel daemon start3# Output: Started daemon (pid: 54321)45# Check daemon status6sentinel daemon status7# Output: Daemon is running (pid: 54321)89# Stop the daemon10sentinel daemon stop11# Output: Stopped daemon (pid: 54321)

When the daemon is running, it checks every 5 seconds for crashed processes and restarts them automatically in the background. The daemon writes its activity to ~/.sentinel/daemon.log (rotated automatically).

Process Groups

Groups let you organize related processes together.

Creating Groups

1# Create a group2sentinel group create workers34# Create a group with environment variables5sentinel group create production --env "NODE_ENV=production" --env "PORT=3000"

Managing Group Membership

1# Add a process to a group (by process ID)2sentinel group add workers 134# Remove a process from a group5sentinel group remove workers 1

Listing Groups

1# List all groups2sentinel group list34# List processes in a specific group5sentinel group list workers

Group Operations

Control all processes in a group:

1# Start all processes in a group2sentinel group start workers34# Stop all processes in a group5sentinel group stop workers67# Restart all processes in a group8sentinel group restart workers

Deleting Groups

1# Delete a group (processes are unassigned but not stopped)2sentinel group delete workers34# Delete a group and stop all its processes5sentinel group delete workers --with-processes67# Alias for --with-processes8sentinel group delete workers --stop

Port Management

Sentinel can allocate and track ports for your processes.

Allocating Ports

1# Allocate a random available port2sentinel port allocate3# Output: Allocated port 8234 (default)45# Allocate a specific port6sentinel port allocate --port 800078# Allocate with a name9sentinel port allocate --name myapp10# Output: Allocated port 9123 (myapp)

Listing Ports

1sentinel port list

Output:

1+-------+---------+------------------+2|  PORT | NAME    | ALLOCATED        |3+-------+---------+------------------+4|  8000 | myapp   | 2024-01-15 10:30 |5|  8234 | default | 2024-01-15 10:35 |6+-------+---------+------------------+

Freeing Ports

1sentinel port free 8000

Metrics Export

Export process metrics in different formats for monitoring and automation.

Exporting Metrics

1# Show metrics as a table (default)2sentinel metrics export34# Export as JSON5sentinel metrics export --format json67# Write metrics to a file8sentinel metrics export --output metrics.json910# Export JSON to a file11sentinel metrics export --format json --output metrics.json

Output (table format):

1+----+----------+-------+---------+------+-------+--------+2| ID | NAME     |   PID | STATUS  |  CPU |   MEM | UPTIME |3+----+----------+-------+---------+------+-------+--------+4|  1 | myserver | 12345 | running | 2.1% | 45 MB |   5m 3s|5|  2 | frontend | 12346 | running | 0.5% | 120MB |   2m 1s|6+----+----------+-------+---------+------+-------+--------+

Output (JSON format):

1[2  {3    "id": 1,4    "name": "myserver",5    "pid": 12345,6    "status": "running",7    "running": true,8    "cpu_percent": 2.1,9    "memory_mb": 45.2,10    "uptime_seconds": 303.011  }12]

Quick Snapshot

Print a one-shot metrics table to stdout:

1sentinel metrics snapshot

This is equivalent to sentinel metrics export without options.

Remote Management

Manage processes on remote hosts via SSH. Remote hosts must have Sentinel installed and accessible via SSH.

Registering Remote Hosts

1# Add a remote host2sentinel remote add server1.example.com34# Add with SSH user5sentinel remote add server1.example.com --user deploy67# Add with custom SSH port8sentinel remote add server1.example.com --user deploy --port 2222

Listing Remote Hosts

1# List all registered remote hosts2sentinel remote list

Output:

1+---------------------+--------+------+------------------+2| HOST                | USER   | PORT | CREATED          |3+---------------------+--------+------+------------------+4| server1.example.com | deploy | 2222 | 2024-01-15 10:30 |5| server2.example.com | -      | -    | 2024-01-15 10:35 |6+---------------------+--------+------+------------------+

Running Commands on Remote Hosts

1# List processes on a remote host2sentinel remote list server1.example.com34# Start a process on a remote host5sentinel remote run server1.example.com "python server.py"67# Stop a process on a remote host (use the ID shown by `remote list`)8sentinel remote stop server1.example.com 1

Removing Remote Hosts

1sentinel remote remove server1.example.com

Configuration

Sentinel stores its state in ~/.sentinel/ by default (override the directory with the SENTINEL_STATE_DIR environment variable):

state.json - Process registry, port allocations, and remote hosts. Written atomically and guarded against corruption.
state.json.lock - Advisory lock coordinating concurrent Sentinel writers (the daemon and CLI commands).
logs/ - Process stdout and stderr logs
daemon.pid - PID file for the restart daemon
daemon.log - Rotating log of daemon activity (restarts, errors)

If state.json is ever unreadable, Sentinel backs it up to state.json.corrupt-, prints a warning, and starts from an empty state instead of silently discarding your data.

Environment Variables

Process Environment

You can pass environment variables to processes:

1# Using an env file2sentinel run "node app.js" --env-file .env

Sentinel also looks for environment files in:

~/.sentinel/.env (global)
./.env (current directory)

Process User

Run commands as a specific local system user:

1# Run by username2sentinel run "python worker.py" --name worker --user deploy34# Run by uid5sentinel run "python worker.py" --name worker --user 1001

Notes:

Switching to another user generally requires root privileges.
If you run Sentinel without root, use your current user.

Group Environment

Groups can have environment variables that are passed to all processes in the group:

1sentinel group create staging --env "DATABASE_URL=postgres://..." --env "DEBUG=true"

Getting Help

View available commands:

1sentinel --help

Get help for a specific command:

1sentinel run --help2sentinel daemon --help3sentinel group --help4sentinel port --help5sentinel metrics --help6sentinel remote --help

License

MIT