fix: query Horizon inside the container and flag lost core connection

- Fetch Horizon via curl inside the container (127.0.0.1:8000) instead of the host port 31401; the host->bridge forwarding hung until the 5 s timeout and made each cycle take ~6 s instead of ~1 s - Report an error when Horizon's core_latest_ledger is 0 (no connection to stellar-core) instead of a false "Synced" with a negative lag - Compute the ingest lag against stellar-core's own ledger - "History Oldest" now shows history_elder_ledger; the latest ledger is listed separately as "History Latest" Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat: config file for refresh interval, cycle timing warning, persistent cbreak
2026-09-23 08:05:10 +00:00 · 2026-09-23 08:05:04 +00:00
3 changed files with 153 additions and 52 deletions
@@ -14,11 +14,29 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ./pi-stat -g / --geo             # With geolocation (implies -p)
 ./pi-stat -s / --show-ip         # Show IP column (hidden by default)
 ./pi-stat -x / --extended        # Show full RX/TX/RX-rate/TX-rate columns (default: ratio only)
-./pi-stat -i [SEC] / --interactive [SEC]  # Interactive auto-refresh (default: 10s)
+./pi-stat -i [SEC] / --interactive [SEC]  # Interactive auto-refresh (default: 5s, configurable)
 ```

 No build step. Requires `pip install docker` and a running Docker daemon. Runs on Linux, WSL2, and Windows.

+## Configuration
+
+Optional JSON config file, first match wins: `$PI_STAT_CONFIG` → `~/.config/pi-stat/config.json` → `<script dir>/pi-stat.json` (see `pi-stat.json`, shipped with the repo). Loaded by `load_config()`; missing, unreadable or non-object files fall back to defaults with a warning on stderr.
+
+| Key | Default | Meaning |
+| --- | --- | --- |
+| `refresh_interval` | `5` | Interactive auto-refresh interval in seconds (clamped to >= 1) |
+
+Precedence for the refresh interval: `-i SEC` on the command line > config file > `DEFAULT_REFRESH` (5).
+
+Standard JSON has no comment syntax; document settings with a `_comment` key instead (any key not read by `config_refresh()` is ignored), e.g.:
+```json
+{
+  "_comment": "refresh_interval in seconds, min. 1 (default: 5)",
+  "refresh_interval": 5
+}
+```
+
 ## Architecture

 Single self-contained script (`pi-stat`). Logic is split into helper functions and a `run_cycle()` function that collects data and renders one full screen, called either once (normal mode) or in a loop (interactive mode).
@@ -27,7 +45,7 @@ Single self-contained script (`pi-stat`). Logic is split into helper functions a

 - **Docker SDK** — container CPU/RAM stats; all commands run inside the auto-detected container (`mainnet` → `testnet2` → `testnet`, first running one) via `exec_run`
 - **Stellar Core HTTP API** — `exec_json()` runs `stellar-core http-command <cmd>` inside the container
- **Horizon API** — `http://localhost:31401`
+- **Horizon API** — `curl http://127.0.0.1:8000/` run inside the container via `exec_json()`, not via the host port `31401` (host→bridge forwarding can hang until timeout if Docker's host firewall rules are broken)
 - **PostgreSQL** — `psql` inside the container; used for peer DB counts and pubkey→IP mapping
 - **ZFS CLI** — `zpool`/`zfs` commands on the host for disk health (`z01pool`); if `zpool` is not found (`HAS_ZFS = False`), falls back to `shutil.disk_usage()` — works on Linux without ZFS, WSL2, and Windows
 - **ip-api.com** — batch geolocation (Country, City), cached in `_geo_cache`; only fetched with `-g`
@@ -49,6 +67,8 @@ Single self-contained script (`pi-stat`). Logic is split into helper functions a

 Outbound peers are numbered in the table. Full node IDs are fetched via `peers?fullkeys=true` for the `droppeer` API call.

+The refresh deadline is anchored to the start of the cycle (`cycle_start`), not its end, so the update's own execution time doesn't stack on top of `refresh_interval`. Execution time is measured and a warning is printed (into the same buffered screen) when a cycle takes >= `REFRESH_WARN_RATIO` (80%) of the interval, or exceeds it outright.
+
 ### Key patterns

 - **Graceful degradation** — every external call is wrapped in try-except; missing data renders as `—`
@@ -57,3 +77,6 @@ Outbound peers are numbered in the table. Full node IDs are fetched via `peers?f
 - **Validator matching** — `load_validators()` parses `stellar-core.cfg` and indexes by IP, name, and public key; matched peers show a truncated key + `(Vn)` label
 - **Geo caching** — `_geo_cache` is a module-level dict; `geo_batch()` only fetches IPs not already cached
 - **Flag parsing** — manual `sys.argv` parsing; `-g` implies `-p`; `-i` accepts an optional next argument as the refresh interval
+- **Config precedence** — `config_refresh()` reads the config value; an explicit `-i SEC` argument overrides it
+- **Cycle timing** — `interactive_loop()` measures cycle duration and warns via `warn()` when it nears or exceeds `refresh_interval`, so a slow update doesn't silently push out the actual refresh cadence
+- **Persistent cbreak** — the tty is switched to cbreak once for the whole session (not reset to canonical between cycles); doing it per-cycle would flush unread input on the mode switch and leave the terminal line-buffered during `run_cycle()`, silently dropping keypresses (e.g. `q`) made while a cycle overruns `refresh_interval`
@@ -38,8 +38,15 @@ except ImportError:
 # --- Configuration ---
 HAS_ZFS              = shutil.which("zpool") is not None
 CANDIDATE_CONTAINERS = ["mainnet", "testnet2", "testnet"]
-HORIZON_URL          = "http://localhost:31401"
+HORIZON_URL          = "http://127.0.0.1:8000"  # queried from inside the container, not via the host port
 CFG_PATH             = os.path.join(os.path.dirname(os.path.realpath(__file__)), "stellar-core.cfg")
+DEFAULT_REFRESH      = 5
+REFRESH_WARN_RATIO   = 0.8  # warn if a cycle takes >= 80% of the refresh interval
+CONFIG_PATHS         = [
+    os.environ.get("PI_STAT_CONFIG"),
+    os.path.join(os.path.expanduser("~"), ".config", "pi-stat", "config.json"),
+    os.path.join(os.path.dirname(os.path.realpath(__file__)), "pi-stat.json"),
+]

 # --- Colors / Icons ---
 GREEN  = "\033[0;32m"
@@ -144,6 +151,33 @@ def geo_str(ip):
        return ", ".join(p for p in geo if p)
    return ""

+def load_config():
+    """Read the first existing config file (JSON object). Returns {} if none is usable."""
+    for path in CONFIG_PATHS:
+        if not path or not os.path.isfile(path):
+            continue
+        try:
+            with open(path, "r", encoding="utf-8") as fh:
+                data = json.load(fh)
+            if isinstance(data, dict):
+                return data
+            print(f"WARN: Config {path}: kein JSON-Objekt – wird ignoriert", file=sys.stderr)
+        except Exception as e:
+            print(f"WARN: Config {path} nicht lesbar: {e}", file=sys.stderr)
+        return {}
+    return {}
+
+
+def config_refresh(cfg):
+    """refresh_interval from config, clamped to >= 1 s; falls back to DEFAULT_REFRESH."""
+    val = cfg.get("refresh_interval", DEFAULT_REFRESH)
+    try:
+        return max(1, int(val))
+    except (TypeError, ValueError):
+        print(f"WARN: refresh_interval '{val}' ungültig – benutze {DEFAULT_REFRESH}s", file=sys.stderr)
+        return DEFAULT_REFRESH
+
+
 def load_validators(cfg_path):
    """Parse [[VALIDATORS]] sections, returning {ip_or_name: (index, public_key)}."""
    if not os.path.exists(cfg_path):
@@ -228,12 +262,12 @@ def run_cycle(client, peers_extended, geo_enabled, show_ip, interactive=False, e
        if raw:
            core_info = raw.get("info")

+    # Query Horizon via curl inside the container: independent of the host's
+    # port forwarding (31401), which can hang until timeout if Docker's
+    # firewall rules on the host are broken.
    hz_data = None
-    try:
-        with urlopen(HORIZON_URL + "/", timeout=5) as resp:
-            hz_data = json.loads(resp.read())
-    except Exception:
-        pass
+    if container:
+        hz_data = exec_json(container, ["curl", "-s", "-m", "5", HORIZON_URL + "/"])

    # =============================================================
    # 1. CONTAINER STATUS
@@ -339,23 +373,36 @@ def run_cycle(client, peers_extended, geo_enabled, show_ip, interactive=False, e
    hdr("🌅 API Status (Horizon)")

    if hz_data is None:
-        err(f"Horizon API not reachable ({HORIZON_URL})")
+        err(f"Horizon API not reachable ({HORIZON_URL} in container '{container_name}')")
    else:
        core_blk  = hz_data.get("core_latest_ledger",    0)
        hist_blk  = hz_data.get("history_latest_ledger", 0)
+        elder_blk = hz_data.get("history_elder_ledger",  0)
        ings_blk  = hz_data.get("ingest_latest_ledger",  0)
-        lag       = core_blk - ings_blk
        closed_at = hz_data.get("history_latest_ledger_closed_at", "?")

-        if lag <= 2:
+        # Reference ledger for the ingest lag: stellar-core's own ledger if we
+        # have it, otherwise what Horizon reports for core. core_latest_ledger
+        # is 0 when Horizon has lost its connection to stellar-core; computing
+        # the lag against 0 would give a negative value and a false "Synced".
+        ref_blk = block if core_info is not None else core_blk
+        lag     = (ref_blk - ings_blk) if ref_blk else None
+
+        if not core_blk:
+            err("Status: Horizon has no connection to stellar-core (core_latest_ledger = 0)")
+            if lag is not None and lag > 2:
+                warn(f"Ingest {lag} ledger behind core")
+        elif lag <= 2:
            ok(f"Status: Synced (Ingest lag: {lag} ledger)")
        else:
            warn(f"Status: Lagging (Ingest {lag} ledger behind core)")
        core_diff = (block - core_blk) if (core_info is not None and core_blk) else None
        core_diff_str = f"  (Δ {core_diff:+d} vs Core)" if core_diff is not None else ""
-        print(f"     Core Latest Ledger:   {core_blk}{core_diff_str}")
+        core_blk_str  = str(core_blk) if core_blk else "— (not connected)"
+        print(f"     Core Latest Ledger:   {core_blk_str}{core_diff_str}")
        print(f"     Ingest Latest Ledger: {ings_blk}  (last closed: {closed_at})")
-        print(f"     History Oldest:       {hist_blk}")
+        print(f"     History Latest:       {hist_blk}")
+        print(f"     History Oldest:       {elder_blk}")

    # =============================================================
    # 5. NODE METRICS (stellar-core)
@@ -758,56 +805,75 @@ def interactive_loop(client, refresh_interval, peers_extended, geo_enabled, show
    traffic_extended = extended_traffic  # local, toggled with 'x'

    old_settings = termios.tcgetattr(sys.stdin)
+    # cbreak wird EINMAL für die gesamte Session gesetzt (nicht erst nach jedem
+    # run_cycle()). Andernfalls bleibt das Terminal während der Ausführung im
+    # zeilengepufferten Modus des alten old_settings, und das anschließende
+    # tty.setcbreak(..., TCSAFLUSH) verwirft dabei sogar noch ungelesene
+    # Eingaben – eine während einer langsamen Ausführung gedrückte Taste wie
+    # 'q' würde so nie ankommen.
+    tty.setcbreak(sys.stdin.fileno())
    try:
        while True:
+            cycle_start = time.monotonic()
            buf = io.StringIO()
            with contextlib.redirect_stdout(buf):
                outbound_peers, container = run_cycle(
                    client, peers_extended, geo_enabled, show_ip, interactive=True,
                    extended_traffic=traffic_extended
                )
+                cycle_duration = time.monotonic() - cycle_start
+                if cycle_duration >= refresh_interval:
+                    warn(f"Update dauerte {cycle_duration:.1f}s – "
+                         f"länger als das Intervall ({refresh_interval}s). "
+                         f"Intervall (refresh_interval) erhöhen.")
+                elif cycle_duration >= refresh_interval * REFRESH_WARN_RATIO:
+                    pct = cycle_duration / refresh_interval * 100
+                    warn(f"Update dauerte {cycle_duration:.1f}s "
+                         f"({pct:.0f}% des {refresh_interval}s-Intervalls) – knapp.")
            sys.stdout.write("\033[2J\033[H" + buf.getvalue())
            sys.stdout.flush()

-            deadline = time.monotonic() + refresh_interval
-            tty.setcbreak(sys.stdin.fileno())
-            try:
-                last_secs = -1
-                while True:
-                    remaining = max(0.0, deadline - time.monotonic())
-                    secs = int(remaining) + (1 if remaining % 1 > 0 else 0)
-                    if secs != last_secs:
-                        x_label = "RX/TX" if traffic_extended else "detail"
-                        text = f"  [d] Drop peer  [a] Drop all  [x] {x_label}  [q] Beenden  Refresh in {secs}s  "
-                        sys.stdout.write("\r" + text)
-                        sys.stdout.flush()
-                        last_secs = secs
+            # Timer läuft ab Beginn der Ausführung, nicht erst ab deren Ende –
+            # so verlängert die Update-Laufzeit nicht zusätzlich das Intervall.
+            deadline = cycle_start + refresh_interval
+            last_secs = -1
+            while True:
+                remaining = max(0.0, deadline - time.monotonic())
+                secs = int(remaining) + (1 if remaining % 1 > 0 else 0)
+                if secs != last_secs:
+                    x_label = "RX/TX" if traffic_extended else "detail"
+                    text = (f"  [d] Drop peer  [a] Drop all  [x] {x_label}  [q] Beenden  "
+                             f"Refresh in {secs}s  (Update: {cycle_duration:.1f}s)  ")
+                    sys.stdout.write("\r" + text)
+                    sys.stdout.flush()
+                    last_secs = secs

-                    if remaining <= 0:
-                        break
+                # Immer mindestens einmal (nicht-blockierend, wenn remaining
+                # bereits 0 ist) auf Eingabe prüfen – auch wenn die Deadline
+                # schon verstrichen ist, weil die Ausführung selbst länger als
+                # das Intervall dauerte. Sonst wird 'q' & Co. in diesem Fall
+                # nie abgefragt, da sofort in den nächsten Zyklus gesprungen würde.
+                timeout = min(remaining, 0.25) if remaining > 0 else 0
+                rlist, _, _ = select.select([sys.stdin], [], [], timeout)
+                if rlist:
+                    ch = sys.stdin.read(1)
+                    if ch in ('q', 'Q', '\x03', '\x04'):
+                        print()
+                        return
+                    elif ch in ('d', 'D') and outbound_peers and container:
+                        print()
+                        do_drop_workflow(outbound_peers, container)
+                        break  # trigger immediate refresh
+                    elif ch in ('a', 'A') and outbound_peers and container:
+                        print()
+                        drop_all_workflow(outbound_peers, container)
+                        break  # trigger immediate refresh
+                    elif ch in ('x', 'X'):
+                        traffic_extended = not traffic_extended
+                        break  # trigger immediate refresh

-                    rlist, _, _ = select.select([sys.stdin], [], [], min(remaining, 0.25))
-                    if rlist:
-                        ch = sys.stdin.read(1)
-                        if ch in ('q', 'Q', '\x03', '\x04'):
-                            termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings)
-                            print()
-                            return
-                        elif ch in ('d', 'D') and outbound_peers and container:
-                            termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings)
-                            print()
-                            do_drop_workflow(outbound_peers, container)
-                            break  # trigger immediate refresh
-                        elif ch in ('a', 'A') and outbound_peers and container:
-                            termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings)
-                            print()
-                            drop_all_workflow(outbound_peers, container)
-                            break  # trigger immediate refresh
-                        elif ch in ('x', 'X'):
-                            traffic_extended = not traffic_extended
-                            break  # trigger immediate refresh
-            finally:
-                termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings)
+                if remaining <= 0:
+                    break

    except KeyboardInterrupt:
        pass
@@ -829,9 +895,17 @@ Options:
  -s, --show-ip          Show IP column (hidden by default)
  -x, --extended         Full RX/TX/RX-rate/TX-rate columns (default: ratio only)
  -i [SEC], --interactive [SEC]
-                         Auto-refresh mode (default interval: 10 s)
+                         Auto-refresh mode (overrides the configured interval)
  -?, --help             Show this help and exit

+Config file (first match wins, JSON):
+  $PI_STAT_CONFIG
+  ~/.config/pi-stat/config.json
+  <script dir>/pi-stat.json
+
+  Keys:
+    refresh_interval   Auto-refresh interval in seconds (default: 5)
+
 Interactive keys:
  d   Drop one outbound peer
  a   Drop all outbound peers
@@ -853,7 +927,7 @@ def main():
    interactive       = "-i" in args or "--interactive" in args
    extended_traffic  = "-x" in args or "--extended" in args

-    refresh_interval = 10
+    refresh_interval = config_refresh(load_config())
    if interactive:
        for flag in ("-i", "--interactive"):
            if flag in args:
@@ -0,0 +1,4 @@
+{
+  "_comment": "refresh_interval in Sekunden, min. 1 (Default: 5)",
+  "refresh_interval": 10
+  }