9 Commits
Author SHA1 Message Date
mikeandClaude Opus 5 2a622046f2 Define mirror targets one way: remote.<name>.<field>
There were two spellings for the same thing -- a flat
remoteurl/remotekey/remotetype/remotevisibility set for a single server,
and remote.<name>.* blocks for several. The flat one is gone; every
target, including a lone one, is now a named block with the fields url,
key, type and visibility.

An existing ~/.mgshrc is converted on the next start. Only the key is
rewritten, so values, comments, alignment, commented-out lines and the
file's 0600 mode survive untouched, and mgsh prints each rename rather
than doing it quietly. The target is named "public", which is what the
old settings called the git remote they created, so a converted setup
keeps pushing to the same place under the same remote name. A file that
carries both spellings keeps what the new one says.

The environment follows the same shape: MGSH_REMOTEURL and friends are
replaced by MGSH_REMOTE_<NAME>_<FIELD>, so MGSH_REMOTE_GITLAB_KEY sets
remote.gitlab.key. The field is read from the end of the variable name,
which leaves target names free to contain underscores.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 18:00:26 +02:00
mikeandClaude Opus 5 61a7059f61 Fold the unpublished projects into the overview table
They were a list underneath the table, which meant reading the same
project names in two different shapes. They are rows now, with the action
in an "init" column that only appears when some row needs it, and they
sort to the bottom as their own group: an un-inited directory is a
different kind of task and should not push the daily ones down.

Every directory under the base gets a row, not just the repositories --
`init` is exactly what turns a plain directory into a project, so leaving
those out would have hidden the ones the column is for. Such a row has no
git state to show and costs no subprocesses either, since projectStatus
now checks for .git before running any.

The count line gained "N to init"; the projects count still counts
repositories, so the two numbers stay meaningful side by side. An
unreachable server marks nothing at all, as before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 17:49:59 +02:00
mikeandClaude Opus 5 8ca05d6ad2 Drop the "not cloned here" line from overview
It answered a question `list` already answers, and it did so on every
run: the point of the overview is the state of the projects you have,
not a second listing of the server. reportInventory became
reportUnpublished and now reports one thing -- the local projects the
server has never seen, which are the ones `init` is for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 17:45:02 +02:00
mikeandClaude Opus 5 cb1ff98c3d Lay overview out as a table
The status field was not a column: "*", "↑2", "✓" and "✓ (no upstream)"
are four different widths, so everything after them started somewhere
else on every line and the eye had to hunt along each row instead of
going down one.

Each field now has its own measured column: name (with the branch
appended when it is not master/main), a one-character dirty marker, the
sync state, host and age, then the mirrors. "(no upstream)" was fifteen
columns wide for something that is not even a problem, and is now "–".
Colour weights the row rather than decorating it -- a project that is
clean and in sync goes grey, the arrows and the dirty marker keep their
colour -- and the rows needing action sort to the top, alphabetically
within each group so positions stay predictable.

padRight counted bytes, which was fine while everything it padded was
ASCII; the arrows and check marks are three bytes and one column, so it
counts runes now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 17:42:37 +02:00
mikeandClaude Opus 5 86fb898df4 Fix list breaking when the server's du fails
Two mistakes in the size support, reported from a real server.

The remote command used "2>/dev/null" to silence du. That is sh syntax,
and the git user's login shell need not be sh: in csh it parses as an
argument "2" followed by a redirection of stdout, so du was handed a
file named "2", complained, and exited non-zero. The redirection is
gone -- without it there is no bogus argument to trip over, and the
command now uses nothing that differs between sh and csh.

Worse, the exit status of the chain is the *last* command's, so that
failing du made sshOut return an error and `list` threw away a listing
that had arrived perfectly intact. It now reports a failure only when
nothing usable came back at all; a listing that parsed is shown whatever
the exit status, simply without the size column.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 17:35:50 +02:00
mikeandClaude Opus 5 3a420093d1 Show repository sizes in list
`list -a` had sizes because archives are files; repositories are
directories, and a long listing reports the inode size for those -- 4096
for every single one. Taking that number would have filled the column
with the same meaningless value, so the real disk usage is asked of `du`
instead, appended to the same remote command so it still costs one round
trip.

The column is dropped entirely when no usable sizes come back, rather
than showing a column of zeroes, so a server without a working `du`
degrades to the previous output. The summary line carries the total.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 17:30:12 +02:00
mikeandClaude Opus 5 8c68b28dc2 Format list as an aligned table
The name is what the eye looks for, but it came last, behind a ragged
date column, so nothing lined up. Worse, colorRepoLine rebuilt the line
with strings.Fields and single spaces, which destroyed the alignment ls
had produced -- the output was aligned only when colour was off.

The listing line is now parsed properly instead of being split at the
size field: name, date and size come out as fields, the date is re-padded
to a fixed twelve columns so "Sep 28  2016" and "Jan  3 14:32" agree, and
the name leads in a column sized to the longest entry. Colour decorates
that layout without changing it, which a test now checks by stripping the
escapes and comparing. `list -a` shows archive sizes, which were parsed
and thrown away before.

An empty result says so instead of printing nothing, which was
indistinguishable from a failure, and the count line matches the rest of
mgsh. The pattern now filters on the repository name rather than the
whole listing line: matching the owner or the date was never intended and
`list 2016` quietly did it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 16:53:54 +02:00
mikeandClaude Opus 5 cad7a4ec2c Turn overview into an inventory across local and server
overview showed dirty and ahead/behind per project, which git can do on
its own. mgsh is the only thing that sees both the local base directory
and the ssh server, and joining those answers the questions git cannot:
which projects were never pushed to the server (candidates for `init`),
and which exist there but not on this machine (candidates for `clone`).
Both lists are printed after the summary. An unreachable server is
reported as such, rather than as "everything is missing".

Each row also names the machine that made the last commit and how long
ago. That costs nothing: `push` has always stamped "[user@host]" into
the commit message, and nothing ever read it back. On a setup spanning
several machines it is usually the piece one actually wanted. Rows also
show which mirror targets the repository has a remote for, which is
local git config and therefore free.

The walk is now concurrent and cheaper per project: `git status
--porcelain=v2 --branch` yields branch, upstream, ahead/behind and dirty
in one subprocess where three were used before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 16:17:40 +02:00
mikeandClaude Opus 5 59da8f376c Check the staged diff for credentials before push commits
`push` runs `git add --all .`, so anything lying in the project gets
committed, and with `mirror = true` it reaches a public server in the
same breath. It is the one action in mgsh that cannot be undone: a
deleted server repository comes back from an archive, a published
credential does not.

The staged diff is now scanned before the commit is made -- private keys,
GitHub/GitLab/Slack/AWS/PyPI tokens, and credential-shaped assignments --
and a hit is shown with file and line before asking whether to continue.
Declining leaves the changes staged but uncommitted, so removing the file
and adding a .gitignore entry is all it takes.

The hard part is not detection but silence. A scanner that cries wolf
gets answered with a reflexive "y" and stops being a safety net, so
values that are plainly environment references, dotted identifiers,
constant names, template slots or masked stand-ins are filtered out. A
test scans mgsh's own README and mgshrc.example -- both full of
credential-shaped text -- and fails if either would trip the check. It
caught the documentation for this very feature, which is why the README
describes the sample output instead of reproducing it.

For a line that legitimately looks like a credential there is
`mgsh:allow`, which suppresses that one line; `secretscan = off` turns
the check off entirely. Only an explicit "off" does that -- a typo in the
setting leaves the safety net in place, which is what the new falsy()
is for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 16:17:30 +02:00
17 changed files with 1803 additions and 237 deletions
+138 -33
View File
@@ -7,6 +7,7 @@ directory. Go port of the original Perl `mgsh` (`mgsh.perl`).
## Contents
- [Build](#build) · [Usage](#usage) · [Commands](#commands) · [Aliases](#aliases)
- [Overview](#overview) · [Credential check](#credential-check)
- [Public mirror (`pushremote`)](#public-mirror-pushremote) ·
[Releases](#releases)
- [Configuration](#configuration) · [Settings reference](#settings-reference) ·
@@ -80,11 +81,11 @@ Run `help` for the full list. Highlights:
| `pushremote [desc]` | mirror the repo to a public server (gitea/github/gitlab) |
| `pull` / `fetch` | pull / fetch from the server |
| `status [-a]` / `diff` | short git status (`-a`: overview of all projects) |
| `overview` | dirty / ahead-behind summary of all projects |
| `overview` | inventory of all projects, local and on the server |
| `log` | show the project log |
| `edit [n]` | interactive rebase of the last n commits |
| `clone [-a] <repo>` | clone a repository (or archive) from the server |
| `list [-a] [pattern]` | list repositories on the server |
| `list [-a] [pattern]` | list repositories on the server (`-a`: archives, with sizes) |
| `show <repo>` | show a repository log directly on the server |
| `archive [comment]` | snapshot the server-side repo into `./archive` |
| `init` | make a new repository from the current directory |
@@ -125,6 +126,111 @@ pushremote targets (in push order):
Tokens are masked, so the output is safe to paste into a bug report.
`config -k` prints just the setting names, one per line.
### Listing the server
`list` shows what is on the git server, name first and aligned, ordered by
modification time — `push` touches the bare repository, so the most recently
worked-on project sits closest to the prompt:
```
< src > list
Betaflight3.0.0 Sep 28 2016 181M
website Mar 3 2024 2.1M
notes Jan 3 14:32 876K
3 repositories · 184M
```
The size is the repository's real disk usage on the server, asked of `du` in
the same round trip as the listing — a long listing reports the inode size for
a directory, which is the same number for every repository and says nothing. If
the server produces no usable sizes the column is left out rather than filled
with zeroes.
`list -a` lists the archives instead, whose sizes come from the listing itself;
a pattern filters by name (`list note`).
### Overview
`overview` (or `status -a`) is the one view that needs mgsh: it is the only
thing that sees the local base directory *and* the git server at once.
```
< src > overview
notes * ↑2 laptop 3h
website * ✓ desktop 2d → hub
Betaflight3.0.0 ✓ workstation 20d → gitea hub
experiments init
sandbox (wip) laptop 1h init
4 projects · 2 dirty · 2 in sync · 2 to init
```
Every field sits in its own column, so the eye can go down one instead of
hunting along each line. The projects that need something done come first, the
settled ones next, and the directories the git server does not have yet come
last — those are a different kind of task. Within each group the order stays
alphabetical, so positions do not jump around.
| column | meaning |
|---|---|
| name | the project, with its branch appended when it is not `master`/`main` |
| `*` | uncommitted changes |
| `↑n` `↓n` | commits ahead of / behind the upstream (`↑2↓1` when both) |
| `✓` | in sync with the upstream |
| `` | the branch tracks nothing — never pushed |
| `init` | the git server does not have this one; run `init` |
| host, age | who last committed and when, from the `[user@host]` stamp |
| `→` | mirror targets this repository has a remote for |
Every directory under the base gets a row, including those that are not
repositories at all — `init` is exactly what turns one into a project, so it
belongs in the table rather than in a list underneath it. Such a row simply has
no git state to show.
The host and age come from the commit itself — `push` writes `[user@host]` into
every message, so `overview` can say where a project was last worked on without
storing anything. On a setup spanning a laptop and a workstation that is usually
the piece of information you actually wanted.
The `init` column is the join no git command can do. If the server cannot be
reached, no row is marked and mgsh says so — not knowing is not the same as
knowing they are missing. The other direction, repositories on the server that
are not here, is what `list` shows.
### Credential check
`push` runs `git add --all .`, so anything lying in the project gets committed —
and with `mirror = true` it reaches a public server in the same breath. That is
the only action in mgsh that cannot be undone: a deleted server repository comes
back from an archive, a published credential does not.
So before anything is committed, the staged diff is checked for private keys,
GitHub/GitLab/Slack/AWS/PyPI tokens and credential-shaped assignments:
```
< src/notes > push new notes
2 possible credential(s) in what is about to be committed:
.env:3 credential assignment
API_KEY="<the offending line is shown in full here>"
deploy_key:1 private key
<the BEGIN … PRIVATE KEY header is shown here>
(set 'secretscan = off' to skip this check)
push anyway? y/N ?
```
Declining stops the push with nothing committed; the changes stay staged, so
`git restore --staged <file>` and a `.gitignore` entry are all it takes.
For a line that only *looks* like a credential and is meant to stay, put
`mgsh:allow` in it — a comment on that line is enough. That is better than
turning the whole check off for one false positive.
This is not a complete secret scanner and does not try to be one. It aims for a
high hit rate on what actually leaks, with few enough false alarms that the
prompt still means something: values that are plainly environment references,
constants, template slots (`<token>`, `${VAR}`) or masked stand-ins are ignored —
a test checks that mgsh's own README and `mgshrc.example`, both full of
credential-shaped text, stay quiet. Switch it off with `secretscan = off`.
### Aliases
`alias <name> '<command>'` defines a reusable shortcut, persisted to
@@ -153,38 +259,37 @@ alias ec '!echo $1' # ec hello -> echo hello (shell)
Besides the internal ssh git server, `pushremote` mirrors the active project to
one or more public hosting servers (Gitea, GitHub or GitLab) over their REST
API. A single server is configured flat:
```ini
remoteurl = https://git.example.com # base URL of the server
remotekey = <personal-access-token> # API token
# remotetype = gitea # optional; auto-detected from remoteurl
# remotevisibility = private # visibility of created repos (default private)
# mirror = true # `push` also mirrors via pushremote
```
Several servers get one `remote.<name>.*` block each:
API. Each server is one `remote.<name>.<field>` block:
```ini
remote.gitea.url = https://git.example.com
remote.gitea.key = <personal-access-token>
remote.gitea.type = gitea # optional; auto-detected from the url
remote.gitea.visibility = private # or public (default private)
remote.hub.url = https://github.com
remote.hub.key = <personal-access-token>
remote.hub.visibility = public
remote.gitlab.url = https://gitlab.example.com
remote.gitlab.key = <personal-access-token>
remote.gitlab.type = gitlab
remote.gitlab.visibility = public
# remotes = gitea, hub # optional: restrict and order the set
# remotes = gitea, gitlab # optional: restrict and order the set
# mirror = true # `push` also mirrors via pushremote
```
`<name>` is yours to pick; there is no other spelling. Older versions had a flat
`remoteurl`/`remotekey` pair for a single server — mgsh converts those to
`remote.public.*` in place on the next start and says so, keeping the git remote
name those versions used.
| command | pushes to |
|------------------------|-----------------------------------------------|
| `pushremote` | every configured target, in order |
| `pushremote @hub` | only `hub` |
| `pushremote @hub @gitea` | those two |
| `pushremote @gitea` | only `gitea` |
| `pushremote @gitea @gitlab` | those two |
| `pushremote a fix` | every target, description "a fix" |
Each target owns a git remote of the same name in the repository (the flat form
uses `public`, as before), so `git push hub` keeps working outside mgsh. A
Each target owns a git remote of the same name in the repository, so
`git push gitlab` keeps working outside mgsh. A
target that fails does not stop the others; with more than one target
`pushremote` prints an `n/m remotes updated` summary. `remotes = …` restricts
and orders the set, which is mostly useful in a project `.mgshrc` — see below.
@@ -198,9 +303,10 @@ The token is sent as a one-shot HTTP auth header: it is never written into the
repo's git config, and it reaches git through the environment rather than the
command line, so it does not show up in the process table. Because `~/.mgshrc`
then holds a credential, mgsh creates it mode `600` and warns at startup if an
existing file is readable by others. The provider is auto-detected from `remoteurl` (`github.com`
GitHub, `gitlab*` → GitLab, otherwise Gitea) and can be forced with
`remotetype`. Set `mirror = true` to have every `push` mirror automatically.
existing file is readable by others. The provider is auto-detected from the url
(`github.com`GitHub, `gitlab*` → GitLab, otherwise Gitea) and can be forced
with `remote.<name>.type`. Set `mirror = true` to have every `push` mirror
automatically.
### Releases
@@ -284,8 +390,10 @@ alias co 'checkout $1'
### Settings reference
Every setting can also be given as an environment variable named `MGSH_<KEY>`
(e.g. `MGSH_GITHOST`), which wins over both files. "Scope" says whether a
project `.mgshrc` may override the setting.
(e.g. `MGSH_GITHOST`), which wins over both files; a mirror field is
`MGSH_REMOTE_<NAME>_<FIELD>`, so `MGSH_REMOTE_GITLAB_KEY` sets
`remote.gitlab.key`. "Scope" says whether a project `.mgshrc` may override the
setting.
| setting | scope | meaning |
|---|---|---|
@@ -299,16 +407,13 @@ project `.mgshrc` may override the setting.
| `gitemail` | global | `user.email` written to the global git config |
| `pushdefault` | global | `push.default` written to the global git config |
| `editor` | project | opener used by `open`/`view` when the project has no Xcode workspace (default `coda`) |
| `remoteurl` | project | base URL of a single mirror server (target name `public`) |
| `remotekey` | project | API token for `remoteurl` |
| `remotetype` | project | `gitea`\|`github`\|`gitlab`; auto-detected from the URL when unset |
| `remotevisibility` | project | `private` (default) or `public` for repositories created by `pushremote` |
| `remote.<name>.url` | project | base URL of the named mirror target |
| `remote.<name>.url` | project | base URL of the mirror target `<name>` |
| `remote.<name>.key` | project | API token for that target |
| `remote.<name>.type` | project | provider override for that target |
| `remote.<name>.visibility` | project | visibility for that target |
| `remote.<name>.type` | project | `gitea`\|`github`\|`gitlab`; auto-detected from the url when unset |
| `remote.<name>.visibility` | project | `private` (default) or `public` for repositories `pushremote` creates |
| `remotes` | project | comma- or space-separated list restricting and ordering the mirror targets |
| `mirror` | project | truthy (`1`/`true`/`yes`/`on`) → every `push` also mirrors |
| `secretscan` | project | `off` disables the credential check `push` runs before committing (on by default; only an explicit `off` disables it) |
The three settings written to the global git config are applied at startup, and
only when they actually differ, so a plain `mgsh status` does not rewrite
+48 -14
View File
@@ -2,7 +2,9 @@ package main
import (
"fmt"
"strconv"
"strings"
"unicode/utf8"
)
// Colors for the prompt, banner and output, using the Catppuccin Mocha palette
@@ -35,25 +37,57 @@ func errorln(msg string) {
fmt.Println(col(cRed, msg))
}
// padRight pads an ASCII string with trailing spaces to width n.
// padRight pads s with trailing spaces to a width of n columns. It counts
// runes, not bytes: the overview pads fields holding ↑ ↓ ✓, each of which is
// one column wide but three bytes long.
func padRight(s string, n int) string {
if len(s) < n {
return s + strings.Repeat(" ", n-len(s))
if l := utf8.RuneCountInString(s); l < n {
return s + strings.Repeat(" ", n-l)
}
return s
}
// colorRepoLine colors a `list` entry: the leading `ls -ltr` date (3 fields) in
// yellow and the repository name in green.
func colorRepoLine(s string) string {
if !useColor {
return s
// formatRepoList renders the server listing for `list`: the name first, in a
// column wide enough for the longest one, then the date, and for archives the
// size. Names come first because that is what the eye scans for; putting the
// ragged date there instead is what made the old output hard to read.
//
// The order is left as it arrives: `ls -ltr` sorts by modification time, and
// `push` touches the bare repository, so the most recently worked-on project
// ends up closest to the prompt.
func formatRepoList(entries []lsEntry, withSize bool) string {
width := 0
for _, e := range entries {
if len(e.name) > width {
width = len(e.name)
}
parts := strings.Fields(s)
if len(parts) >= 4 {
date := strings.Join(parts[:3], " ")
name := strings.Join(parts[3:], " ")
return col(cYellow, date) + " " + col(cGreen, name)
}
return col(cGreen, s)
var b strings.Builder
for _, e := range entries {
fmt.Fprintf(&b, " %s %s", col(cGreen, padRight(e.name, width)), col(cYellow, e.date))
if withSize {
fmt.Fprintf(&b, " %s", col(cGray, fmt.Sprintf("%7s", humanSize(e.size))))
}
b.WriteByte('\n')
}
return b.String()
}
// humanSize renders a byte count compactly, the way `ls -h` does: a decimal
// only while it still carries information, so "3.2M" but "512K".
func humanSize(n int64) string {
const unit = 1024
if n < unit {
return strconv.FormatInt(n, 10) + "B"
}
div, exp := int64(unit), 0
for v := n / unit; v >= unit && exp < 4; v /= unit {
div *= unit
exp++
}
v := float64(n) / float64(div)
if v < 10 {
return fmt.Sprintf("%.1f%c", v, "KMGTP"[exp])
}
return fmt.Sprintf("%.0f%c", v, "KMGTP"[exp])
}
+127 -21
View File
@@ -14,32 +14,85 @@ import (
var (
optRe = regexp.MustCompile(`^-(\w)$`)
numRe = regexp.MustCompile(`^\d+$`)
// a `ls -ltr` long-listing line: mode, link count, owner, group, size, then
// the date columns and the name. Owner and group are matched as opaque
// a `ls -ltr` long-listing line: mode, link count, owner, group, size, the
// three date columns, then the name. Owner and group are matched as opaque
// fields — the bare repositories need not belong to a user or group
// literally named "git".
lsEntryRe = regexp.MustCompile(`^\S+\s+\d+\s+\S+\s+\S+\s+\d+\s+(.*)$`)
lsEntryRe = regexp.MustCompile(`^\S+\s+\d+\s+\S+\s+\S+\s+(\d+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(.*)$`)
gitDirRe = regexp.MustCompile(`^(.*)\.git$`)
sanRe = regexp.MustCompile(`[,;:\\/='"|?><-]+`)
wsRe = regexp.MustCompile(`\s+`)
)
// lsEntry extracts the "<date columns> <name>" tail of a `ls -ltr` line whose
// entry name ends in suffix, with the suffix removed. It returns "" for any
// other line (the leading "total" line, entries of a different kind).
func lsEntry(line, suffix string) string {
// listMarker separates the two sections of the combined listing command, so
// `list` gets both the long listing and the disk usage in one round trip.
const listMarker = "---mgsh---"
// duRe matches one `du -sk` line: kilobytes, then the path.
var duRe = regexp.MustCompile(`^(\d+)\s+(.*)$`)
// splitAtMarker divides the remote output into the part before and after the
// marker line. Everything is in the first section when the marker is absent —
// which is what happens when only a plain listing was asked for.
func splitAtMarker(lines []string, marker string) (before, after []string) {
for i, ln := range lines {
if strings.TrimSpace(ln) == marker {
return lines[:i], lines[i+1:]
}
}
return lines, nil
}
// parseDuSizes turns `du -sk` output into a name -> bytes map. A long listing
// reports the inode size for a directory — the same number for every bare
// repository — so this is the only way to say how large one actually is.
// A symlinked repository reports the size of the link, not of its target.
func parseDuSizes(lines []string) map[string]int64 {
out := map[string]int64{}
for _, ln := range lines {
m := duRe.FindStringSubmatch(strings.TrimRight(ln, "\r"))
if m == nil {
continue
}
kb, err := strconv.ParseInt(m[1], 10, 64)
if err != nil {
continue
}
out[strings.TrimPrefix(strings.TrimSpace(m[2]), "./")] = kb * 1024
}
return out
}
// lsEntry is one parsed entry of the server's listing.
type lsEntry struct {
name string // with the ".git" / ".git.tar.gz" suffix removed
date string // the ls date columns, normalised to a fixed 12 columns
size int64
}
// parseLsEntry reads one `ls -ltr` line whose entry name ends in suffix. It
// returns false for anything else: the leading "total" line, entries of another
// kind, or output that does not look like a long listing at all.
func parseLsEntry(line, suffix string) (lsEntry, bool) {
m := lsEntryRe.FindStringSubmatch(strings.TrimSpace(line))
if m == nil {
return ""
return lsEntry{}, false
}
name := m[1]
name := m[5]
if i := strings.Index(name, " -> "); i >= 0 {
name = name[:i] // a symlinked bare repo lists as "link.git -> target.git"
}
if !strings.HasSuffix(name, suffix) {
return ""
return lsEntry{}, false
}
return strings.TrimSuffix(name, suffix)
size, _ := strconv.ParseInt(m[1], 10, 64)
return lsEntry{
name: strings.TrimSuffix(name, suffix),
// ls pads these itself, but only in its own column widths; re-pad so
// "Sep 28 2016" and "Jan 3 14:32" line up at 12 either way
date: fmt.Sprintf("%s %2s %5s", m[2], m[3], m[4]),
size: size,
}, true
}
// validProject reports whether name is usable as a project name: a single path
@@ -184,20 +237,66 @@ func runCommandDepth(line string, depth int) bool {
if opt["a"] {
path, suffix = "./archive", ".git.tar.gz"
}
pat := word(words, 1)
lines, err := sshOut("/bin/ls -ltr " + shq(path))
if err != nil {
errorln("could not list repositories on the git server")
break
one, many := "repository", "repositories"
if opt["a"] {
one, many = "archive", "archives"
}
for _, ln := range lines {
if pat != "" && !strings.Contains(strings.ToLower(ln), strings.ToLower(pat)) {
pat := strings.ToLower(word(words, 1))
remote := "/bin/ls -ltr " + shq(path)
if !opt["a"] {
// archives are files and carry a real size; repositories are
// directories, whose listed size is the inode's, so ask du in the
// same round trip. Nothing shell-specific here on purpose: the
// login shell may be csh, where "2>/dev/null" is not a redirection
// but an argument followed by one.
remote += "; echo " + shq(listMarker) + "; du -sk *.git"
}
lines, err := sshOut(remote)
lsLines, duLines := splitAtMarker(lines, listMarker)
sizes := parseDuSizes(duLines)
var entries []lsEntry
var total int64
for _, ln := range lsLines {
e, ok := parseLsEntry(ln, suffix)
// the pattern filters the name, not the whole listing line — an
// accidental match on the date or the owner helps nobody
if !ok || (pat != "" && !strings.Contains(strings.ToLower(e.name), pat)) {
continue
}
if name := lsEntry(ln, suffix); name != "" {
fmt.Println(colorRepoLine(name))
if !opt["a"] {
e.size = sizes[e.name+suffix] // 0 when du said nothing
}
total += e.size
entries = append(entries, e)
}
// The exit status belongs to the last command in the chain, so a `du`
// that fails must not discard a listing that arrived intact. Only
// complain when nothing usable came back at all.
if len(entries) == 0 {
if err != nil {
errorln("could not list " + many + " on the git server")
break
}
what := "no " + many + " on the git server"
if pat != "" {
what = "no " + many + " matching '" + word(words, 1) + "'"
}
fmt.Println(col(cGray, what))
break
}
// no size column when the server gave no usable sizes, rather than a
// column of zeroes
fmt.Print(formatRepoList(entries, total > 0))
label := many
if len(entries) == 1 {
label = one
}
summary := fmt.Sprintf("%d %s", len(entries), label)
if total > 0 {
summary += " · " + humanSize(total)
}
fmt.Println(col(cGray, summary))
case "show": // show a repository's log directly on the server
prj := PRJ
@@ -272,6 +371,13 @@ func runCommandDepth(line string, depth int) bool {
}
comment := strings.Join(fields[1:], " ")
git(DIR, "add", "--all", ".")
// last look before anything is committed: `add --all` sweeps up whatever
// is lying around, and with mirroring on it goes straight to a public
// server. Nothing has been committed yet, so declining costs nothing.
if !secretsApproved(DIR) {
errorln("push cancelled — your changes are staged but not committed")
break
}
msg := strings.TrimSpace(fmt.Sprintf("[%s@%s] %s", USER, HOST, comment))
git(DIR, "commit", "-m", msg) // may be "nothing to commit"; continue anyway
if !gitOK(DIR, "push") {
@@ -711,7 +817,7 @@ var helpItems = []struct{ cmd, desc string }{
{"pull", "pull changes from git server"},
{"fetch", "fetch changes from git server"},
{"status [-a]", "short git status (-a: overview of all projects)"},
{"overview", "status of all projects (dirty, ahead/behind)"},
{"overview", "inventory of all projects, local and on the server"},
{"diff [args]", "show git diff"},
{"edit [number]", "edit last [number] commits (default is 10)"},
{"clone [-a] <repository>", "clone repository from git server (-a for archive)"},
+1 -7
View File
@@ -82,18 +82,12 @@ func fetchServerRepos() {
if serverFetched {
return
}
lines, err := sshOut("/bin/ls .")
repos, err := serverRepoNames()
if err != nil {
// a transient failure (server down, no network) must not cache an
// empty list for the rest of the session — the next Tab tries again
return
}
var repos []string
for _, ln := range lines {
if m := gitDirRe.FindStringSubmatch(strings.TrimSpace(ln)); m != nil {
repos = append(repos, m[1])
}
}
// a missing ./archive is a permanent, unremarkable state: still cache
var archives []string
if lines, err := sshOut("/bin/ls archive"); err == nil {
+143 -29
View File
@@ -6,6 +6,7 @@ import (
"os"
"path/filepath"
"regexp"
"slices"
"sort"
"strings"
)
@@ -33,11 +34,8 @@ type Config struct {
GitEmail string // git user.email to set globally ("" = leave alone)
PushDefault string // git push.default to set globally ("" = leave alone)
Editor string // editor/opener used as fallback by `open` ("" = coda)
RemoteURL string // public mirror server base URL (Gitea/GitHub/GitLab)
RemoteKey string // API token for the mirror server (used by `pushremote`)
RemoteType string // "gitea"|"github"|"gitlab" (auto-detected when empty)
RemoteVis string // visibility of created repos: "private" (default)|"public"
Mirror string // truthy -> `push` also mirrors via `pushremote`
SecretScan string // falsy -> `push` skips the credential scan
Remotes []RemoteTarget
RemoteNames string // "remotes": explicit, ordered subset of targets to use
}
@@ -53,22 +51,25 @@ type RemoteTarget struct {
Vis string // "private" (default) | "public"
}
// legacyRemoteName is the target name for the flat remoteurl/remotekey pair,
// matching the git remote that earlier versions created.
// legacyRemoteName is the target the pre-4.1 flat remoteurl/remotekey settings
// are migrated to. It matches the git remote those versions created, so a
// converted configuration keeps pushing to the same place.
const legacyRemoteName = "public"
// legacyRemoteKeys maps the old flat spelling onto the named-target form. A
// mirror target is defined one way now, not two.
var legacyRemoteKeys = map[string]string{
"remoteurl": "remote." + legacyRemoteName + ".url",
"remotekey": "remote." + legacyRemoteName + ".key",
"remotetype": "remote." + legacyRemoteName + ".type",
"remotevisibility": "remote." + legacyRemoteName + ".visibility",
}
// mirrorTargets returns the usable mirror targets in configured order, plus the
// names of targets that are defined but unusable (missing url or key) so the
// caller can complain about them instead of silently skipping.
func (c Config) mirrorTargets() (usable []RemoteTarget, incomplete []string) {
var all []RemoteTarget
if c.RemoteURL != "" || c.RemoteKey != "" {
all = append(all, RemoteTarget{
Name: legacyRemoteName, URL: c.RemoteURL, Key: c.RemoteKey,
Type: c.RemoteType, Vis: c.RemoteVis,
})
}
all = append(all, c.Remotes...)
all := c.Remotes
// `remotes = a, b` narrows and orders the set — a project .mgshrc uses it
// to mirror to only some of the globally configured servers.
@@ -146,11 +147,54 @@ func loadConfig() Config {
m := parseConfig(string(data))
applyConfig(&c, m)
warnConfigPerms(path, m)
migrateRemoteKeys(path, string(data))
}
applyEnv(&c)
return c
}
// migrateRemoteKeys converts the pre-4.1 flat remote settings in a config file
// to the remote.<name>.<field> spelling, so a mirror target is defined one way
// and not two. Only the key is rewritten: values, comments, blank lines and the
// file's permissions stay exactly as they are, and commented-out lines are left
// alone. Reports what it changed rather than doing it silently.
func migrateRemoteKeys(path, data string) {
lines := strings.Split(data, "\n")
var renamed []string
for i, ln := range lines {
trimmed := strings.TrimLeft(ln, " \t")
if trimmed == "" || strings.HasPrefix(trimmed, "#") {
continue
}
sep := strings.IndexAny(trimmed, "=:")
if sep < 0 {
continue
}
key := strings.TrimRight(trimmed[:sep], " \t")
dotted, ok := legacyRemoteKeys[strings.ToLower(key)]
if !ok {
continue
}
indent := ln[:len(ln)-len(trimmed)]
gap := trimmed[len(key):sep] // whatever alignment was there
lines[i] = indent + dotted + gap + trimmed[sep:]
renamed = append(renamed, key+" → "+dotted)
}
if len(renamed) == 0 {
return
}
if err := os.WriteFile(path, []byte(strings.Join(lines, "\n")), configMode); err != nil {
errorln("could not update " + path + ": " + err.Error())
return
}
fmt.Println(col(cGray, path+": mirror settings renamed to the remote.<name>.* form"))
for _, r := range renamed {
fmt.Println(col(cGray, " "+r))
}
}
// projectGlobalOnly lists settings a project-level .mgshrc must not change:
// `base` decides where projects live in the first place, and the git identity
// keys are written to the user's *global* git config at startup — applying
@@ -278,12 +322,14 @@ func writeConfigTemplate(path string) {
b.WriteString("# pushdefault = matching\n")
b.WriteString("# editor = code\n\n")
b.WriteString("# --- public mirrors for `pushremote` ---\n")
b.WriteString("# One block per server; `pushremote` pushes to all of them,\n")
b.WriteString("# `pushremote @hub` to a single one.\n")
b.WriteString("# remote.hub.url = https://github.com\n")
b.WriteString("# remote.hub.key = <personal-access-token>\n")
b.WriteString("# remote.hub.visibility = public\n")
b.WriteString("# remotes = hub # optional: restrict/order the set\n")
b.WriteString("# One 'remote.<name>.*' block per server. `pushremote` pushes to all\n")
b.WriteString("# of them, `pushremote @gitlab` to a single one. <name> is also the\n")
b.WriteString("# git remote created in the repository.\n")
b.WriteString("# remote.gitlab.url = https://gitlab.example.com\n")
b.WriteString("# remote.gitlab.key = <personal-access-token>\n")
b.WriteString("# remote.gitlab.type = gitlab # optional; detected from the url\n")
b.WriteString("# remote.gitlab.visibility = private # or public (default private)\n")
b.WriteString("# remotes = gitlab # optional: restrict/order the set\n")
b.WriteString("# mirror = true # `push` also mirrors\n\n")
b.WriteString("# A project may override any of these (except base and the git\n")
b.WriteString("# identity) in its own <project>/.mgshrc.\n")
@@ -364,13 +410,36 @@ func applyConfig(c *Config, m map[string]string) {
set("gitemail", &c.GitEmail)
set("pushdefault", &c.PushDefault)
set("editor", &c.Editor)
set("remoteurl", &c.RemoteURL)
set("remotekey", &c.RemoteKey)
set("remotetype", &c.RemoteType)
set("remotevisibility", &c.RemoteVis)
set("remotes", &c.RemoteNames)
set("mirror", &c.Mirror)
applyRemoteTargets(c, m)
set("secretscan", &c.SecretScan)
applyRemoteTargets(c, foldLegacyRemoteKeys(m))
}
// foldLegacyRemoteKeys rewrites the pre-4.1 flat remote settings into the
// named-target form, so a configuration that has not been converted yet still
// works while it is being read. The file itself is converted by
// migrateRemoteKeys; this only makes the current run behave.
func foldLegacyRemoteKeys(m map[string]string) map[string]string {
folded, copied := m, false
for old, dotted := range legacyRemoteKeys {
v, ok := m[old]
if !ok || v == "" {
continue
}
if _, taken := m[dotted]; taken {
continue // an explicit new-style setting always wins
}
if !copied { // copy on first write, never touch the caller's map
folded = make(map[string]string, len(m))
for k, val := range m {
folded[k] = val
}
copied = true
}
folded[dotted] = v
}
return folded
}
// remoteFieldRe matches a named mirror target setting: remote.<name>.<field>.
@@ -437,10 +506,55 @@ func applyEnv(c *Config) {
env("MGSH_GITEMAIL", &c.GitEmail)
env("MGSH_PUSHDEFAULT", &c.PushDefault)
env("MGSH_EDITOR", &c.Editor)
env("MGSH_REMOTEURL", &c.RemoteURL)
env("MGSH_REMOTEKEY", &c.RemoteKey)
env("MGSH_REMOTETYPE", &c.RemoteType)
env("MGSH_REMOTEVISIBILITY", &c.RemoteVis)
env("MGSH_REMOTES", &c.RemoteNames)
env("MGSH_MIRROR", &c.Mirror)
env("MGSH_SECRETSCAN", &c.SecretScan)
applyRemoteEnv(c)
}
// remoteFields are the settings a mirror target is made of.
var remoteFields = []string{"url", "key", "type", "visibility"}
// applyRemoteEnv reads MGSH_REMOTE_<NAME>_<FIELD>, the environment spelling of
// a remote.<name>.<field> setting — MGSH_REMOTE_GITLAB_KEY for
// remote.gitlab.key. The field is taken from the end, so a target name may
// contain underscores itself.
func applyRemoteEnv(c *Config) {
const prefix = "MGSH_REMOTE_"
// sorted, so a target these variables introduce lands in the push order the
// same way on every run
envs := os.Environ()
sort.Strings(envs)
for _, kv := range envs {
eq := strings.IndexByte(kv, '=')
if eq < 0 {
continue
}
name, value := kv[:eq], kv[eq+1:]
if value == "" || !strings.HasPrefix(name, prefix) {
continue
}
rest := name[len(prefix):]
us := strings.LastIndexByte(rest, '_')
if us <= 0 {
continue
}
target, field := strings.ToLower(rest[:us]), strings.ToLower(rest[us+1:])
if !slices.Contains(remoteFields, field) {
continue // MGSH_REMOTES and anything else that merely starts alike
}
t := &c.Remotes[c.remoteIndex(target)]
switch field {
case "url":
t.URL = value
case "key":
t.Key = value
case "type":
t.Type = value
case "visibility":
t.Vis = value
}
}
}
+16
View File
@@ -152,6 +152,22 @@ func sshOut(remote string) ([]string, error) {
return lines, err
}
// serverRepoNames lists the bare repositories on the git server, without the
// ".git" suffix.
func serverRepoNames() ([]string, error) {
lines, err := sshOut("/bin/ls .")
if err != nil {
return nil, err
}
var out []string
for _, ln := range lines {
if m := gitDirRe.FindStringSubmatch(strings.TrimSpace(ln)); m != nil {
out = append(out, m[1])
}
}
return out, nil
}
// serverEntryExists reports whether entry is present in the remote directory
// path (relative to the git user's home). The error is returned rather than
// folded into the bool so a failed lookup is never mistaken for "not there".
+298 -37
View File
@@ -9,6 +9,7 @@ import (
"strings"
"testing"
"time"
"unicode/utf8"
)
func TestSanitizeComment(t *testing.T) {
@@ -74,19 +75,65 @@ func TestFormatLogRecentCompact(t *testing.T) {
}
}
func TestColorRepoLine(t *testing.T) {
func TestFormatRepoList(t *testing.T) {
useColor = false
in := "Sep 28 2016 Betaflight3.0.0"
if got := colorRepoLine(in); got != in {
t.Errorf("colorRepoLine with color off changed input: %q", got)
entries := []lsEntry{
{name: "short", date: "Sep 28 2016", size: 4096},
{name: "a-much-longer-name", date: "Jan 3 14:32", size: 1536},
}
useColor = true
got := colorRepoLine(in)
if !strings.Contains(got, "Betaflight3.0.0") || !strings.Contains(got, cGreen) || !strings.Contains(got, cYellow) {
t.Errorf("colorRepoLine did not color parts: %q", got)
out := formatRepoList(entries, false)
lines := strings.Split(strings.TrimRight(out, "\n"), "\n")
if len(lines) != 2 {
t.Fatalf("expected 2 lines, got %d: %q", len(lines), out)
}
// order is preserved: `ls -ltr` already sorted by modification time
if !strings.Contains(lines[0], "short") || !strings.Contains(lines[1], "a-much-longer-name") {
t.Errorf("order not preserved: %q", out)
}
// the date starts at the same column on every line
if strings.Index(lines[0], "Sep") != strings.Index(lines[1], "Jan") {
t.Errorf("date column not aligned:\n%s", out)
}
if strings.Contains(out, "4.0K") {
t.Errorf("size shown for repositories: %q", out)
}
if withSize := formatRepoList(entries, true); !strings.Contains(withSize, "4.0K") ||
!strings.Contains(withSize, "1.5K") {
t.Errorf("archive sizes missing: %q", withSize)
}
// colour must decorate the layout, never change it
useColor = true
colored := formatRepoList(entries, false)
useColor = false
strip := func(s string) string {
for _, c := range []string{cReset, cGreen, cYellow, cGray} {
s = strings.ReplaceAll(s, c, "")
}
return s
}
if strip(colored) != out {
t.Errorf("colour changed the layout:\n%q\n%q", strip(colored), out)
}
}
func TestHumanSize(t *testing.T) {
cases := []struct {
n int64
want string
}{
{0, "0B"}, {512, "512B"}, {1024, "1.0K"}, {1536, "1.5K"},
{1024 * 1024, "1.0M"}, {3 * 1024 * 1024 * 1024, "3.0G"},
// past 10 the decimal carries nothing, as with `ls -h`
{512 * 1024, "512K"}, {99 * 1024 * 1024, "99M"},
}
for _, c := range cases {
if got := humanSize(c.n); got != c.want {
t.Errorf("humanSize(%d) = %q, want %q", c.n, got, c.want)
}
}
}
func TestParseConfig(t *testing.T) {
@@ -147,38 +194,49 @@ gitemail = # value is only a comment
}
}
func TestLsEntry(t *testing.T) {
cases := []struct{ line, suffix, want string }{
// ownership is not assumed: any user/group must list
{"drwxr-xr-x 7 git git 4096 Sep 28 2016 myproj.git", ".git", "Sep 28 2016 myproj"},
{"drwxr-xr-x 7 deploy deploy 4096 Sep 28 2016 myproj.git", ".git", "Sep 28 2016 myproj"},
{"drwxr-xr-x 7 mike staff 4096 Sep 28 2016 myproj.git", ".git", "Sep 28 2016 myproj"},
{"drwxr-xr-x. 7 git users 4096 Sep 28 2016 myproj.git", ".git", "Sep 28 2016 myproj"},
// archives only match the archive suffix, and vice versa
{"-rw-r--r-- 1 git git 512 Sep 28 2016 myproj.git.tar.gz", ".git.tar.gz", "Sep 28 2016 myproj"},
{"-rw-r--r-- 1 git git 512 Sep 28 2016 myproj.git.tar.gz", ".git", ""},
{"drwxr-xr-x 7 git git 4096 Sep 28 2016 myproj.git", ".git.tar.gz", ""},
func TestParseLsEntry(t *testing.T) {
cases := []struct {
line, suffix string
name, date string
size int64
ok bool
}{
// ownership is not assumed: any user/group must parse
{"drwxr-xr-x 7 git git 4096 Sep 28 2016 myproj.git", ".git", "myproj", "Sep 28 2016", 4096, true},
{"drwxr-xr-x 7 deploy deploy 4096 Sep 28 2016 myproj.git", ".git", "myproj", "Sep 28 2016", 4096, true},
{"drwxr-xr-x. 7 git users 4096 Sep 28 2016 myproj.git", ".git", "myproj", "Sep 28 2016", 4096, true},
// a recent entry carries a time instead of a year, and still lines up
{"drwxr-xr-x 7 mike staff 224 Jan 3 14:32 myproj.git", ".git", "myproj", "Jan 3 14:32", 224, true},
// a symlinked bare repo lists its target too — only the link name counts
{"lrwxrwxrwx 1 git git 14 Sep 28 2016 myproj.git -> /srv/other.git", ".git", "myproj", "Sep 28 2016", 14, true},
// archives carry a size worth showing
{"-rw-r--r-- 1 git git 524288 Sep 28 2016 myproj.git.tar.gz", ".git.tar.gz", "myproj", "Sep 28 2016", 524288, true},
// suffixes must not cross over
{"-rw-r--r-- 1 git git 512 Sep 28 2016 myproj.git.tar.gz", ".git", "", "", 0, false},
{"drwxr-xr-x 7 git git 4096 Sep 28 2016 myproj.git", ".git.tar.gz", "", "", 0, false},
{"lrwxrwxrwx 1 git git 5 Sep 28 2016 notes -> x.git", ".git", "", "", 0, false},
// non-entries
{"total 48", ".git", ""},
{"", ".git", ""},
{"drwxr-xr-x 7 git git 4096 Sep 28 2016 notes", ".git", ""},
{"total 48", ".git", "", "", 0, false},
{"", ".git", "", "", 0, false},
{"drwxr-xr-x 7 git git 4096 Sep 28 2016 notes", ".git", "", "", 0, false},
}
for _, c := range cases {
if got := lsEntry(c.line, c.suffix); got != c.want {
t.Errorf("lsEntry(%q, %q) = %q, want %q", c.line, c.suffix, got, c.want)
e, ok := parseLsEntry(c.line, c.suffix)
if ok != c.ok {
t.Errorf("parseLsEntry(%q, %q) ok = %v, want %v", c.line, c.suffix, ok, c.ok)
continue
}
if !ok {
continue
}
}
func TestLsEntrySymlink(t *testing.T) {
// a symlinked bare repo lists its target too — only the link name counts
in := "lrwxrwxrwx 1 git git 14 Sep 28 2016 myproj.git -> /srv/other.git"
if got := lsEntry(in, ".git"); got != "Sep 28 2016 myproj" {
t.Errorf("lsEntry(symlink) = %q, want %q", got, "Sep 28 2016 myproj")
if e.name != c.name || e.size != c.size {
t.Errorf("parseLsEntry(%q) = %+v, want name %q size %d", c.line, e, c.name, c.size)
}
// every date renders to the same width, whichever form ls used
if e.date != c.date || len(e.date) != 12 {
t.Errorf("parseLsEntry(%q) date = %q (len %d), want %q at 12",
c.line, e.date, len(e.date), c.date)
}
// and a symlink to something that is not a repo must not match
if got := lsEntry("lrwxrwxrwx 1 git git 5 Sep 28 2016 notes -> x.git", ".git"); got != "" {
t.Errorf("lsEntry(non-repo symlink) = %q, want empty", got)
}
}
@@ -410,7 +468,7 @@ remote.broken.url = https://nowhere.example # no key -> unusable
}
func TestMirrorTargetsLegacyAndSelection(t *testing.T) {
// the flat remoteurl/remotekey pair stays supported, as target "public"
// the pre-4.1 flat pair still loads, folded onto the target "public"
var c Config
applyConfig(&c, parseConfig("remoteurl = https://git.example.com\nremotekey = tok\n"))
usable, _ := c.mirrorTargets()
@@ -749,6 +807,7 @@ func TestTruthy(t *testing.T) {
func TestFormatProjStatus(t *testing.T) {
useColor = false
defer func() { useColor = false }()
w := overviewWidths{label: 12, sync: 5, host: 7}
cases := []struct {
s projStatus
contains []string
@@ -763,10 +822,16 @@ func TestFormatProjStatus(t *testing.T) {
{projStatus{name: "d", branch: "feature", dirty: true},
[]string{"d", "*", "(feature)"}, nil},
{projStatus{name: "e", branch: "master"}, // clean, no upstream
[]string{"e", "no upstream"}, []string{"*"}},
[]string{"e", ""}, []string{"*", "✓"}},
{projStatus{name: "f", branch: "master", hasUpstream: true, ahead: 1, behind: 2},
[]string{"f", "↑1↓2"}, []string{"✓"}}, // diverged shows both
{projStatus{name: "g", branch: "master", hasUpstream: true, lastHost: "laptop",
mirrors: []string{"hub", "gitea"}},
[]string{"g", "laptop", "→ hub gitea"}, nil},
}
for _, c := range cases {
got := formatProjStatus(c.s, 8)
c.s.isRepo = true // these all describe real repositories
got := formatProjStatus(c.s, w)
for _, sub := range c.contains {
if !strings.Contains(got, sub) {
t.Errorf("formatProjStatus(%+v) = %q, missing %q", c.s, got, sub)
@@ -780,6 +845,62 @@ func TestFormatProjStatus(t *testing.T) {
}
}
// TestOverviewColumnsAlign is the point of the table: every field has to start
// at the same column on every row, whatever the name lengths or the multi-byte
// status glyphs do.
func TestOverviewColumnsAlign(t *testing.T) {
useColor = false
// host names must not occur anywhere else in a row, or the index search
// below would find them inside a project or branch name instead
rows := []projStatus{
{name: "a", branch: "master", hasUpstream: true, ahead: 12, behind: 3, lastHost: "workstation"},
{name: "a-very-long-project-name", branch: "wip", dirty: true, lastHost: "buildbox"},
{name: "mid", branch: "main", hasUpstream: true, lastHost: "laptop"},
}
w := measureOverview(rows)
var widths []int
for _, r := range rows {
line := formatProjStatus(r, w)
// the host column starts right after the padded sync field
idx := strings.Index(line, r.lastHost)
if idx < 0 {
t.Fatalf("host %q missing from %q", r.lastHost, line)
}
widths = append(widths, utf8.RuneCountInString(line[:idx]))
}
for i := 1; i < len(widths); i++ {
if widths[i] != widths[0] {
t.Errorf("host column starts at %d on row %d, %d on row 0:\n%s",
widths[i], i, widths[0], strings.Join([]string{
formatProjStatus(rows[0], w), formatProjStatus(rows[i], w)}, "\n"))
}
}
}
func TestAttentionRank(t *testing.T) {
ranks := []struct {
s projStatus
want int
}{
{projStatus{isRepo: true, dirty: true}, 0},
{projStatus{isRepo: true, ahead: 1}, 0},
{projStatus{isRepo: true, behind: 1}, 0},
{projStatus{isRepo: true, hasUpstream: true}, 1},
{projStatus{isRepo: true}, 1}, // clean, no upstream
// not on the server is a different kind of task and goes last, even
// when the working tree is dirty — it cannot be pushed anyway
{projStatus{isRepo: true, notOnServer: true}, 2},
{projStatus{isRepo: true, dirty: true, notOnServer: true}, 2},
{projStatus{notOnServer: true}, 2},
}
for _, c := range ranks {
if got := attentionRank(c.s); got != c.want {
t.Errorf("attentionRank(%+v) = %d, want %d", c.s, got, c.want)
}
}
}
func TestDetectRemoteKind(t *testing.T) {
cases := []struct {
url, override string
@@ -886,3 +1007,143 @@ func TestExpandAlias(t *testing.T) {
}
}
}
func TestSplitAtMarker(t *testing.T) {
lines := []string{"a", "b", "---mgsh---", "c", "d"}
before, after := splitAtMarker(lines, "---mgsh---")
if strings.Join(before, ",") != "a,b" || strings.Join(after, ",") != "c,d" {
t.Errorf("split = %v / %v", before, after)
}
// no marker: everything is the first section, so a server that produced no
// du output simply yields no sizes
before, after = splitAtMarker([]string{"a", "b"}, "---mgsh---")
if strings.Join(before, ",") != "a,b" || after != nil {
t.Errorf("split without marker = %v / %v", before, after)
}
}
func TestParseDuSizes(t *testing.T) {
lines := []string{
"185432\tBetaflight3.0.0.git",
"2144\twebsite.git",
"876 spaced-with-blanks.git", // some du implementations use spaces
"1024\t./with-dot-slash.git",
"1500\tmy project.git", // a name with a space survives
"garbage",
"",
}
got := parseDuSizes(lines)
want := map[string]int64{
"Betaflight3.0.0.git": 185432 * 1024,
"website.git": 2144 * 1024,
"spaced-with-blanks.git": 876 * 1024,
"with-dot-slash.git": 1024 * 1024,
"my project.git": 1500 * 1024,
}
if len(got) != len(want) {
t.Fatalf("parseDuSizes = %v, want %d entries", got, len(want))
}
for k, v := range want {
if got[k] != v {
t.Errorf("parseDuSizes[%q] = %d, want %d", k, got[k], v)
}
}
}
// TestMigrateRemoteKeys rewrites the pre-4.1 flat spelling in place. Only the
// key changes: values, comments and everything else stay byte for byte.
func TestMigrateRemoteKeys(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, ".mgshrc")
before := `# my config
base = /home/me/src
remoteurl = https://git.example.com # the mirror
remotekey = s3cr3t-token
remotevisibility: public
# remotetype = gitea (commented out, must stay put)
alias co 'checkout $1'
`
if err := os.WriteFile(path, []byte(before), 0600); err != nil {
t.Fatal(err)
}
out := captureStdout(t, func() { migrateRemoteKeys(path, before) })
got := readFile(t, path)
for _, want := range []string{
"remote.public.url = https://git.example.com # the mirror",
"remote.public.key = s3cr3t-token",
" remote.public.visibility: public",
"# remotetype = gitea (commented out, must stay put)",
"base = /home/me/src",
"alias co 'checkout $1'",
} {
if !strings.Contains(got, want) {
t.Errorf("migrated file missing %q:\n%s", want, got)
}
}
if strings.Contains(got, "\nremoteurl") || strings.Contains(got, "\nremotekey") {
t.Errorf("old spelling left behind:\n%s", got)
}
if !strings.Contains(out, "remoteurl → remote.public.url") {
t.Errorf("migration was not reported: %q", out)
}
// the file keeps its private mode
if fi, err := os.Stat(path); err != nil {
t.Fatal(err)
} else if fi.Mode().Perm() != 0o600 {
t.Errorf("mode after migration = %04o, want 0600", fi.Mode().Perm())
}
// running again changes nothing and says nothing
second := captureStdout(t, func() { migrateRemoteKeys(path, readFile(t, path)) })
if strings.TrimSpace(second) != "" {
t.Errorf("a converted file was migrated again: %q", second)
}
if readFile(t, path) != got {
t.Error("a second migration changed the file")
}
}
// TestLegacyKeysDoNotOverrideExplicitOnes: a config carrying both spellings must
// keep what the new one says.
func TestLegacyKeysDoNotOverrideExplicitOnes(t *testing.T) {
var c Config
applyConfig(&c, parseConfig(
"remoteurl = https://old.example\nremote.public.url = https://new.example\n"+
"remote.public.key = tok\n"))
targets, _ := c.mirrorTargets()
if len(targets) != 1 || targets[0].URL != "https://new.example" {
t.Errorf("targets = %+v, want the remote.public.url value", targets)
}
}
// TestRemoteEnvOverrides: MGSH_REMOTE_<NAME>_<FIELD> is the environment
// spelling of remote.<name>.<field>.
func TestRemoteEnvOverrides(t *testing.T) {
t.Setenv("MGSH_REMOTE_GITLAB_URL", "https://gitlab.example")
t.Setenv("MGSH_REMOTE_GITLAB_KEY", "env-token")
t.Setenv("MGSH_REMOTE_GITLAB_VISIBILITY", "public")
t.Setenv("MGSH_REMOTE_MY_HUB_URL", "https://hub.example") // name with an underscore
t.Setenv("MGSH_REMOTE_MY_HUB_KEY", "hub-token")
t.Setenv("MGSH_REMOTES", "") // must not be mistaken for a target field
var c Config
applyConfig(&c, parseConfig("remote.gitlab.url = https://from-file.example\nremote.gitlab.key = file-token\n"))
applyEnv(&c)
targets, incomplete := c.mirrorTargets()
if len(incomplete) != 0 {
t.Fatalf("incomplete targets: %v", incomplete)
}
byName := map[string]RemoteTarget{}
for _, tg := range targets {
byName[tg.Name] = tg
}
if g := byName["gitlab"]; g.URL != "https://gitlab.example" || g.Key != "env-token" || g.Vis != "public" {
t.Errorf("env did not override the file: %+v", g)
}
// the field is taken from the end, so the name may contain underscores
if h := byName["my_hub"]; h.URL != "https://hub.example" || h.Key != "hub-token" {
t.Errorf("MGSH_REMOTE_MY_HUB_* = %+v, want target my_hub", h)
}
}
+19 -13
View File
@@ -23,25 +23,31 @@ gitpath = /home/git
# editor = code
# --- pushremote: mirror to public servers (gitea/github/gitlab) via their API ---
# A single server, the flat form (this target is named "public"):
# remoteurl = https://git.example.com
# remotekey = <personal-access-token>
# remotetype = gitea # optional; auto-detected from remoteurl
# remotevisibility = private # visibility of created repos (default private)
# One "remote.<name>.<field>" block per server, with the fields url, key, type
# and visibility. <name> is yours to pick and becomes the git remote created in
# the repository, so `git push gitlab` keeps working outside mgsh.
#
# `pushremote` pushes to every configured server in the order given,
# `pushremote @gitlab` to a single one.
#
# Or any number of named servers. `pushremote` pushes to all of them in the
# order given, `pushremote @hub` to a single one. Each target gets a git remote
# of the same name in the repository.
# remote.gitea.url = https://git.example.com
# remote.gitea.key = <personal-access-token>
# remote.hub.url = https://github.com
# remote.hub.key = <personal-access-token>
# remote.hub.type = github # optional; auto-detected from the url
# remote.hub.visibility = public # default private
# remotes = gitea, hub # optional: restrict and order the set
# remote.gitea.type = gitea # optional; auto-detected from the url
# remote.gitea.visibility = private # or public (default private)
#
# remote.gitlab.url = https://gitlab.example.com
# remote.gitlab.key = <personal-access-token>
# remote.gitlab.type = gitlab
# remote.gitlab.visibility = public
#
# remotes = gitea, gitlab # optional: restrict and order the set
# mirror = true # `push` also mirrors via pushremote
# --- safety ---
# `push` checks the staged diff for private keys and API tokens before it
# commits, and asks before continuing. Only an explicit "off" disables it.
# secretscan = off
# --- per-project overrides ---
# A <project>/.mgshrc overrides all of the above for that project only, except
# base, gitname, gitemail and pushdefault, which stay global. Typical use:
+316 -54
View File
@@ -1,25 +1,49 @@
package main
// overview.go — the `overview` command (also reachable as `status -a`): a
// one-line-per-project summary of every git project under BASE, showing the
// dirty state and how far each branch is ahead/behind its upstream.
// overview.go — the `overview` command (also reachable as `status -a`).
//
// mgsh is the only thing that sees all three places a project can live: the
// local base directory, the internal ssh server, and the public mirrors. Joining
// those answers the questions plain git cannot — which projects were never
// pushed to the server, and which machine last touched each one (every `push`
// stamps "[user@host]" into the commit message, so that comes for free).
import (
"fmt"
"os"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
"unicode/utf8"
)
// projStatus is the collected state of one project for the overview.
type projStatus struct {
name string
branch string
isRepo bool // has a .git of its own
dirty bool
ahead, behind int
hasUpstream bool
notOnServer bool // known to be missing from the git server
lastHost string // machine that made the last commit, from "[user@host]"
lastWhen time.Time // when that was
mirrors []string // configured mirror remotes present in this repo
}
// overviewAll prints a status summary for all git projects under BASE.
// commitHostRe pulls the host out of the "[user@host] subject" line that `push`
// writes, so the overview can say where a project was last worked on.
var commitHostRe = regexp.MustCompile(`^\[[^@\]]*@([^\]]+)\]`)
// overviewScanLimit bounds how many projects are inspected at once. The work is
// all subprocess latency, so some concurrency helps a lot and more does not.
const overviewScanLimit = 8
// overviewAll prints a status summary for all git projects under BASE, plus the
// projects that exist on only one side of the local/server divide.
func overviewAll() {
entries, err := os.ReadDir(BASE)
if err != nil {
@@ -27,30 +51,52 @@ func overviewAll() {
return
}
var rows []projStatus
width := 0
// ask the server while the local tree is being walked
type serverList struct {
names []string
err error
}
srvCh := make(chan serverList, 1)
go func() {
names, err := serverRepoNames()
srvCh <- serverList{names, err}
}()
// every directory gets a row, repository or not: one that is not a
// repository yet is exactly what `init` is for, and putting it in the table
// beats a separate list underneath
var local []string
for _, e := range entries {
if !e.IsDir() || strings.HasPrefix(e.Name(), ".") {
continue
}
dir := BASE + "/" + e.Name()
if !isDir(dir + "/.git") {
continue
}
rows = append(rows, projectStatus(e.Name(), dir))
if len(e.Name()) > width {
width = len(e.Name())
if e.IsDir() && !strings.HasPrefix(e.Name(), ".") {
local = append(local, e.Name())
}
}
rows := scanProjects(local)
srv := <-srvCh
markUnpublished(rows, srv.names, srv.err)
// what needs doing first, alphabetical within each group
sort.SliceStable(rows, func(i, j int) bool {
return attentionRank(rows[i]) < attentionRank(rows[j])
})
if len(rows) == 0 {
fmt.Println(col(cGray, "no git projects under "+BASE))
fmt.Println(col(cGray, "nothing under "+BASE))
return
}
dirtyN, syncN := 0, 0
w := measureOverview(rows)
repoN, dirtyN, syncN, initN := 0, 0, 0, 0
for _, r := range rows {
fmt.Println(formatProjStatus(r, width))
fmt.Println(formatProjStatus(r, w))
if r.notOnServer {
initN++
}
if !r.isRepo {
continue
}
repoN++
if r.dirty {
dirtyN++
}
@@ -58,51 +104,267 @@ func overviewAll() {
syncN++
}
}
fmt.Printf("%s\n", col(cGray, fmt.Sprintf("%d projects · %d dirty · %d in sync", len(rows), dirtyN, syncN)))
summary := fmt.Sprintf("%d projects · %d dirty · %d in sync", repoN, dirtyN, syncN)
if initN > 0 {
summary += fmt.Sprintf(" · %d to init", initN)
}
fmt.Println(col(cGray, summary))
if srv.err != nil {
fmt.Println(col(cGray, " git server not reachable — local view only"))
}
}
// projectStatus gathers the git state of a single project directory.
// scanProjects collects the state of every project concurrently. Each project
// costs two git subprocesses, and serially that is the slowest thing mgsh does.
func scanProjects(names []string) []projStatus {
rows := make([]projStatus, len(names))
sem := make(chan struct{}, overviewScanLimit)
var wg sync.WaitGroup
for i, n := range names {
wg.Add(1)
go func(i int, n string) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
rows[i] = projectStatus(n, BASE+"/"+n)
}(i, n)
}
wg.Wait()
return rows
}
// markUnpublished flags the rows the git server has never seen — the ones
// `init` is for. A listing that failed leaves every row unmarked: not knowing
// is not the same as knowing they are missing, and marking all of them would
// tell the user to re-init their whole base directory.
func markUnpublished(rows []projStatus, server []string, err error) {
if err != nil {
return
}
onServer := map[string]bool{}
for _, n := range server {
onServer[n] = true
}
for i := range rows {
rows[i].notOnServer = !onServer[rows[i].name]
}
}
// projectStatus gathers the git state of a single project directory. A
// directory without a repository is reported as it is, and costs no
// subprocesses at all.
func projectStatus(name, dir string) projStatus {
s := projStatus{name: name, branch: "-"}
if out, err := gitCapture(dir, "rev-parse", "--abbrev-ref", "HEAD"); err == nil {
s.branch = strings.TrimSpace(out)
}
if out, err := gitCapture(dir, "status", "--porcelain"); err == nil && strings.TrimSpace(out) != "" {
s.dirty = true
}
// left/right counts against the upstream: "<behind>\t<ahead>"
if out, err := gitCapture(dir, "rev-list", "--left-right", "--count", "@{upstream}...HEAD"); err == nil {
if _, e := fmt.Sscanf(strings.TrimSpace(out), "%d\t%d", &s.behind, &s.ahead); e == nil {
s.hasUpstream = true
}
if s.isRepo = isDir(dir + "/.git"); !s.isRepo {
return s
}
readStatus(&s, dir)
readLastCommit(&s, dir)
s.mirrors = configuredMirrors(dir)
return s
}
// formatProjStatus renders one aligned overview row.
func formatProjStatus(s projStatus, width int) string {
var marks []string
// readStatus fills in branch, upstream, ahead/behind and dirty from a single
// `git status` — the porcelain v2 header carries all four.
func readStatus(s *projStatus, dir string) {
out, err := gitCapture(dir, "status", "--porcelain=v2", "--branch")
if err != nil {
return
}
for _, ln := range splitLines(out) {
if !strings.HasPrefix(ln, "# ") {
s.dirty = true // any entry line means the tree is not clean
continue
}
f := strings.Fields(ln)
if len(f) < 3 {
continue
}
switch f[1] {
case "branch.head":
s.branch = f[2]
case "branch.upstream":
s.hasUpstream = true
case "branch.ab":
if len(f) >= 4 {
s.ahead, _ = strconv.Atoi(strings.TrimPrefix(f[2], "+"))
s.behind, _ = strconv.Atoi(strings.TrimPrefix(f[3], "-"))
}
}
}
}
// readLastCommit records when the project was last committed to and from which
// machine, taken from the "[user@host]" prefix `push` writes.
func readLastCommit(s *projStatus, dir string) {
out, err := gitCapture(dir, "log", "-1", "--format=%ct%x00%s")
if err != nil {
return
}
parts := strings.SplitN(strings.TrimSpace(out), "\x00", 2)
if len(parts) != 2 {
return
}
if epoch, err := strconv.ParseInt(parts[0], 10, 64); err == nil {
s.lastWhen = time.Unix(epoch, 0)
}
if m := commitHostRe.FindStringSubmatch(parts[1]); m != nil {
s.lastHost = m[1]
}
}
// configuredMirrors returns the mirror targets this repository actually has a
// remote for — free to determine, since it is only local git config.
func configuredMirrors(dir string) []string {
targets, _ := cfg.mirrorTargets()
if len(targets) == 0 {
return nil
}
out, err := gitCapture(dir, "remote")
if err != nil {
return nil
}
have := map[string]bool{}
for _, r := range splitLines(out) {
have[strings.TrimSpace(r)] = true
}
var found []string
for _, t := range targets {
if have[t.Name] {
found = append(found, t.Name)
}
}
return found
}
// overviewWidths are the column widths of the overview table, measured from the
// rows so every field starts at the same place. Ragged columns were what made
// the old one-line-per-project output hard to read.
type overviewWidths struct {
label, sync, host int
hint bool // any row carries an action hint
}
// hintWidth is the width of the action column, sized for its only word.
const hintWidth = 4
// measureOverview sizes the columns for a set of rows.
func measureOverview(rows []projStatus) overviewWidths {
var w overviewWidths
for _, r := range rows {
w.label = max(w.label, utf8.RuneCountInString(projLabel(r)))
w.sync = max(w.sync, utf8.RuneCountInString(syncState(r)))
w.host = max(w.host, utf8.RuneCountInString(r.lastHost))
w.hint = w.hint || r.notOnServer
}
return w
}
// projLabel is the first column: the project, with its branch appended when it
// is not the usual one. Keeping the branch attached to the name costs no extra
// column and keeps the table narrow.
func projLabel(s projStatus) string {
if s.branch != "" && s.branch != "-" && s.branch != "master" && s.branch != "main" {
return s.name + " (" + s.branch + ")"
}
return s.name
}
// syncState renders the relation to the upstream as one short field: ahead,
// behind, both, in sync, or "" for a branch that tracks nothing. The old
// spelled-out "(no upstream)" was fifteen columns wide and pushed every
// following field out of line.
func syncState(s projStatus) string {
switch {
case !s.isRepo:
return "" // nothing to compare: there is no repository here yet
case s.ahead > 0 && s.behind > 0:
return fmt.Sprintf("↑%d↓%d", s.ahead, s.behind)
case s.ahead > 0:
return fmt.Sprintf("↑%d", s.ahead)
case s.behind > 0:
return fmt.Sprintf("↓%d", s.behind)
case s.hasUpstream:
return "✓"
default:
return ""
}
}
// syncColor weights a row visually: anything needing action is coloured, a
// project that is clean and in sync recedes into grey.
func syncColor(s projStatus) string {
switch {
case s.behind > 0:
return cRed
case s.ahead > 0:
return cGreen
default:
return cGray
}
}
// attentionRank groups the rows: work in progress at the top, then everything
// that is settled, and last the directories the server does not have yet. With
// many projects, scanning the whole list for the two dirty ones is the actual
// work — and an un-inited directory is a different kind of task, not something
// to push past the daily ones.
func attentionRank(s projStatus) int {
switch {
case s.notOnServer:
return 2
case s.dirty || s.ahead > 0 || s.behind > 0:
return 0
default:
return 1
}
}
// formatProjStatus renders one row of the overview table.
func formatProjStatus(s projStatus, w overviewWidths) string {
dirty := " "
if s.dirty {
marks = append(marks, col(cYellow, "*"))
}
if s.ahead > 0 {
marks = append(marks, col(cGreen, fmt.Sprintf("↑%d", s.ahead)))
}
if s.behind > 0 {
marks = append(marks, col(cRed, fmt.Sprintf("↓%d", s.behind)))
}
state := strings.Join(marks, " ")
if state == "" {
if s.hasUpstream {
state = col(cGreen, "✓")
} else {
state = col(cGray, "✓ (no upstream)")
}
dirty = "*"
}
line := " " + col(cGreen, padRight(s.name, width+2)) + state
if s.branch != "master" && s.branch != "main" && s.branch != "-" {
line += col(cGray, " ("+s.branch+")")
var b strings.Builder
b.WriteString(" ")
b.WriteString(col(cGreen, padRight(projLabel(s), w.label)))
b.WriteString(" " + col(cYellow, dirty) + " ")
b.WriteString(col(syncColor(s), padRight(syncState(s), w.sync)))
if w.host > 0 {
age := ""
if !s.lastWhen.IsZero() {
age = shortAge(time.Since(s.lastWhen))
}
b.WriteString(" " + col(cGray, padRight(s.lastHost, w.host)))
b.WriteString(" " + col(cGray, fmt.Sprintf("%4s", age)))
}
if w.hint {
hint := ""
if s.notOnServer {
hint = "init"
}
b.WriteString(" " + col(cYellow, padRight(hint, hintWidth)))
}
if len(s.mirrors) > 0 {
b.WriteString(col(cGray, " → "+strings.Join(s.mirrors, " ")))
}
return strings.TrimRight(b.String(), " ")
}
// shortAge renders a duration compactly: 90s -> "1m", 36h -> "1d".
func shortAge(d time.Duration) string {
switch {
case d < time.Minute:
return "now"
case d < time.Hour:
return fmt.Sprintf("%dm", int(d.Minutes()))
case d < 24*time.Hour:
return fmt.Sprintf("%dh", int(d.Hours()))
default:
return fmt.Sprintf("%dd", int(d.Hours()/24))
}
return line
}
+182
View File
@@ -0,0 +1,182 @@
package main
// overview_test.go — the inventory view: what mgsh knows that plain git cannot.
import (
"errors"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// TestReadStatusParsesPorcelainV2 covers the single `git status` call that
// replaced three separate ones: branch, upstream, ahead/behind and dirty all
// come out of its header.
func TestReadStatusParsesPorcelainV2(t *testing.T) {
dir := t.TempDir()
bare := filepath.Join(t.TempDir(), "o.git")
mustGit(t, "", "init", "--bare", "-q", bare)
mustGit(t, dir, "init", "-q")
mustGit(t, dir, "config", "user.name", "t")
mustGit(t, dir, "config", "user.email", "t@e")
mustGit(t, dir, "commit", "-q", "--allow-empty", "-m", "[mike@laptop] work")
mustGit(t, dir, "remote", "add", "origin", bare)
mustGit(t, dir, "push", "-q", "-u", "origin", "HEAD")
var s projStatus
readStatus(&s, dir)
if !s.hasUpstream || s.ahead != 0 || s.behind != 0 || s.dirty {
t.Errorf("clean synced repo = %+v", s)
}
if s.branch == "" || s.branch == "-" {
t.Errorf("branch not read: %q", s.branch)
}
// one unstaged file and one unpushed commit
if err := os.WriteFile(filepath.Join(dir, "x"), []byte("x"), 0644); err != nil {
t.Fatal(err)
}
mustGit(t, dir, "commit", "-q", "--allow-empty", "-m", "[mike@desktop] more")
s = projStatus{}
readStatus(&s, dir)
if !s.dirty {
t.Error("untracked file did not register as dirty")
}
if s.ahead != 1 {
t.Errorf("ahead = %d, want 1", s.ahead)
}
// and the host stamp `push` writes is picked up
var l projStatus
readLastCommit(&l, dir)
if l.lastHost != "desktop" {
t.Errorf("lastHost = %q, want desktop", l.lastHost)
}
if l.lastWhen.IsZero() {
t.Error("lastWhen not read")
}
}
// TestCommitHostRe: only mgsh's own "[user@host]" stamp counts.
func TestCommitHostRe(t *testing.T) {
cases := map[string]string{
"[mike@laptop] fixed a thing": "laptop",
"[mike@build-01] ": "build-01",
"[@host] no user": "host",
"fixed a thing": "",
"[not a stamp] text": "",
"see [a@b] mid-line": "",
}
for subj, want := range cases {
got := ""
if m := commitHostRe.FindStringSubmatch(subj); m != nil {
got = m[1]
}
if got != want {
t.Errorf("host of %q = %q, want %q", subj, got, want)
}
}
}
// TestMarkUnpublishedUnreachableServer: a listing that failed must leave every
// row unmarked. Not knowing is not the same as knowing they are missing —
// marking all of them would tell the user to re-init their whole base.
func TestMarkUnpublishedUnreachableServer(t *testing.T) {
rows := []projStatus{{name: "a"}, {name: "b"}}
markUnpublished(rows, nil, errors.New("network is unreachable"))
for _, r := range rows {
if r.notOnServer {
t.Errorf("%s marked as missing although the server could not be listed", r.name)
}
}
}
// TestMarkUnpublished flags only what the server really does not have.
func TestMarkUnpublished(t *testing.T) {
rows := []projStatus{{name: "both"}, {name: "onlyhere"}}
markUnpublished(rows, []string{"both", "onlythere"}, nil)
if rows[0].notOnServer {
t.Error("a project present on both sides was marked")
}
if !rows[1].notOnServer {
t.Error("a local-only project was not marked")
}
}
// TestUnpublishedRowsCarryTheHint: the entries live in the table now, with the
// action in their own column, rather than in a list underneath it.
func TestUnpublishedRowsCarryTheHint(t *testing.T) {
useColor = false
rows := []projStatus{
{name: "published", isRepo: true, hasUpstream: true},
{name: "fresh", isRepo: true, notOnServer: true},
{name: "notarepo", notOnServer: true},
}
w := measureOverview(rows)
if !w.hint {
t.Fatal("hint column not reserved although rows need it")
}
got := []string{}
for _, r := range rows {
got = append(got, formatProjStatus(r, w))
}
if strings.Contains(got[0], "init") {
t.Errorf("a published project was hinted: %q", got[0])
}
for _, i := range []int{1, 2} {
if !strings.Contains(got[i], "init") {
t.Errorf("row %d missing the init hint: %q", i, got[i])
}
}
// a directory that is not a repository has no sync state to report
if strings.ContainsAny(got[2], "✓–↑↓") {
t.Errorf("non-repository row claims a git state: %q", got[2])
}
// with nothing to hint the column disappears entirely
if measureOverview(rows[:1]).hint {
t.Error("hint column reserved although no row needs it")
}
}
func TestShortAge(t *testing.T) {
cases := []struct {
d time.Duration
want string
}{
{30 * time.Second, "now"},
{90 * time.Second, "1m"},
{2 * time.Hour, "2h"},
{36 * time.Hour, "1d"},
{20 * 24 * time.Hour, "20d"},
}
for _, c := range cases {
if got := shortAge(c.d); got != c.want {
t.Errorf("shortAge(%v) = %q, want %q", c.d, got, c.want)
}
}
}
// captureStdout collects everything a function prints.
func captureStdout(t *testing.T, fn func()) string {
t.Helper()
old := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stdout = w
done := make(chan string)
go func() {
var b strings.Builder
io.Copy(&b, r)
done <- b.String()
}()
fn()
w.Close()
os.Stdout = old
return <-done
}
+11 -16
View File
@@ -4,23 +4,18 @@ package main
// git hosting server (Gitea, GitHub or GitLab), creating the repository via the
// server's REST API when it does not exist yet.
//
// Configuration (in ~/.mgshrc, a project .mgshrc, or MGSH_* env) — either a
// single flat target:
// Configuration (in ~/.mgshrc, a project .mgshrc, or MGSH_* env): one
// remote.<name>.<field> block per server, which `pushremote` mirrors to in turn.
//
// remoteurl = https://git.example.com base URL of the server
// remotekey = <api-token> personal access token
// remotetype = gitea|github|gitlab optional; auto-detected from the URL
// remote.gitlab.url = https://gitlab.example.com
// remote.gitlab.key = <api-token>
// remote.gitlab.type = gitlab optional; detected from the url
// remote.gitlab.visibility = public or private (the default)
// remotes = gitlab optional: restrict/order the set
//
// or any number of named ones, which `pushremote` mirrors to in turn:
//
// remote.gitea.url = https://git.example.com
// remote.gitea.key = <api-token>
// remote.hub.url = https://github.com
// remote.hub.key = <api-token>
// remote.hub.visibility = public
// remotes = gitea, hub optional: restrict/order the set
//
// Each target owns a git remote of the same name in the repository.
// Each target owns a git remote of its own name in the repository. There is no
// second spelling: the pre-4.1 flat remoteurl/remotekey pair is migrated to
// remote.public.* on load.
//
// The token is used for the API calls and, via an HTTP Basic auth header, for
// the git push. It is never written into the repository's git config, and it is
@@ -287,7 +282,7 @@ func handlePushRemote(args string) {
targets = pickRemotes(targets, names)
if len(targets) == 0 {
if len(names) == 0 { // an unknown @name already reported itself
errorln("pushremote needs 'remoteurl'/'remotekey' or a 'remote.<name>.*' block in " + configFile())
errorln("pushremote needs a 'remote.<name>.url' and 'remote.<name>.key' in " + configFile())
}
return
}
+62
View File
@@ -329,3 +329,65 @@ func mustGit(t *testing.T, dir string, args ...string) {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
}
// TestListSurvivesFailingDu: `list` chains the listing and `du` into one remote
// command, and the exit status is the *last* command's. A server whose du fails
// — a shell that mis-parses the arguments, a du that is not there, a permission
// problem — must still get its repositories listed.
func TestListSurvivesFailingDu(t *testing.T) {
useProject(t, "x")
fakeServer(t, func(cmd string) (string, error) {
return "total 4\n" +
"drwxr-xr-x 7 git git 4096 Jan 3 14:32 notes.git\n" +
"drwxr-xr-x 7 git git 4096 Sep 28 2016 website.git\n" +
listMarker + "\n",
errors.New("exit status 1") // du blew up, ls did not
})
out := captureStdout(t, func() { runCommand("list") })
if strings.Contains(out, "could not list") {
t.Errorf("a failing du discarded a good listing:\n%s", out)
}
for _, want := range []string{"notes", "website", "2 repositories"} {
if !strings.Contains(out, want) {
t.Errorf("listing missing %q:\n%s", want, out)
}
}
// without sizes there must be no size column, not a column of zeroes
if strings.Contains(out, "0B") {
t.Errorf("zero sizes shown when du produced none:\n%s", out)
}
}
// TestListReportsATrulyFailedListing: when nothing usable came back, the error
// still has to surface.
func TestListReportsATrulyFailedListing(t *testing.T) {
useProject(t, "x")
fakeServer(t, func(cmd string) (string, error) {
return "", errors.New("ssh: connect failed")
})
out := captureStdout(t, func() { runCommand("list") })
if !strings.Contains(out, "could not list") {
t.Errorf("a failed listing was not reported:\n%s", out)
}
}
// TestListSendsNoShellSpecificSyntax guards the bug this replaced: the remote
// command is run by the git user's login shell, which may be csh, where
// "2>/dev/null" is an argument followed by a redirection rather than a
// redirection of stderr.
func TestListSendsNoShellSpecificSyntax(t *testing.T) {
useProject(t, "x")
sent := fakeServer(t, func(cmd string) (string, error) { return "", nil })
captureStdout(t, func() { runCommand("list") })
if len(*sent) == 0 {
t.Fatal("list sent nothing")
}
for _, c := range *sent {
if strings.Contains(c, "2>") || strings.Contains(c, "&>") {
t.Errorf("remote command uses sh-only redirection: %q", c)
}
}
}
+206
View File
@@ -0,0 +1,206 @@
package main
// secrets.go — a last look at what `push` is about to commit.
//
// `push` runs `git add --all .`, so anything lying in the project — an .env, a
// stray key file, a token pasted into a config — is committed and pushed, and
// with `mirror = true` it reaches a *public* server in the same breath. That is
// the one action in mgsh that cannot be undone: a deleted server repository can
// come back from an archive, a published credential is burnt.
//
// So the staged diff is scanned for a small set of high-signal patterns before
// the commit is made. This is not a complete secret scanner and does not try to
// be one; it aims for a high hit rate on the things that actually leak, with
// few enough false alarms that the prompt still means something. Turn it off
// with `secretscan = off`.
import (
"fmt"
"regexp"
"strconv"
"strings"
)
// secretHit is one suspicious added line.
type secretHit struct {
file string
lineNo int
kind string
text string
}
// secretPattern matches one kind of credential. `certain` patterns are
// unmistakable and are reported as they are; the others match a shape that
// merely looks like a secret and are filtered through looksLikePlaceholder.
type secretPattern struct {
kind string
re *regexp.Regexp
certain bool
}
var secretPatterns = []secretPattern{
{"private key", regexp.MustCompile(`-----BEGIN (?:[A-Z]+ )?PRIVATE KEY-----`), true},
{"GitHub token", regexp.MustCompile(`\bgh[pousr]_[A-Za-z0-9]{20,}`), true},
{"GitLab token", regexp.MustCompile(`\bglpat-[A-Za-z0-9_-]{16,}`), true},
{"AWS access key", regexp.MustCompile(`\b(?:AKIA|ASIA)[0-9A-Z]{16}\b`), true},
{"Slack token", regexp.MustCompile(`\bxox[baprs]-[A-Za-z0-9-]{10,}`), true},
{"PyPI token", regexp.MustCompile(`\bpypi-AgEIcHlwaS5vcmc[A-Za-z0-9_-]{10,}`), true},
{"credential assignment", regexp.MustCompile(
`(?i)\b(?:password|passwd|secret|api[_-]?key|apikey|access[_-]?key|auth[_-]?token|token)\b` +
`\s*[:=]\s*(?:"([^"\s]{12,})"|'([^'\s]{12,})'|([^\s"';,]{20,}))\s*;?\s*$`), false},
}
var (
diffFileRe = regexp.MustCompile(`^\+\+\+ b/(.*)$`)
diffHunkRe = regexp.MustCompile(`^@@ -\d+(?:,\d+)? \+(\d+)`)
// a value that is plainly a reference or a stand-in, not a credential
constRefRe = regexp.MustCompile(`^[A-Z][A-Z0-9_]*$`)
dottedRefRe = regexp.MustCompile(`^[\w-]+(?:\.[\w-]+)+$`)
maskedRe = regexp.MustCompile(`^[*x•.]+$`)
)
// placeholderWords are the values people write when they mean "fill this in".
var placeholderWords = map[string]bool{
"changeme": true, "change_me": true, "password": true, "secret": true,
"token": true, "your_token": true, "your-token": true, "yourtoken": true,
"todo": true, "none": true, "null": true, "example": true, "redacted": true,
}
// looksLikePlaceholder reports whether a matched value is obviously not a real
// credential: a template slot, an environment reference, a constant name, or a
// masked stand-in. Documentation and example files are full of these, and every
// one of them that reaches the prompt makes the prompt worth less.
func looksLikePlaceholder(v string) bool {
v = strings.Trim(v, `"'`)
if v == "" {
return true
}
if strings.ContainsAny(v, "<>${}()") { // <token>, ${VAR}, $(cmd), {{ tpl }}
return true
}
if maskedRe.MatchString(v) || constRefRe.MatchString(v) || dottedRefRe.MatchString(v) {
return true
}
if placeholderWords[strings.ToLower(v)] {
return true
}
// a value made of one repeated character carries no information
if strings.Count(v, string(v[0])) == len(v) {
return true
}
return false
}
// scanDiff finds suspicious added lines in a unified diff. Only added lines are
// examined: removing a secret is what we want people to do.
func scanDiff(diff string) []secretHit {
var hits []secretHit
file := ""
lineNo := 0
for _, ln := range strings.Split(diff, "\n") {
switch {
case strings.HasPrefix(ln, "+++ "):
file = ""
if m := diffFileRe.FindStringSubmatch(ln); m != nil {
file = m[1]
}
continue
case strings.HasPrefix(ln, "@@"):
if m := diffHunkRe.FindStringSubmatch(ln); m != nil {
lineNo, _ = strconv.Atoi(m[1])
}
continue
case strings.HasPrefix(ln, "---") || strings.HasPrefix(ln, "diff ") ||
strings.HasPrefix(ln, "index ") || strings.HasPrefix(ln, "new file") ||
strings.HasPrefix(ln, "deleted file") || strings.HasPrefix(ln, "similarity "):
continue
case strings.HasPrefix(ln, "-"):
continue // removed line: not our problem
case !strings.HasPrefix(ln, "+"):
lineNo++ // context line
continue
}
text := ln[1:]
if kind := matchSecret(text); kind != "" {
hits = append(hits, secretHit{file: file, lineNo: lineNo, kind: kind, text: text})
}
lineNo++
}
return hits
}
// allowMarker suppresses the check for one line. Any scanner needs a per-line
// escape: a project will eventually hold something credential-shaped on
// purpose, and switching the whole check off for that is far too blunt.
const allowMarker = "mgsh:allow"
// matchSecret returns the kind of credential a line appears to contain, or "".
func matchSecret(text string) string {
if strings.Contains(text, allowMarker) {
return ""
}
for _, p := range secretPatterns {
m := p.re.FindStringSubmatch(text)
if m == nil {
continue
}
if p.certain {
return p.kind
}
// the first non-empty capture group is the value that was assigned
value := ""
for _, g := range m[1:] {
if g != "" {
value = g
break
}
}
if !looksLikePlaceholder(value) {
return p.kind
}
}
return ""
}
// secretScanEnabled reports whether the scan runs. It is on unless explicitly
// switched off, so a typo in the setting leaves the safety net in place.
func secretScanEnabled() bool { return !falsy(cfg.SecretScan) }
// secretsApproved scans what `push` has staged. With nothing suspicious found
// it returns true silently; otherwise it shows the findings and asks. Returns
// false when the push should stop.
func secretsApproved(dir string) bool {
if !secretScanEnabled() {
return true
}
diff, err := gitCapture(dir, "diff", "--cached", "-U0", "--no-color")
if err != nil {
return true // nothing staged, or no HEAD yet: not our call to block
}
hits := scanDiff(diff)
if len(hits) == 0 {
return true
}
fmt.Println(col(cRed, fmt.Sprintf("%d possible credential(s) in what is about to be committed:", len(hits))))
for _, h := range hits {
where := h.file
if h.lineNo > 0 {
where += ":" + strconv.Itoa(h.lineNo)
}
fmt.Printf(" %s %s\n %s\n",
col(cYellow, where), col(cGray, h.kind), col(cRed, ellipsis(strings.TrimSpace(h.text), 100)))
}
fmt.Println(col(cGray, " (set 'secretscan = off' to skip this check)"))
return yesno("push anyway?", false)
}
// ellipsis shortens s to at most n characters.
func ellipsis(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n-1] + "…"
}
+212
View File
@@ -0,0 +1,212 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// TestMatchSecretCatchesRealCredentials: the shapes that actually leak.
func TestMatchSecretCatchesRealCredentials(t *testing.T) {
lines := []string{
"-----BEGIN OPENSSH PRIVATE KEY-----",
"-----BEGIN RSA PRIVATE KEY-----",
" ghp_aB3dEfGh1jKlMn0pQrStUvWxYz012345678",
"GITLAB=glpat-aB3dEfGh1jKlMn0pQrSt",
`aws_access_key_id = AKIAIOSFODNN7EXAMPLE`,
"slack: xoxb-1234567890-abcdefghij",
`API_KEY="s3cr3tV4lu3W1thStuff"`,
"password = hunter2hunter2hunter2",
"token: 'aB3dEfGh1jKlMn0pQrSt'",
"auth-token=9f8e7d6c5b4a39281706abcdef123456",
}
for _, ln := range lines {
if matchSecret(ln) == "" {
t.Errorf("missed a credential in %q", ln)
}
}
}
// TestMatchSecretIgnoresNoise: everyday code and documentation must not trip
// the prompt, or people learn to answer "yes" without reading it.
func TestMatchSecretIgnoresNoise(t *testing.T) {
lines := []string{
"remotekey = <personal-access-token>", // our own README
"# remotekey = <personal-access-token>", // and mgshrc.example
"token = process.env.GITHUB_TOKEN", // reference, not a value
"const token = getToken()", // call
"password = ${DB_PASSWORD}", // template
`api_key = "changeme"`, // placeholder
"secret: TODO", //
"key gh***************xk", // masked, from `config`
"password = xxxxxxxxxxxxxxxxxxxxxxx", // masked
"// the token is never persisted in the repo", // prose
"apiKey := os.Getenv(\"MGSH_REMOTEKEY\")", // lookup
"token = SOME_CONSTANT_NAME", // constant
"secret = my.config.value", // dotted reference
"+++ b/token.go", // diff furniture
"password = short", // too short to be one
"Authorization: Basic <base64(owner:token)>", // documentation
"remote.hub.key = <personal-access-token>",
}
for _, ln := range lines {
if kind := matchSecret(ln); kind != "" {
t.Errorf("false positive (%s) on %q", kind, ln)
}
}
}
// TestScanDiffReportsFileAndLine: the report has to point at the right place,
// and must ignore removed lines — deleting a secret is the desired action.
func TestScanDiffReportsFileAndLine(t *testing.T) {
diff := `diff --git a/.env b/.env
new file mode 100644
--- /dev/null
+++ b/.env
@@ -0,0 +1,3 @@
+HOME=/tmp
+API_KEY="s3cr3tV4lu3W1thStuff"
+DEBUG=1
diff --git a/old.txt b/old.txt
--- a/old.txt
+++ b/old.txt
@@ -7,1 +7,0 @@
-password = hunter2hunter2hunter2
`
hits := scanDiff(diff)
if len(hits) != 1 {
t.Fatalf("expected exactly one hit, got %d: %+v", len(hits), hits)
}
h := hits[0]
if h.file != ".env" {
t.Errorf("file = %q, want .env", h.file)
}
if h.lineNo != 2 {
t.Errorf("lineNo = %d, want 2", h.lineNo)
}
if !strings.Contains(h.text, "API_KEY") {
t.Errorf("text = %q", h.text)
}
}
// TestScanDiffCountsLinesAcrossHunks keeps the line numbers honest when a file
// is edited in several places.
func TestScanDiffCountsLinesAcrossHunks(t *testing.T) {
diff := `+++ b/config.yml
@@ -1,0 +1,1 @@
+harmless: yes
@@ -40,0 +41,2 @@
+also fine
+aws_key = AKIAIOSFODNN7EXAMPLE
`
hits := scanDiff(diff)
if len(hits) != 1 || hits[0].lineNo != 42 {
t.Fatalf("hits = %+v, want one at line 42", hits)
}
}
// TestSecretsApprovedBlocksThePush drives the real thing: a staged .env, the
// scan, and the answer deciding whether push continues.
func TestSecretsApprovedBlocksThePush(t *testing.T) {
dir := t.TempDir()
mustGit(t, dir, "init", "-q")
mustGit(t, dir, "config", "user.name", "t")
mustGit(t, dir, "config", "user.email", "t@e")
mustGit(t, dir, "commit", "-q", "--allow-empty", "-m", "base")
old := cfg
defer func() { cfg = old }()
cfg = Config{}
// clean tree: no prompt, no interference
asked := fakeAnswers(t, false)
if !secretsApproved(dir) {
t.Fatal("a clean tree must not block the push")
}
if len(*asked) != 0 {
t.Fatalf("asked about a clean tree: %v", *asked)
}
if err := os.WriteFile(filepath.Join(dir, ".env"),
[]byte("API_KEY=\"s3cr3tV4lu3W1thStuff\"\n"), 0600); err != nil {
t.Fatal(err)
}
mustGit(t, dir, "add", "--all", ".")
declined := fakeAnswers(t, false)
if secretsApproved(dir) {
t.Error("a staged credential must stop the push when declined")
}
if len(*declined) == 0 {
t.Error("the user was never asked")
}
accepted := fakeAnswers(t, true)
if !secretsApproved(dir) {
t.Error("an explicit yes must let the push through")
}
if len(*accepted) == 0 {
t.Error("the user was never asked")
}
// and the escape hatch really switches it off
cfg.SecretScan = "off"
never := fakeAnswers(t, false)
if !secretsApproved(dir) {
t.Error("secretscan = off must not block")
}
if len(*never) != 0 {
t.Errorf("secretscan = off still asked: %v", *never)
}
}
// TestSecretScanDefaultsToOn: only a deliberate "off" disables it, so a typo
// leaves the safety net in place.
func TestSecretScanDefaultsToOn(t *testing.T) {
old := cfg
defer func() { cfg = old }()
for _, v := range []string{"", "true", "on", "yes", "wharrgarbl", "1"} {
cfg = Config{SecretScan: v}
if !secretScanEnabled() {
t.Errorf("secretscan = %q disabled the scan", v)
}
}
for _, v := range []string{"off", "0", "false", "no", " OFF "} {
cfg = Config{SecretScan: v}
if secretScanEnabled() {
t.Errorf("secretscan = %q did not disable the scan", v)
}
}
}
// TestOwnDocsDoNotTripTheScanner: mgsh's own README and example config are full
// of credential-shaped text; committing mgsh itself must stay quiet.
func TestOwnDocsDoNotTripTheScanner(t *testing.T) {
for _, f := range []string{"README.md", "mgshrc.example"} {
data, err := os.ReadFile(f)
if err != nil {
t.Fatal(err)
}
for i, ln := range strings.Split(string(data), "\n") {
if kind := matchSecret(ln); kind != "" {
t.Errorf("%s:%d would trip the scanner (%s): %q", f, i+1, kind, ln)
}
}
}
}
// TestAllowMarkerSuppressesOneLine: the per-line escape for something that only
// looks like a credential and is meant to stay.
func TestAllowMarkerSuppressesOneLine(t *testing.T) {
line := `API_KEY="s3cr3tV4lu3W1thStuff"`
if matchSecret(line) == "" {
t.Fatal("test line is not detected at all")
}
if kind := matchSecret(line + " # mgsh:allow — sample value"); kind != "" {
t.Errorf("mgsh:allow did not suppress the hit (%s)", kind)
}
if kind := matchSecret("-----BEGIN OPENSSH PRIVATE KEY----- mgsh:allow"); kind != "" {
t.Errorf("mgsh:allow did not suppress a certain pattern (%s)", kind)
}
}
+2 -2
View File
@@ -44,6 +44,7 @@ func showConfig() {
{"pushdefault", cfg.PushDefault},
{"editor", cfg.Editor},
{"mirror", cfg.Mirror},
{"secretscan", cfg.SecretScan},
{"remotes", cfg.RemoteNames},
}
@@ -146,8 +147,7 @@ func configKeys() []string {
keys := []string{
"base", "githost", "gitport", "gituser", "gitpath", "gitkey",
"gitname", "gitemail", "pushdefault", "editor",
"remoteurl", "remotekey", "remotetype", "remotevisibility",
"remotes", "mirror",
"remotes", "mirror", "secretscan",
}
sort.Strings(keys)
return keys
+11
View File
@@ -39,6 +39,17 @@ func truthy(s string) bool {
return false
}
// falsy reports whether a config string explicitly means "off". It is not the
// negation of truthy: for a setting that defaults to on, an unset value or a
// typo must leave it on, and only a deliberate "off" may switch it off.
func falsy(s string) bool {
switch strings.ToLower(strings.TrimSpace(s)) {
case "0", "false", "no", "off":
return true
}
return false
}
func fileExists(p string) bool {
fi, err := os.Stat(p)
return err == nil && !fi.IsDir()
+1 -1
View File
@@ -1 +1 @@
4.0.24
4.0.42