mcp-readitall/README.md

# mcp-readitall

A Go-based [Model Context Protocol](https://modelcontextprotocol.io) (MCP)
server that fetches any website — including modern SPAs that need a real
browser to render — and converts the page into clean Markdown. It also
supports per-host authenticated sessions so you can read content that lives
behind a login wall.

The server speaks MCP over **stdio** and is designed to be wired into any
MCP-aware client (Claude Desktop, Claude Code, Cursor, etc.).

## Features

- **Real browser rendering** via [chromedp] (a pure-Go Chrome DevTools
  Protocol client) + system Chromium — handles SPAs, JS hydration,
  lazy-loaded content, client-side routing. No Node.js bridge.
- **Markdown conversion** with the [html-to-markdown] plugin pipeline:
  tables, code blocks (with language detection), strikethrough, commonmark
  semantics, relative-URL resolution.
- **Per-host authenticated sessions** — the `login` tool opens a visible
  browser window for you to authenticate, then persists cookies +
  localStorage so future reads reuse the session automatically.
- **Session lifecycle tools** — `list_sessions` and `clear_session` for
  inspection and revocation.
- **Content extraction** — `extract` and the `selector` parameter on
  `read_url` let you pull just the part of a page you care about.
- **Cross-process safe** — session state is persisted to disk (0600) so the
  server can restart without losing auth.

[html-to-markdown]: https://github.com/JohannesKaufmann/html-to-markdown
[chromedp]: https://github.com/chromedp/chromedp

## Tools

| Tool | Purpose |
| --- | --- |
| `read_url` | Fetch a URL (with optional CSS selector), render JS, return Markdown. |
| `login` | Open a browser, let the operator log in, persist the session. |
| `list_sessions` | Show every host for which a session is saved. |
| `clear_session` | Delete a saved session. |
| `extract` | Read a URL and return only the part under a CSS selector. |

### `read_url` arguments

| Field | Type | Description |
| --- | --- | --- |
| `url` | string (required) | Absolute http(s) URL. |
| `selector` | string | Optional CSS selector to extract a region. |
| `wait_for` | string | Optional CSS selector to wait for (useful for SPAs that hydrate after `networkidle`). |
| `timeout_seconds` | int | Overall timeout. Default 30. |
| `max_chars` | int | Truncate the returned Markdown. Default ~200k. |

### `login` arguments

| Field | Type | Description |
| --- | --- | --- |
| `url` | string (required) | The URL to open (typically the login page). |
| `success_url` | string | Substring of the post-login URL (e.g. `/dashboard`). Polled every second. |
| `success_selector` | string | CSS selector that becomes visible once auth succeeds (e.g. an avatar menu). |
| `timeout_seconds` | int | How long to wait for the operator to complete login. Default 300. |

`success_url` **or** `success_selector` is required so the tool knows when
to stop waiting and save the session.

## Build & install

Requirements:

- Go 1.22+
- A Chromium-family browser on `$PATH` or installed at the standard
  location — `google-chrome`, `chromium`, `chromium-browser`, or Edge.
  macOS: `brew install --cask chromium` (or the standard Chrome install).
  Linux: `apt install chromium` / `dnf install chromium`.

```bash
# Build the server binary
go build -o bin/mcp-readitall ./cmd/readitall
```

> If the project is not under git you may need `-buildvcs=false`:
> `go build -buildvcs=false -o bin/mcp-readitall ./cmd/readitall`

chromedp will auto-discover Chromium via [`exec.LookPath`] and the usual
Chromium env vars (`CHROME_BIN`, `CHROMIUM_BIN`, etc.). If you have a
non-standard install, point at the binary directly:

```bash
CHROME_BIN=/Applications/Chromium.app/Contents/MacOS/Chromium ./bin/mcp-readitall
```

[`exec.LookPath`]: https://pkg.go.dev/os/exec#LookPath

## Wire into an MCP client

### Claude Desktop (`~/Library/Application Support/Claude/claude_desktop_config.json`)

```json
{
  "mcpServers": {
    "readitall": {
      "command": "/absolute/path/to/mcp-readitall/bin/mcp-readitall"
    }
  }
}
```

### Claude Code (`.mcp.json` in your project)

```json
{
  "mcpServers": {
    "readitall": {
      "command": "/absolute/path/to/mcp-readitall/bin/mcp-readitall"
    }
  }
}
```

Restart the client after editing the config.

## Usage flow

### 1. Read a public page

Ask your MCP client:

> "Read https://news.ycombinator.com and summarize the top 5 stories"

Under the hood the model calls `read_url`. The server launches a
headless Chromium, navigates, waits for the body to be ready (plus a
short grace period for hydration, or a `wait_for` selector if supplied),
converts the HTML to Markdown, and returns it.

### 2. Read an authenticated page

> "Log me into GitHub, then read my notifications"

The model first calls:

```json
{
  "name": "login",
  "arguments": {
    "url": "https://github.com/login",
    "success_url": "/",
    "timeout_seconds": 300
  }
}
```

A real browser window pops up on your machine. Log in normally. When the
URL changes to contain `/`, the tool saves the session and closes the
window. Subsequent calls to `read_url` for `github.com` will reuse the
saved cookies + localStorage automatically.

To invalidate a session:

> "Clear my session for github.com"

The model calls `clear_session`.

## Session storage

Saved sessions live in:

- `~/.mcp-readitall/sessions/<host>/state.json` by default
- Override with `READITALL_SESSIONS_DIR=/some/path`

State files are written with mode `0600` (owner read/write only) and
use Playwright's standard storage state shape (`{cookies:[…], origins:[{origin,
localStorage:[…]}]}`) — sessions captured by Playwright can be loaded by
us, and vice versa. Treat them as secrets — they grant access to your
authenticated sessions.

## Tests

```bash
go test -race ./...
```

The suite is hermetic — no browser is launched. It covers:

- Hostname normalization (good / bad / edge).
- Storage-state validation.
- Session-store CRUD and concurrency safety.
- HTML→Markdown conversion (headings, lists, tables, code, entities, URL
  resolution, security-sensitive stripping of `<script>` / `<style>` /
  comments).
- Tool helpers (URL host extraction, comma-separated selector parsing,
  output formatter).

## Limitations

- The `login` tool needs a **visible** browser window. Running it on a
  remote / headless host with no display (and no VNC / X-forwarding) will
  not work — Chrome exits immediately when it has no window to draw to.
- Requires a Chromium-family browser on the host. `chromedp` does not
  bundle one.
- The reader is single-tab per call. Reading multiple pages in parallel
  inside one tool call is not supported yet.
- We treat `networkidle` as "body is ready + 500 ms grace" (or whatever
  you pass in `wait_for`). Sites that keep firing background requests
  indefinitely may need a `wait_for` selector rather than rely on
  automatic settling.

## License

MIT.