# Log streams

On Enterprise, a log stream copies your organisation’s records to your own Amazon S3 bucket, S3-compatible storage or webhook, in batches, in order and at least once, so you can keep them as long as you like.

## Plans and roles

Log streams are part of Enterprise, up to 10 per organisation. On Free and Standard, creating one answers `402`. Admins and owners create, test, pause and remove streams. After a downgrade, streams over the new limit are paused, not deleted.

## Amazon S3 and S3-compatible storage

1. Create a bucket and a writer
  Create a bucket, and an IAM user or role whose only permission is to write under your prefix (below). Create an access key for it.
2. Add the stream
  In **Log streams**, choose **Amazon S3** and enter the bucket, its region, an optional prefix, and the access key ID and secret access key. For Cloudflare R2, MinIO or other S3-compatible storage, also enter the endpoint, such as `https://<account id>.r2.cloudflarestorage.com` with region `auto`.
3. Send a test
  **Send test** writes one test record now, as its own object, without moving the stream forward.

Each batch is one gzip-compressed NDJSON object (`Content-Type: application/x-ndjson`, `Content-Encoding: gzip`), one record per line, at:

```
<prefix>/clitrail/<organisation id>/<yyyy>/<mm>/<dd>/<hh>/<batch>.ndjson.gz
```

The batch name is two 12-digit numbers joined by `-`; it grows from one batch to the next, so objects list in order, and stays the same when a batch is written again. A batch is written every 5 minutes, or sooner at 5,000 records or 16 MB. Requests are signed with AWS Signature Version 4. AWS buckets are addressed virtual-hosted style unless the bucket name has dots; custom endpoints use path style.

### IAM policy

Allow only `s3:PutObject`, only under your prefix. CLItrail never reads, lists or deletes objects:

```json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "CLItrailLogStream",
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::YOUR_BUCKET/YOUR_PREFIX/clitrail/*"
  }]
}
```

Without a prefix, the resource is `arn:aws:s3:::YOUR_BUCKET/clitrail/*`. If the bucket encrypts with your own KMS key, also allow `kms:GenerateDataKey` on that key. On Cloudflare R2, use an API token with Object Read & Write on that bucket only.

## Webhook

Enter an https URL on a public address. CLItrail generates a `whsec_` signing secret and shows it once; **Rotate secret** replaces it and shows the new one once. Every minute, or sooner at 1,000 records or 4 MB, it posts a batch:

```json
{"records": [
  {"schema": "clitrail.telemetry.v1", "id": "tl_mufbcblk_3f9a1c7e5b2d4a60", "time": "2026-09-24T09:12:03.512Z", "kind": "inbound", …},
  {"schema": "clitrail.telemetry.v1", "id": "tl_mufbcblo_8e0b6d2f4a1c9e37", …}
]}
```

Requests carry the [Standard Webhooks](https://www.standardwebhooks.com/) headers `webhook-id`, `webhook-timestamp` and `webhook-signature`, and `User-Agent: CLItrail-LogStreams/1`. Verify them exactly as for [webhook destinations](https://clitrail.com/docs/webhooks#verify-signatures). `webhook-id` names the batch and stays the same on every retry, so deduplicate on it. Test deliveries have IDs starting with `tltest_`.

## Delivery

- **In order, at least once.** Each stream keeps a cursor. A batch that fails keeps the cursor where it was and is sent again unchanged, with the same object key or `webhook-id`, so a retry overwrites or is deduplicated.
- **Backoff.** Retries wait 1, 2, 5, 10, 30, then 60 minutes, or longer when your endpoint sends `Retry-After` (up to an hour). After three failures in a row, an alert record is written.
- **From creation.** A new stream starts with the records written after it was created.
- **Bounded backlog.** A stream that cannot deliver keeps at most 1,000,000 records or 7 days. Past that, the oldest are skipped, and only with an alert record saying how many.
- **Health.** The stream shows its last success, last error, pending records and lag. Pause a stream to stop delivery; resuming continues from its cursor.
- **Tests.** Up to 10 tests an hour per organisation, across all its streams.

Credentials are write-only and sealed in the [separate secret store](https://clitrail.com/docs/secrets); error messages are scrubbed of them before they are shown.