Integration · Object storage

Drop a file in a bucket. Get your schema back.

Read the newest object under a prefix on a schedule, map it against your template, and write the mapped result back as a timestamped file. SigV4 against S3 and every S3-compatible store — R2, MinIO, Spaces, B2, Scaleway.

Get an API key Read the docs

Method: POST
Connector kind: 1

How it works

What actually happens on a sync run

A bucket is where most regulated data already lands: an export job writes a CSV every night and something has to pick it up. The S3 connector lists your prefix, takes the newest object modified since the last successful run, parses it by its own key extension, and hands the grid to the cascade. Point the same connector the other way and the mapped rows are serialized into one file and put back under a timestamped key. One credential, encrypted once, used for both directions.

1
List the prefix
A SigV4-signed ListObjectsV2 scoped to config.prefix, normalized into the same StoredObject[] shape all three providers return.
2
Filter by watermark
Objects whose modified is newer than the last successful sync survive; the newest of those is chosen. Nothing new is not_modified, not a failure.
3
Parse by key
The object’s own extension picks the parser — csv, tsv, json, xml, and the binary formats xlsx, xlsm, parquet and docx included.
4
Map and land
The grid becomes an upload with your workspace retention, then runs the cascade against the connector’s template.

What you get

Built for files that keep arriving

Any S3

AWS, and everything that speaks its dialect

The same connector reaches Cloudflare R2, MinIO, DigitalOcean Spaces, Backblaze B2 and Scaleway — set an endpoint and the signature is unchanged.

HowSigV4 is computed in lib/cloudAuth.ts; config.endpoint only swaps the host. Provider errors come back verbatim — an AccessDenied is the actionable thing — but never the signature.

Incremental

Only what landed since last time

A scheduled pull compares each object’s modified time against the last successful sync and takes the newest survivor. An empty result is “nothing new”, reported as success.

HowThe watermark is connector.last_synced_at, overridable per call with ?since=. A failed run does NOT bump it, so the next tick retries the same window rather than skipping it.

Write-back

The result lands as one timestamped object

A bucket takes objects, not rows, so a write serializes the whole accepted result set into a single file and puts it under a key that will not collide with yesterday’s.

HowThe default key is export-{date}-{time}.csv under your prefix; destination.format picks csv, json, xlsx or another emit format. batches is 1 — the object lands or it does not.

Secrets

The credential never travels in a request body

You reference a connector by id. Keys, tokens and service-account JSON are encrypted at rest and read only by the code that makes the call.

HownormalizeSecretField() folds every provider spelling — private_key, token, secret_access_key, account_key, service_account_json — into one auth_value field, which is KEK-envelope-encrypted before the row is written. A GET masks it to a 4-character hint. If encryption fails the field is dropped rather than stored in plaintext.

Configuration

The connector record, field by field

Anything you send under a provider’s own spelling for the secret is normalized into auth_value and envelope-encrypted before the row is written — one encrypted field, one place to get right.

Key	Required	What it is
bucket	Required	Bucket name. Its presence is what selects the credentialled path over the legacy presigned-URL form.
access_key_id	Required	Access key id. Validated at save time.
auth_valuesecret	Required	The secret access key. `secret_access_key` is accepted as an alias and folded into this field before storage.
region	Optional	Signing region. Defaults to us-east-1.
endpoint	Optional	Custom host for an S3-compatible store — R2, MinIO, Spaces, B2, Scaleway. Omit for AWS.
session_tokensecret	Optional	STS session token, for temporary credentials.
prefix	Optional	Key prefix that scopes both the read and the write, e.g. incoming/.
format	Optional	Override the parser instead of inferring it from the key.
url	Optional	Legacy alternative: a presigned https URL. Readable, but a presigned URL cannot be written to — a write returns `config_invalid`.

In code

A nightly export, mapped on arrival.

Save the bucket once from the dashboard, then pull it from anywhere with a bearer key. The credential stays in the connector record; the request carries only its id.

POST/v1/connectorsSave the connector. The secret is encrypted before it reaches Postgres.session
POST/v1/connectors/{id}/testMake a real call and report what was actually proven.session
POST/v1/connectors/{id}/syncPull now. Accepts ?since=<iso> to override the watermark.bearer
POST/v1/gatewayAny input in, this destination populated, a delivery report out.bearer
POST/v1/connectors/{id}/rotate-secretReplace the credential in place; the old one becomes unrecoverable.session

The response’s layer_distribution is the cost story: a recurring export settles in the deterministic layers, so the metered layer is never called at all.
A presigned-URL connector still works and still reads — it just cannot be a destination, because there is nothing to sign a PUT with.
Objects are capped at 25 MiB and every call times out at 60 seconds. A larger export should be split by the job that writes it.

POST /v1/connectors

{
  "kind": "s3",
  "name": "Nightly lab export",
  "template_id": "lab_results",
  "config": {
    "bucket": "acme-clinical-exports",
    "prefix": "labs/incoming/",
    "region": "eu-central-1",
    "access_key_id": "AKIA…",
    "secret_access_key": "…"
  }
}

curl

curl -X POST https://api.adaptivmapr.com/v1/connectors/con_9f2c…/sync \
  -H "Authorization: Bearer $MAPR_API_KEY"

response

{
  "ok": true,
  "status": "parsed",
  "upload_id": "upl_7d13…",
  "row_count": 1840,
  "accepted": 1839,
  "format": "csv",
  "incremental": "incremental",
  "layer_distribution": { "statistics": 9, "heuristic": 21, "fuzzy": 4, "semantic": 0, "llm": 0 }
}

→ newest object since the watermark · 34 headers resolved · 0 reached an LLM

Limits & failure modes

What it refuses, and what it tells you

Code	When	What to do
`400 ssrf_blocked`	The configured host resolves to a private, link-local or loopback address.	Every outbound request is DNS-resolved and checked before it is made, on the scheduled path and the on-demand path alike. The reason is returned with the code.
`400 config_invalid`	A credential or identifier the provider needs is missing or malformed.	Refused at save time rather than at 3am during a scheduled run. The message names the exact key.
`413 fetch_too_large`	The chosen object is larger than 25 MiB.	The parser’s own ceiling — an object is an upload by another route and the same memory limits apply.
`400 config_invalid (write)`	A destination write is attempted against a presigned-URL connector.	The message says it plainly: a presigned URL can be read but not written to. Store real credentials to write.
`502 fetch_failed`	The provider refused the list or the get.	The provider’s own message is surfaced verbatim; the watermark is not advanced, so the next tick retries the same window.
`400 parse_failed`	The object is not readable as the format its key claims.	Set `config.format` to override extension sniffing, or fix the producer.

Incremental sync

Real. The listing is filtered to objects modified after the watermark and the newest survivor is taken, so a bucket that gains one file a night is read once a night. Nothing new returns status: "not_modified" with row_count: 0 — a success, not an error, and the watermark still advances only on success.

PHI & residency

A bucket sync runs under the workspace’s routing policy, not around it. PHI routing is a separate axis from the data mode: it sends X-PHI and X-Region to phi-cloud so a regulated run lands on an in-region, BAA-eligible model, it costs +20% on the whole charge, and it is locked until the workspace accepts the BAA in Settings → Security & Data. An explicit PHI ask without an acceptance is 403 agreement_required, never a silent downgrade. A standard run keeps the workspace’s region pin — the region decides where compute may run, and the sandbox refuses a region-less run.

What it costs

Billed on the same prepaid wallet

Moving bytes is not a line item. A sync that pulls a file and a destination write that lands the rows are both part of one map, and the map is what the wallet sees. There is no free tier, no seats and no contract — top up from $10, a balance shared across the phi-cloud suite.

Charge	Rate	Notes
Every map	$0.001	A flat per-map fee — a few tokens — charged even when the run was fully deterministic or hit the layout cache and used no AI at all.
AI, only when it ran	at cost × 2	Layer-5 cleanup, any-to-any convert and structural reshape bill the phi-cloud tokens actually consumed. Bring your own model key and it is × 0.5.
PHI / enterprise routing	+20%	Multiplies the whole charge, flat fee included — and only when the run genuinely got that routing. Locked until the workspace accepts the BAA in-app.

Questions

Before you wire it up

Does this work with Cloudflare R2, MinIO or Backblaze B2?

Yes. The s3 kind is SigV4 over an S3-compatible API, so setting config.endpoint to your store’s host is the whole difference. AWS S3, Cloudflare R2, MinIO, DigitalOcean Spaces, Backblaze B2 and Scaleway all work through the same connector; only the host and region change.

Where is my secret access key stored?

In the connector record, envelope-encrypted with a key-encryption key before the row reaches Postgres, and held service-role-only. It is never accepted in an API request body — you reference the connector by id — and a GET masks it to a four-character hint. POST /v1/connectors/{id}/rotate-secret replaces it in place and records rotations_count and last_rotated_at; there is deliberately no two-key window, so a rotation makes the old key unrecoverable.

What happens if two files land between syncs?

The newest one is taken. The connector reads one object per run by design — the alternative is an unbounded fan-out inside a single Worker invocation. If your producer writes several files per window, either run the sync more often, or trigger POST /v1/connectors/{id}/sync per file from the job that writes it.

Can AdaptivMapr write results back into a different bucket?

Yes. Sources and destinations are separate connector records, so a read connector on one bucket and a destination connector on another is the normal shape. POST /v1/gateway takes an input and a destination.connector_id in one call and returns a delivery report — including the object_key the file landed under — instead of the data.

Is the object contents sent to an AI model?

Only if the deterministic layers cannot resolve a header and the metered layer is configured. Schema-only mode is the default posture and ships headers plus at most three sample rows clamped to 80 characters each; full-data is gated on the PHI entitlement. Layers 4 and 5 are config-gated and fail soft to off.

Google Cloud Storage →Azure Blob Storage →Connectors & webhooks →

Verified against lib/objectStore.ts · lib/connectorSyncRunner.ts · lib/destinations.ts · app/api/v1/connectors/route.ts

Amazon S3 is a trademark of Amazon.com, Inc. Cloudflare R2, MinIO, DigitalOcean Spaces, Backblaze B2 and Scaleway are trademarks of their respective owners. Named here to describe interoperability only — no affiliation, endorsement or partnership is claimed.

Point it at Amazon S3. Get your schema back.

Start with a $10 prepaid wallet. Every map draws a few tokens; in schema-only mode only headers and up to three sample rows, clamped to 80 characters a cell, ever leave you.

Get an API key All integrations