Data format¶

A BigClust project is a directory containing an info file plus the data files it names. This page is the complete specification; if you would rather read a worked example first, create a local dataset builds one end to end.

/my_clustering/
    info                <- JSON: what is here and how to read it
    meta.parquet        <- per-neuron metadata                   (required)
    embeddings.parquet  <- 2D coordinates for the scatter plot   (optional)
    distances.parquet   <- pairwise distances                    (optional)
    features.parquet    <- high-dimensional features             (optional)

Data files may be Parquet (recommended), Apache Arrow Feather, CSV or JSON. Parquet is preferred throughout: compact, self-describing, columnar, and it round-trips dtypes and MultiIndex columns properly.

The info file¶

JSON, no extension. Only meta is validated as required — but a project with no embeddings has no coordinates to draw, so in practice you want both.

{
    "name": "my_clustering",
    "dataset": "hemibrain_v1.2",
    "description": "My clustering dataset",
    "version": "0.1",
    "date_created": "2024-06-01",
    "meta": {
        "file": "meta.parquet",
        "color": "color",
        "last_updated": "2024-06-01"
    },
    "embeddings": {
        "file": "embeddings.parquet"
    },
    "distances": {
        "type": "connectivity",
        "file": "distances.parquet",
        "metric": "cosine"
    },
    "features": {
        "type": "connectivity",
        "file": "connections.parquet"
    }
}

Top-level keys¶

meta¶

The per-observation table — one row per neuron.

"meta": {
    "file": "meta.parquet",
    "color": "color",
    "last_updated": "2024-06-01"
}

Columns¶

Any additional columns are loaded and become available for colouring, labelling, filtering, hover info and scope filters. Ship everything you have — extra columns cost almost nothing and each one is a way to interrogate the embedding.

Colour¶

meta.color accepts three forms:

Anything cmap understands is a valid colour — hex strings, named colours, [r, g, b] lists. The default is white.

Meta sources¶

Optional. Declares where each dataset's annotations can be read from, so the snapshot can be refreshed in-app. Never written to.

"meta": {
    "file": "meta.parquet",
    "sources": {
        "hemibrain": {
            "backend": "neuPrint",
            "config": { "dataset": "hemibrain:v1.2.1" },
            "columns": { "label": "type", "soma_side": "somaSide" },
            "last_updated": "2026-06-14"
        }
    }
}

id and dataset are never mapped or updated, and neither is any column whose name starts with _.

Several datasets sharing one setup can use a comma-separated key, or an explicit datasets list in the value:

"sources": {
    "brain_left,brain_right": {
        "backend": "neuPrint",
        "config": { "dataset": "..." },
        "columns": { "label": "type" }
    }
}

Each listed dataset is still matched by its own IDs. Saving from the dialog regroups identical setups back into one comma-keyed entry.

Refreshed values live in the session. Use Export → Project to persist them, or refresh the project on disk in the first place with bigclust2 update. See refreshing meta data.

embeddings¶

The 2D coordinates the scatter plot draws.

"embeddings": { "file": "embeddings.parquet" }

or, if the coordinates are columns of the meta table:

"embeddings": { "columns": ["x_coord", "y_coord"] }

float32 is plenty for 2D scatter coordinates.

Multiple embeddings¶

Make embeddings a list. Each entry may carry its own features and/or distances, which override the top-level ones for that entry.

"embeddings": [
    {
        "name": "connectivity (UMAP)",
        "file": "embeddings_conn.parquet",
        "features": { "file": "connections.parquet", "type": "connectivity" },
        "distances": { "file": "distances.parquet", "metric": "cosine" }
    },
    {
        "name": "morphology (NBLAST)",
        "file": "embeddings_morph.parquet",
        "distances": { "file": "nblast.parquet" }
    },
    {
        "name": "raw layout",
        "columns": ["x", "y"]
    }
]

Entries without a name get embedding 1, embedding 2, … The first entry is active on load.

Features and distances must always belong to an embedding. A project that declares features or distances with no embeddings at all has them ignored, with a warning in the log.

In the app, switch with the dropdown at the top of the Embeddings tab or press Space to cycle. See the data model for why this is useful.

distances¶

A square, symmetric matrix of pairwise distances.

"distances": {
    "type": "connectivity",
    "file": "distances.parquet",
    "metric": "cosine"
}

Shape must be (n_observations, n_observations), with index and column names being the IDs, in the same order as meta.

Store values as float32. A distance matrix grows with the square of your observation count and is normally the largest file in a project; float32 halves it with no practical loss.

Store the full matrix even if it is symmetric — BigClust expects square, not triangular.

KNN graph¶

For large projects, supply a k-nearest-neighbours graph instead of a full matrix. It is dramatically smaller and loads much faster.

"distances": {
    "type": "knn",
    "file": "knn.parquet",
    "metric": "cosine"
}

type may be prefixed with the source the neighbours came from — "nblast:knn" — for your own bookkeeping; it is still recognised as a KNN graph.

The file is a wide table, one row per observation, in the same order as meta. For k neighbours:

k is inferred from the number of nn_idx_* columns. Row indices rather than IDs keeps the graph unambiguous even when a project contains duplicate IDs. Neighbours that fall outside a filtered subset are dropped automatically; metric is informational.

What a KNN graph costs you. A red note appears in the affected tabs:

See the data model.

features¶

The high-dimensional description of each observation.

"features": {
    "type": "connectivity",
    "file": "connections.parquet"
}

Shape is (n_observations, n_features), with the index matching the meta id order.

Use MultiIndex columns to group features — upstream versus downstream connections, isomorphic versus dimorphic partners. Each top level becomes a checkbox under Feature Subset in the Embeddings tab and a filter in the Feature Comparison widget, which is how you ask "lay these neurons out by their inputs only". Build the DataFrame with a pandas.MultiIndex before writing; pyarrow preserves it. Read the file back and check df.columns is still a MultiIndex.

Feature matrices are usually wide and sparse. Store counts as the smallest dtype that fits.

neuroglancer¶

Configures the 3D viewer.

"neuroglancer": {
    "source": {
        "HbR": "precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.2/segmentation",
        "HbL": "precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.2/segmentation"
    },
    "neuropil_mesh": "https://github.com/navis-org/navis-flybrains/raw/refs/heads/main/flybrains/meshes/JRCFIB2018Fraw.ply",
    "color": {
        "HbR": [255, 0, 0],
        "HbL": [0, 0, 255]
    }
}

Note this is where gs:// and s3:// are fine — they are handed to cloud-volume, not to BigClust's own loader. The HTTP-only restriction applies to the project directory, not to Neuroglancer sources.

Source schemes¶

Meshes vs. skeletons¶

Neurons render as meshes where meshes are available and as skeletons (navis.TreeNeuron) otherwise. BigClust always asks a source for a mesh first and only falls back to its skeletons if the source has none, or if that particular segment has not been meshed. A skeleton is drawn as its neurites plus a soma marker, and behaves like a mesh everywhere else — colouring, selection, caching and transforms all apply unchanged.

Local skeletons¶

skeletons:// serves skeletons from a directory of SWC files on this machine:

"neuroglancer": {
    "source": "skeletons://skeletons/"
}

Each file must be named after the segment ID it contains — 12345.swc serves the neuron with id 12345. Both .swc and .swc.gz are read; files whose name is not a plain number are ignored, as they cannot be addressed by ID. The directory is indexed once, on the first neuron you load.

The path may be absolute, start with ~, or — as above — be relative to the project directory. Relative paths are only resolved for local projects.

Transforms¶

Bring datasets in different spaces into a common one when meshes are loaded.

"transforms": [
    {
        "apply_to": "HbL",
        "type": "landmarks",
        "file": "landmarks.csv",
        "source_cols": ["x1", "y1", "z1"],
        "target_cols": ["x2", "y2", "z2"]
    }
]

Multiple projects¶

A directory whose info file is a list of relative paths is a collection:

/projects/
    info
    /clustering_1/
        info
        ...
    /clustering_2/
        info
        ...

[
    "clustering_1",
    "clustering_2"
]

Point BigClust at /projects/ and you get a project picker.

Writing good Parquet¶

General¶

• Prefer Parquet over Feather for distribution. Feather is fine as local scratch; Parquet compresses better and is more portable across versions.
• Write with pyarrow — df.to_parquet(..., engine="pyarrow"). Avoid the fastparquet engine for matrices with non-string column names; it handles them inconsistently.
• Compress. snappy (the pandas default) is a good speed/size trade-off; zstd gives the smallest files for slightly slower writes. Both decompress fast.
• Keep row order consistent across every file. See the warning at the top of this page.

Per file¶

Verify before shipping¶

import pandas as pd

meta = pd.read_parquet("meta.parquet")
dist = pd.read_parquet("distances.parquet")
feat = pd.read_parquet("features.parquet")

# Shapes and dtypes
print(meta.shape, dist.shape, feat.shape)
print(dist.dtypes.unique())                    # expect float32

# Required columns present
assert {"id", "label", "dataset"} <= set(meta.columns)

# Row order consistent - the failure this catches is silent otherwise
assert list(dist.index) == list(meta["id"])
assert len(feat) == len(meta)

# Distances square and symmetric
assert dist.shape[0] == dist.shape[1]

# MultiIndex features survived the round trip
print(type(feat.columns), feat.columns[:3])

The row-order assertions are the important ones. Everything else fails loudly on load; that one does not fail at all.