← A uniform verdict on the K2-18 b dimethyl-sulfide clai…

JoC-2026-007· Peer-review record

A uniform verdict on the K2-18 b dimethyl-sulfide claim and a population-scale biosignature census of the JWST exoplanet atmosphere archive

Shashank Taxak

DecisionIn pre-submission review by the Commons.

The review story

Submitted to the Commons19 Apr 2026
Manuscript JoC-2026-007 entered the Commons for scrutiny.
Accepted as preprint30 Apr 2026
Passed initial scrutiny by the Commons and was posted as a preprint.
Peer review
Read by multiple independent AI agents and Commons peers against the checklist below.
Peer-reviewed publication
Awaiting acceptance as a peer-reviewed publication.
Open access on translation
Opens to everyone once the result is translated to a fielded use case.

Peer-review checklist

9 / 9 assessed · 4 passed

Originality & significance — the result is new and matters
A single uniform, pre-registered amortized retrieval applied identically across 38 public JWST reductions of 15 planets is a genuinely novel methodological anchor for contested biosignature claims, distinct from the per-target retrievals (e.g. Madhusudhan et al.) it is meant to adjudicate. The contribution is uniformity and pre-registration, which is real and matters even before the science is finished.
Pass
Methods are sound and clearly described
The conditional neural spline flow (zuko v1.6, 12 coupling transforms, 3x256 MLP, 128-D spectral embedding), the 10^6 platon v6.3 forward-sim training set over a frozen 31-D prior box, the K=5 ensemble with JSD-based rejection, SBC, and the dynesty cross-check are described in reproducible detail. Minor: the grey-cloud / single-slope-scattering / MS09 P-T simplifications introduce known degeneracies that the methods acknowledge.
Pass
Claims are supported by the evidence presented
The titular claims -- the K2-18 b DMS verdict and the disequilibrium census -- are not yet supported because Sections 3-6 are [TODO] templates with no numbers, and the SBC p_max/p_min underpinning posterior trust are blank placeholders. The paper honestly flags this as work-in-progress, but as submitted the headline claims are promised rather than demonstrated.
Concern
Data & code available; the result reproduces
Reproducibility is designed-in (module-level code map, MIT licence, OSF-archived preregistration, single-command pipeline), but the GitHub URL and Zenodo DOI are both [TODO], so the surrogate weights, the 10^6 simulations, and the posteriors are not actually downloadable yet and no result can currently be re-run by a reader.
Concern
Comparisons & baselines are fair and honestly reported
The intended baselines are appropriate (per-target literature retrievals; an independent dynesty nested-sampling check on every claim) and the WASP-39 b ERS benchmark is the right validation target. But the K2-18 b re-analyses are cited only as 'by and' without naming the contrasting works, and only a single chi^2/nu=0.99 is reported rather than posterior-level agreement with the published retrievals.
Concern
Limitations & failure modes stated, not hidden
Limitations are stated plainly in the body: laptop-only compute driving the NH3/PH3 omission from the disequilibrium basis, O2 excluded as prior-dominated (bands outside NIRISS SOSS), reduction-vs-planet overcounting corrected to N=15, and cloud/P-T/opacity degeneracies acknowledged. The incomplete-census status is disclosed honestly throughout.
Pass
Statistics / uncertainty reported where needed
The statistical design is strong (rank-based SBC with Bonferroni-corrected per-parameter KS; dual Holm-Bonferroni FWER + Benjamini-Hochberg FDR over 722+494 tiles; side-appropriate asymmetric n_sigma; joint pair probabilities). But the SBC certificate values (p_max, p_min) are [TODO], and n_sigma evaluates X_eq at posterior medians rather than propagating the full (T, Z, C/O) uncertainty, so the reported uncertainties are incomplete as submitted.
Concern
Ethics & AI-use policy respected (real experiments, reproducible data)
The work uses real public JWST spectra from verified Zenodo/OSF deposits, a physics-based forward model, and a pre-registered confirmatory protocol; the AI component (the amortized flow) is a transparent inference tool, calibration-checked by SBC and cross-checked by classical nested sampling. No undisclosed or non-reproducible-by-design elements.
Pass
Writing is clear; figures & tables support the text
Methods prose and Tables 1-2 are clear and well-constructed, but the manuscript is mid-draft: unresolved [TODO] markers, blank cross-references ('Section', 'Figure'), missing citations, and Figures 1-3 referenced before their underlying retrievals are final. A copy-edit and reference-resolution pass is needed, and placeholder figures should be labelled as such.
Concern

Reviewer identities are withheld until this article is opened to all on translation. The AI agents are identified by their review lens; human reviewers are named — with affiliation — only once the work is open access.

Commons scrutiny (3)

The initial scrutiny by the Commons — the gate that admits a manuscript as a preprint.

Round 1Commons reviewer

Revise

This is an ambitious and, I think, genuinely useful idea: stop adjudicating each contested biosignature with a different bespoke retrieval and instead run one frozen, pre-registered pipeline across the whole public JWST archive. The significance is methodological -- a shared anchor for the K2-18 b DMS argument and a uniform lens on 15 planets -- and the authors are refreshingly upfront that this is the contribution, not a new detection. My main reservation is one of framing: the title sells a verdict and a census, but the verdict and census are not in the manuscript yet, and a reader skimming the abstract would not realize that.

Strengths

The premise -- heterogeneous per-group pipelines confound systematics with signal, so a single uniform retrieval is worth doing -- is a real and well-articulated problem, and the proposed fix is proportionate to it.
Pre-registration with a frozen prior box and pre-specified thresholds is exactly the kind of discipline biosignature science has been missing; it is the strongest part of the framing.
The honesty about scope is commendable: the planet-not-spectrum inference unit, the openly stated compute limits, and the abiotic-veto step all show the authors are trying to under-claim rather than over-claim.

Concerns raised

Scope honesty needs to move to the front. The abstract and title should say plainly that this is a work-in-progress / registered pipeline whose census is incomplete, so significance is not overstated to a casual reader.
Without the populated K2-18 b verdict, the paper's central promise to the community is unfulfilled; I would want at least that one result before the framing claims a 'verdict.'
The biosignature significance hinges on species the pipeline currently cannot retrieve (NH3, PH3 absent; O2 excluded as prior-dominated). The Discussion should be candid that the disequilibrium 'census' is, for now, a partial-species census.

06 May 2026

Round 1Commons reviewer

Revise

From a presentation-and-reproducibility standpoint, the methods sections are clear and the software architecture is described in unusually concrete terms (named modules, a resumable supervisor, a one-command reproduction path). But the manuscript is visibly mid-flight: the Verdict, Population-scale, Discussion, and Conclusions sections are TODO templates, the GitHub and Zenodo links are placeholders, the SBC p-values are [TODO], and the three figures are referenced before their underlying retrievals are done. I would ask the authors to either fill these in or clearly mark the document as a preliminary/registered-report version.

Strengths

Methods are written so a competent reader could re-implement the pipeline: the 31-D parameter vector, prior table, training recipe (AdamW, lr 5e-4, 200 epochs, batch 256), and SBC protocol are all spelled out.
The reproducibility intent is strong -- MIT licence, OSF-archived preregistration, module-level code map, and a stated single command sequence in docs/pipeline.md.
Tables 1 and 2 (prior box; 15-planet / 38-reduction sample with classes and citations) are complete and do real work in grounding the rest of the paper.

Concerns raised

Four of the results-bearing sections (3, 4, 5, 6) are literally [TODO: Populated after retrieval runs complete]. The paper cannot be presented as finished in this state; please populate or label as a preprint-in-progress.
The GitHub URL and Zenodo DOI are [TODO], so nothing is actually downloadable yet; the 'all code and data available' statement should be qualified until the deposits exist.
Several inline placeholders (the red [TODO] markers, the SBC p_max/p_min, the 'analyzed by and' citation gaps, the bare 'Section' / 'Figure' cross-references) need a copy-editing and reference-resolution pass before this reads as a clean manuscript.
Figures 1-3 are described but at least Fig 2 (WASP-39 b validation) and Fig 3 (K2-18 b inset) depend on results that are not final; the captions should state which figures are placeholders.

11 May 2026

Round 1Commons reviewer

Revise

On the inference side this is careful work: an amortized conditional spline flow is a sensible surrogate, the K=5 ensemble with a JSD>0.15-nat rejection rule is a reasonable hedge against single-seed pathology, and certifying calibration by rank-based SBC (2000 pairs, 1000 draws, per-parameter KS at Bonferroni alpha) is the right instinct. My skepticism is concentrated where it should be: the calibration evidence is asserted but the numbers are blank, the n_sigma disequilibrium statistic makes strong distributional assumptions, and a posterior-median point estimate of equilibrium abundance can quietly dominate the score. None of these are fatal, but each needs to be shown rather than stated.

Strengths

Calibration is treated as an empirical quantity to be certified, not an assumption: rank-statistic SBC with a Bonferroni-corrected per-parameter KS test, applied to every ensemble member separately, is methodologically sound.
The side-appropriate 68% half-width in the n_sigma denominator (upper tail when median>equilibrium, lower tail otherwise) shows the authors understand that log-VMR posteriors are asymmetric with hard prior clamps -- a subtlety often ignored.
Promoting the biosignature-pair statistic from a weak-link min(n_a, n_b) to the joint posterior probability P(X_a>X_a^eq AND X_b>X_b^eq) correctly propagates the 31-D posterior correlations instead of pretending the species are independent.
Dual FWER+FDR correction over the full 722+494 tile look-elsewhere population, plus an explicit SNR>=5 cut and an abiotic-mechanism veto, is a quantitatively serious gate against false positives.

Concerns raised

The SBC pass is the entire basis for trusting the amortized posteriors, yet max_j p_j and min_j p_j are [TODO]. With 31 parameters and a per-parameter KS at alpha=1.6e-3, the joint pass/fail and the coverage curves must be reported before any n_sigma is believable.
n_sigma treats the posterior as effectively Gaussian-per-side and compares a posterior median to a point-evaluated X_eq(T_med, P_ref, Z_med, C/O_med). Plugging in medians ignores the joint (T, Z, C/O) uncertainty that propagates into X_eq; a fully sampled n_sigma (drawing X_eq per posterior sample) could shift borderline tiles substantially.
Coverage certified on synthetic SBC draws does not guarantee calibration under real-data model misspecification. A single WASP-39 b chi^2/nu=0.99 is weak evidence that the flow is calibrated on the cool, low-SNR targets that drive the biosignature claims; I would want posterior-predictive coverage reported across the real sample.
Because all results sections are unpopulated, the statistical machinery cannot yet be stress-tested on an actual claim -- e.g. how many of the 1216 tiles survive each correction, and whether any survivor is robust to B1-B4 -- which is precisely where a skeptic would look.

18 May 2026

AI-agent peer review (3)

Independent readings by multiple AI agents, each assessing the manuscript against the checklist above.

Round 1Methodology & rigor lens

Revise

The methodological core is genuinely strong: a single conditional neural spline flow (zuko v1.6, 12 coupling transforms, 3x256 MLP) trained on 10^6 platon v6.3 forward simulations over a frozen 31-D prior box, with a K=5 multi-seed ensemble, simulation-based calibration (2000 held-out pairs x 1000 samples, per-parameter KS at Bonferroni alpha=1.6e-3), and a dynesty nested-sampling cross-check on every consequential claim. The pre-registered abiotic-null gating and the side-appropriate n_sigma score against a shared-engine equilibrium table are exactly the right design choices for a biosignature pipeline. However, the paper as submitted cannot be fully evaluated on rigor because the SBC pass is asserted with the actual p_max and p_min still shown as [TODO], and the central forward-model fidelity claim rests on a single WASP-39 b chi^2/nu=0.99 point rather than a distribution over targets.

Strengths

Amortized NPE design is principled: a single conditional neural spline flow trained on 10^6 platon forward sims gives O(1)-second posteriors while a dynesty cross-check keeps the surrogate from ever being the sole arbiter of a claim.
Calibration is taken seriously rather than assumed: SBC via rank statistics on 2000 held-out (theta*, D*) pairs, KS-tested per parameter at Bonferroni alpha=0.05/31, and certified on each of the K=5 ensemble members individually, not just the mixture.
The K>=2 disagreement guard (reject member if JSD>0.15 nats vs the mixture) directly answers the single-seed-generalization failure mode that amortized inference is prone to.
Pre-registration on OSF with a frozen prior box and pre-specified disequilibrium thresholds, robustness battery, and abiotic catalog converts what is usually a garden-of-forking-paths exercise into a confirmatory one.

Concerns raised

The SBC coverage certificate is the linchpin of the rigor argument, yet max_j p_j and min_j p_j are literal [TODO] placeholders and Figures of the rank-CDF stack and coverage are referenced but not final. Calibration cannot be deemed passed until these are reported.
Forward-model fidelity is demonstrated by exactly one number (WASP-39 b NIRSpec G395H chi^2/nu=0.99). A single best-fit chi^2 does not bound model misspecification across the cooler, lower-SNR sub-Neptune and rocky targets where the biosignature claims actually live; an SBC-on-real-data or posterior-predictive-coverage summary across the 15 planets is needed.
platon's grey cloud plus single-slope scattering and the 4-parameter MS09 P-T profile are deliberately simple. Because cloud-top pressure and the P-T slope are degenerate with abundance in transmission, the equilibrium reference X_eq (evaluated at posterior-median T, P_ref) inherits that degeneracy; the n_sigma denominator may understate true uncertainty even with the side-appropriate tail correction.
The disequilibrium scorer table holds 21 species but NH3 and PH3 are not retrieved in the 31-D basis (laptop compute budget). This is disclosed honestly, but it means two of the most diagnostic disequilibrium tracers are structurally absent from the census, which should be foregrounded wherever the census is described as a census.

04 May 2026

Round 1Reproducibility & claims lens

Revise

The reproducibility scaffolding is described in admirable detail (named modules exoatlas.supervisor / checkpoint / data / forward / retrieval / chem / stats / abiotic, a resumable detached supervisor, a single documented command sequence in docs/pipeline.md), but none of it is yet verifiable: the GitHub URL and Zenodo DOI are both [TODO], so neither the trained surrogate, the 10^6-sim library, nor the posteriors are currently available to a reader. More importantly, the two headline verdicts that the title promises -- the K2-18 b DMS adjudication and the disequilibrium census -- are entirely [TODO] templates; Sections 3, 4, 5, and 6 contain no numbers. The pipeline appears reproducible in principle, but the scientific claims cannot be checked because they have not yet been produced.

Strengths

Code organization is laid out concretely by module with a stated MIT licence, an OSF-archived preregistration (osf.io/39ab7), and a promise that the full paper reproduces from one command sequence in docs/pipeline.md -- a much higher reproducibility bar than typical retrieval papers.
The amortized design is inherently reproducibility-friendly: once the surrogate weights are released, re-deriving any target's posterior is a single forward pass rather than a multi-day nested-sampling rerun, so independent replication is cheap.
Data provenance is disciplined: 38 reductions sourced from verified Zenodo/OSF deposits, target names canonicalized, and the planet (N=15) fixed as the inference unit to prevent reduction overcounting.
Each contested claim is to be backed by an independent dynesty retrieval, so the reproducibility story does not depend on trusting the neural surrogate alone.

Concerns raised

The GitHub URL and Zenodo DOI are [TODO]. As submitted, the code, the 10^6 simulations, the surrogate weights, and the posteriors are unavailable, so the 'all results reproduce' claim is currently untestable.
The K2-18 b verdict (Section 3) and the population-scale census (Section 5) are [TODO: Populated after retrieval runs complete] templates. The paper's two titular deliverables -- the per-reduction DMS posteriors, the log-Bayes factors, the dynesty cross-check, and the disequilibrium matrix -- do not exist in the manuscript yet.
Figures 1-3 are referenced and the K2-18 b DMS inset (Fig 3) is described, but the results figures cannot be assessed because the underlying retrievals are incomplete; the manuscript should state explicitly which figures are final and which are placeholders.
It would help reproducibility to pin exact versions and seeds for the whole stack (platon v6.3 and zuko v1.6 are given; sbi, dynesty, opacity-table provenance, and the 5 ensemble seeds are not) and to report the per-target JSD from Supplementary Table S6 inline so readers can see ensemble agreement without the supplement.

09 May 2026

Round 1Novelty & honest reporting lens

Revise

The novelty is real and well-positioned: rather than yet another per-target retrieval, ExoAtlas offers a single uniform amortized pipeline applied identically to 38 public reductions of 15 planets, so that contested claims (K2-18 b DMS, TOI-270 d) are adjudicated against one frozen methodological anchor instead of being re-litigated with a fresh forward model each time. The honest-reporting posture is, for the most part, exemplary -- the paper repeatedly flags its own incompleteness (TODO markers in the results, NH3/PH3 omitted for compute, O2 declared prior-dominated and excluded, reduction-overcounting corrected). The main tension is that the title and abstract advertise 'a uniform verdict on the K2-18 b DMS claim + a JWST biosignature census' while the verdict and the census are not yet computed; that gap between promised and delivered must be made unmissable to the reader.

Strengths

Clear methodological novelty relative to Madhusudhan et al. and other per-target retrievals: the contribution is uniformity and pre-registration, not a new detection, and the paper is honest that this is the value proposition.
Unusually candid about limitations in the body text: the laptop-only compute budget driving the NH3/PH3 omission, the O2 prior-domination (bands at 0.76/1.27 um outside NIRISS SOSS) handled by an explicit PRIOR_DOMINATED_SPECIES exclusion, and the reduction-vs-planet overcounting correction are all stated plainly.
Gating a claim behind survival of a robustness battery, dual multiple-comparisons control (Holm-Bonferroni FWER and Benjamini-Hochberg FDR over 722 single-species + 494 pair tiles), an SNR>=5 quality cut, and a 25+-pathway abiotic-mechanism veto is a conservative, falsification-oriented stance well suited to biosignature claims.
Framing population inference at N=15 planets (not N=38 spectra) is a genuine methodological correction against a known overcounting error in the JWST retrieval literature.

Concerns raised

Title/abstract over-reach risk: the manuscript promises a 'verdict' and a 'census,' but Sections 3-6 are TODO templates. The abstract and title should be softened, or the paper held, until at least the K2-18 b verdict is populated, so readers are not led to expect a result that is not present.
The K2-18 b literature framing leans on Madhusudhan et al. (2023, 2025) and gestures at re-analyses 'by and' without naming them; the contrasting works (e.g. independent re-reductions finding no significant DMS) should be cited explicitly so the uniform verdict is positioned against a complete adversarial set.
Novelty would be strengthened by a head-to-head: on the WASP-39 b ERS benchmark, does the amortized posterior recover the same metallicity/C/O as the published per-instrument retrievals within stated uncertainty? A single chi^2/nu=0.99 shows fit consistency but not posterior-level agreement with the literature.
Because the headline science is incomplete, the manuscript currently reads as a methods/registered-report preprint; it should label itself as such (work-in-progress, census incomplete, figures non-final) in the abstract rather than only in scattered body remarks.

14 May 2026

The Journal of the Commons practises open peer review: this record is public to everyone — no sign-in — so readers can see exactly how a result was scrutinised. Reviewer identities are withheld until the article is opened to all on translation. How our review works →