Docs / Guide

Output files

All output files are written to the directory specified by -o (default: recore_output/).

Two guarantees hold for every table here. TSV outputs are written as single-line records: tabs, line breaks and null characters inside annotation-derived text are normalised to spaces, so a GFF product, note or alias cannot corrupt a downstream table. And when an output directory is reused, Recore clears the known optional reports before writing the new run, so a stale reference_*, rejected-evidence, NSC, synteny or corrected-GFF file from a previous run cannot be mistaken for a current result.

rescue_report.tsv#

Primary output. One row per rescued gene. Tab-separated.

Columns#

Fields that do not apply to a row (the fragment columns of a single-hit rescue, an annotation match where none overlapped) are written empty, and booleans are TRUE or FALSE.

Example#

gene_cluster	genome	rescue_type	identity	coverage	contig	start	end	strand	confidence	synteny	synteny_score	synteny_support_n	synteny_reference_contexts	synteny_neighbors	synteny_upstream_matches	synteny_downstream_matches	orf_integrity_signal	annotation_match	fragment_n	fragment_query_gap_bp	fragment_target_gap_bp	fragment_colinear	fragment_same_cds	fragment_note
group_2044	L1_AS00349	DROPPED	100.0	100.0	L1_AS00349	245301	246100	+	1.198	PASS	1.000	12	18	1	1	1	in_frame_span	Rv0210c						
group_681	L4_AS00146	MISSING_CDS	99.2	89.5	L4_AS00146	2195924	2197220	+	1.066	PASS	0.833	4	5	2	1	1	in_frame_span							missing_cds_no_substantial_gff_overlap
group_0187	A4_AS00248	SPLIT:group_0188	99.2	99.8	A4_AS00248	1203401	1204300	+	1.087	PASS	1.000	8	9	1	1	1	in_frame_span	Rv1099c						
group_3940	L1_AS00366	PSEUDOGENE:2stops	99.0	100.0	L1_AS00366	1783748	1784254	-	0.495	PASS	0.750	3	4	2	1	1	disrupted_orf	Rv2651c						
group_2101	L2_AS00401	ALIGNMENT_SPLIT:2frags:98%cov	99.6	98.4	L2_AS00401	552100	553020	+	0.882	PASS	1.000	6	6	1	1	1	fragmented_orf		2	9	9	TRUE	FALSE	alignment_split

Filtering suggestions#

# High-confidence rescues only
awk '$10 > 0.8' rescue_report.tsv

# Synteny-validated only
awk '$11 == "PASS"' rescue_report.tsv

# Non-pseudogene rescues
grep -v "PSEUDOGENE" rescue_report.tsv

# Same-locus full-length rescues with strong support
grep -E "DROPPED|MISSING_CDS|SPLIT" rescue_report.tsv | awk '$10 > 0.9 && $11 == "PASS"'

synteny_rejected_report.tsv#

Written only when --synteny is enabled and one or more candidate rescues have explicitly incompatible flanking context. The schema is identical to rescue_report.tsv, but these rows are audit records: they are not added to gene_presence_absence_recore.csv and are not written to corrected GFF output.

This file is useful for reviewing high-identity paralog/cross-family hits that look like sequence rescues but fail the local genomic-context check.

paralog_rejected_report.tsv#

Written only when one or more PARALOG candidates lack explicit synteny support. The schema is identical to rescue_report.tsv, but these rows are audit records: they are not added to gene_presence_absence_recore.csv and are not written to corrected GFF output.

This is the conservative guardrail for multi-copy families: sequence similarity alone is not allowed to rescue a missing gene when the annotation says the hit belongs to a different cluster.

orthology_evidence.tsv#

Written for every run. One row per accepted rescue plus every rejected synteny or paralog candidate.

publication_status takes one of these values:

Only the first four describe a corrected cell, so a value starting with corrected or publication always means the cell is present in gene_presence_absence_corrected.csv.

Use this table as the primary audit trail for why a candidate entered, or did not enter, the corrected matrix.

reference_genome_audit.tsv#

Written only when --reference-genome is provided. The value can be an exact matrix column name or an embedded ASxxxxx identifier.

This file is meant for reference-absence claims: keep new_to_reference for strictly absent clusters, remove reference_like_or_present, and manually review reference_like_ambiguous. The publication_confidence field uses clean, complex_paralog, event_derived, fragmented, pseudogene, unresolved, or not_reference_absent. complex_paralog is used when the exact cluster remains absent from the reference but discarded/subthreshold sequence evidence points to a partial or paralogous reference-like copy. event_derived is used when the exact cluster remains separate in the matrix but its representative sequence is best explained as a reference-gene allele or fragment. Accepted sequence-only evidence that fails the adaptive orthology-quality guard is kept as reference_like_ambiguous with publication_confidence=unresolved.

reference_cluster_events.tsv#

Written only when --reference-genome is provided and Recore finds reference-absent clusters with strong sequence evidence to a reference-present cluster. This file is audit-only: it never merges clusters and never changes the binary corrected matrix.

Rows in this file also appear in reference_genome_audit.tsv as EVENT_DERIVED_REFERENCE_CLUSTER with final_publication_status=flag_reference_event_derived_cluster and publication_confidence=event_derived.

evolutionary_events.tsv#

Written for every run. This file is audit-only: it does not merge clusters and does not change gene_presence_absence_recore.csv. When --tree is provided, Recore parses the rooted Newick tree, matches tips to matrix columns by exact name or embedded ASxxxxx identifier, and applies Fitch parsimony to the corrected binary state for each cluster.

Use this file as the top-level evolutionary interpretation table. The most important guardrail is that phylogenetic_pattern and molecular_mechanism are deliberately separate: a clade-associated presence can be an acquisition, a reference loss, a duplicated/paralogous copy, or an event-derived allele depending on the sequence, synteny, GFF, and reference evidence.

cluster_locus_context.tsv#

Written when GFF annotations are loaded. One row per observed locus context for each cluster.

cluster_locus_summary.tsv#

Written when GFF annotations are loaded. One row per cluster.

Clusters flagged as possible_multilocus_or_paralog are not automatically changed, but they should be reviewed before treating accessory-gene calls as biological.

cluster_synteny_blocks.tsv#

Written when GFF annotations are loaded. This is an audit-only synteny-block summary using up to five upstream and five downstream cluster neighbours. When a reference genome is supplied, each block also stores a representative gene-level neighbourhood and the best matching reference/H37Rv neighbourhood inferred from shared flanking clusters, even when the focal cluster itself is absent from the reference.

gene_presence_absence_recore.csv#

Drop-in replacement for Panaroo's gene_presence_absence.csv. Identical format and column order.

Changes from the original#

1. Rescued cells: Empty cells (gene absent) become RESCUED:contig:start-end where coordinates are 1-based.

   Before: ""
   After:  "RESCUED:L1_AS00349:245301-246100"

2. Matrix-policy filter: --matrix-policy controls which accepted rescues become binary presence. strict corrects only STRICT_PRESENT/ALIAS_CLUSTER; balanced also corrects full-length SEQUENCE_ONLY; sensitive also permits partial/disrupted accepted evidence. In all modes, the adaptive orthology-quality guard can keep weak accepted evidence out of the binary matrix.

3. NSC-merged cells (only if --nscluster): Cells where a merged cluster would have added presence become NSC_MERGED.

4. Metadata columns updated: No. isolates, No. sequences, and Avg sequences per isolate are recomputed from the final cells. Multicopy cells separated by ; count as multiple sequences but one isolate.

5. NSC clusters removed: Panaroo clusters that were merged by NSC into a representative cluster are dropped entirely (their rows are absent from the output).

Compatibility#

This file can be used directly as input to downstream tools that accept Panaroo's format:

• distree for phylogenetic distance analysis
• fstic for allele frequency-based FST calculations
• Custom R/Python scripts expecting Panaroo output.

gene_presence_absence_recore.Rtab#

Written for every run, in the same pass as gene_presence_absence_recore.csv, so the two agree on every cell. It is the corrected matrix as 1 and 0 in Panaroo's gene_presence_absence.Rtab layout: a Gene column, then one column per genome, tab-separated. A cell is 1 when the corrected CSV has anything in it (input presence, RESCUED:... or NSC_MERGED), and clusters merged away by NSC are left out, as in the CSV. This is the file pyseer's --pres, and most R code written for Panaroo, reads.

gene_presence_absence_recore_states.csv#

Written for every run. Same row/column layout as the corrected matrix, but each genome cell stores the biological interpretation rather than a binary presence call.

Possible states:

• PRESENT_INPUT: present in the original matrix.
• ABSENT: searched, and no evidence under current thresholds.
• NOT_SEARCHED_LOW_PREVALENCE: never searched, because the cluster is in fewer genomes than --min-prevalence.
• NOT_SEARCHED_NO_CLUSTER_SEQUENCE: never searched, because the cluster has no sequence in the pan-genome reference.
• NOT_SEARCHED_NO_FASTA: never searched, because the genome has no FASTA file.
• STRICT_PRESENT: accepted same-locus rescue.
• ALIAS_CLUSTER: accepted split/alias or NSC-compatible cluster evidence.
• SEQUENCE_ONLY: full-length sequence evidence without context/annotation support.
• PARTIAL_HOMOLOGY: accepted fragmented or alignment-split evidence.
• PSEUDOGENE: accepted disrupted sequence evidence.
• HOMOLOG_PARALOG: rejected or ambiguous paralog-like evidence.
• REJECTED_CONTEXT: rejected incompatible-context evidence.

Use this file when a binary matrix would overstate biology, especially for pseudogenes, partial genes, PE/PPE-like families, or publication figures that need to show uncertainty explicitly. A NOT_SEARCHED_* cell is absent because the input matrix says so; Recore did not test it. When a cell has two reasons, the cluster's is written, in the order of the gap lines in summary.txt.

cluster_relationships.tsv#

Written for every run. The file may contain only the header when no inter-cluster relationships are detected.

This table summarizes reproducible evidence that one Panaroo cluster is related to another. It is intended as an audit layer: Recore reports relationships and recommendations, but only nucleotide similarity clustering (--nscluster) changes the output matrix automatically.

Columns#

Interpretation#

• collapse_alias: near-complete, high-identity, synteny-supported evidence that the query is not an independent cluster.
• review_alias_without_full_synteny: high sequence support, but synteny is absent, mixed, or not tested.
• review_annotation_match: the rescued locus overlaps a feature already assigned to another cluster; inspect as a possible naming or boundary issue.
• review_paralog_keep_distinct_by_default: related sequence family member; keep distinct unless additional evidence supports collapse.

cluster_components.tsv / cluster_component_members.tsv#

Written for every run. These tables convert pairwise relationships into connected components, so split clusters, annotation-matched clusters, NSC members, and paralog-linked clusters can be reviewed as one unit.

cluster_components.tsv contains one row per non-singleton component:

cluster_component_members.tsv contains one row per member cluster:

These components are also reported in evolutionary_events.tsv so manuscript highlighting can deprioritize redundant aliases or paralog components.

genome_qc.tsv#

Written for every run. One row per genome with rescue and annotation burden metrics.

Flags and what they do to qc_status:

summary.txt#

Human-readable run summary. This one is from the tutorial run:

Recore Summary
==================================================
Clusters:              24
Genomes:               12
Total gaps:            50
Gaps passing filter:   39
Gaps searched:         39
Skipped (prevalence):  11
Rescued:               11
Matrix policy:         Balanced
Matrix-corrected:      8
  rescue cells:        8
Remaining gaps:        42

Core genome (strict):  14 → 17 (+3)
Soft-core (≥95%):      14 → 17 (+3)

Rescue types:
  DROPPED:       6
  MISSING_CDS:   2
  SPLIT:         0
  FRAGMENTED:    0
  ALIGNMENT_SPLIT: 0
  START_VARIANT: 0
  PARALOG:       0
  PSEUDOGENE:    3

Synteny validation:
  PASS:  11
  FAIL:  0

The gap lines add up. Every empty cell of the input matrix is in Total gaps, and each one is accounted for exactly once:

Matrix-corrected is the number of cells written into the corrected matrix as presence, and Remaining gaps is the number of cells still empty in it, both counted from the file as it is written. Without --nscluster, Total gaps - Matrix-corrected = Remaining gaps. With --nscluster a NSC merged cells line appears under Matrix-corrected, rows merged into a representative are no longer in the file, and the summary ends with a Nucleotide similarity clustering block.

Interpreting core genome changes#

• Core +3 in the tutorial: Rv0003, Rv0006 and Rv0012 were missing from a few genomes and are now found in all of them, core genes the matrix had excluded. Rv0009 stays out, because balanced keeps pseudogenes out of the binary matrix.
• With --nscluster the soft-core can shrink: clusters that looked soft-core on their own may be redundant copies of another cluster and are merged into it.

run_manifest.json#

Machine-readable run manifest written for every run. It records:

• tool and tool_version;
• schema_version (recore.outputs.v7);
• created_utc, the time the run finished writing outputs, and command_line, the exact invocation;
• input paths used for the run;
• key parameters, including matrix policy, prevalence threshold, synteny, NSC, reference genome, tree, and mobile-element settings;
• headline counts such as clusters, genomes, searched gaps, direct rescue corrections, NSC-merged corrections, rejected synteny/paralog candidates, and NSC groups.

Use this file when archiving outputs for a manuscript or comparing runs across parameter settings.

nscluster_report.tsv#

Written only if --nscluster is enabled and at least one merge was found.

Example#

ns_cluster	representative	n_members	members
NS_0	group_2891	3	group_2891;group_2892;group_2893
NS_1	group_0441	2	group_0441;group_0442

These clusters had nearly identical nucleotide sequences but were split by Panaroo due to amino acid-level differences (typically from upstream annotation artifacts).

corrected_gff/*.recore.gff3#

Written only when --write-corrected-gff is enabled.

For each genome with at least one rescue that corrected the matrix, Recore writes a GFF3 file under corrected_gff/; evidence kept out of the matrix (held_out_policy, held_out_quality) is not written. If an original GFF was supplied with --annotations, its records are preserved and Recore appends rescued CDS features, on the GFF's own seqid even when that seqid had to be mapped onto a differently named FASTA contig. If the original GFF contains a ##FASTA section, rescue features are inserted before ##FASTA and the FASTA tail is preserved after them. If no original GFF is available for a rescued genome, the file contains only the appended rescue features.

Appended rescue features use source Recore, type CDS, and include:

These GFFs are designed as auditable corrected annotations. If a rescue overlaps an existing feature, the original feature is preserved and the appended Recore feature records the rescue evidence explicitly.

rescued_genes.fna / rescued_genes.faa#

Written only when --write-rescued-fasta is enabled: the sequence of every cell the binary matrix gained, for functional annotation (eggNOG-mapper, InterProScan, BLAST). Record IDs are the feature IDs of corrected_gff/*.recore.gff3, and the description carries the genome, the cluster, the locus, the rescue type and the phase:

>recore_Rv0003_L1_AS00101_contig1_3333_4517 genome=L1_AS00101 gene_cluster=Rv0003 locus=L1_AS00101_contig1:3333-4517(+) rescue_type=DROPPED phase=0

rescued_genes.fna has the nucleotide span read in the gene's direction (reverse-complemented on the minus strand). rescued_genes.faa translates it from the phase the alignment fixes, with the bacterial code (NCBI table 11); a first codon that is the gene's own start reads M whichever start it is, and the final stop is left off. A rescue built from several alignments has no single reading frame, so it is in the .fna with phase=unknown and not in the .faa.

recore_report.html#

Written only when --write-html-report is enabled.

This is a self-contained browser report for run review and manuscript-facing QC. It summarizes:

• input matrix size, searchable gaps, accepted evidence, corrected matrix cells, rejected synteny/paralog evidence, and core/soft-core changes;
• orthology actions and confidence tiers from orthology_evidence.tsv;
• evolutionary priorities and dominant mechanisms from evolutionary_events.tsv;
• per-genome QC status from genome_qc.tsv;
• the top candidate clusters ranked by evolutionary_signal_score;
• direct links to the exact output files used to generate the summary.

When --tree is supplied and the matched tree/matrix is small enough for a responsive browser view, the report also embeds a phylogeny plus corrected matrix explorer. The explorer prunes the tree to matched matrix tips, keeps matrix rows in tree order, lets the user search/select clusters, recolors tree tips dynamically by the selected cluster state or evolutionary mechanism, and lets users click matrix columns to change the selected cluster. The matrix can also be recolored by evolutionary mechanism, with a legend explaining each observed process from evolutionary_events.tsv. MTBC-style genome names can be colored by lineage with the mycolorsTB Nature palette; in lineage mode, monophyletic branches are colored too. Clicking mechanism cards adds or removes mechanisms from the matrix-column filter, and selected-cluster details show the reference cell/gene or the reference-derived event source when available. Dense views can be zoomed with the row and column sliders in each pane header, with the mouse wheel over the tree or the column header, and in fullscreen. The viewer's own page documents it in full: see Run report. When cluster_synteny_blocks.tsv is present, the viewer can open a full-width synteny band under the matrix, drawing up to ten ranked unique five-neighbour blocks for the selected gene cluster. The rows are unique by ordered displayed gene/locus architecture, ignoring internal cluster gaps, length differences, and coordinate differences; each row stores one representative locus and all carrier genomes for that architecture. The band is hidden by default and opens from the layers button or the S key. Once visible, the seam above it is draggable and a scale slider expands or compresses the gene-arrow layout. The selected block marks its carrier genomes on the tree, the clickable H37Rv/reference row marks the reference tip, the focal cluster is highlighted, and clicking any gene arrow jumps to that cluster in the matrix. Gene arrows carry strand orientation and proportional length. Labels prefer reference locus IDs where available, and each gene tooltip includes cluster, locus, coordinates, strand, and length. Shared neighbouring clusters are connected only between adjacent synteny rows.

The embedded explorer is intentionally size-limited. Larger runs keep the static audit summary and report why the explorer was omitted, rather than writing a sluggish HTML file.

The HTML report is an audit dashboard, not a replacement for the TSV outputs. Use the TSV files for exact filtering, statistics, and downstream figures.

Coordinate System#

All coordinates in Recore output are 1-based (standard for bioinformatics formats like GFF3, VCF, GenBank):

• rescue_report.tsv: start column is 1-based
• gene_presence_absence_recore.csv: RESCUED:contig:start-end coordinates are 1-based

minimap2 internally uses 0-based half-open coordinates (PAF format). Recore converts these to 1-based for all output.

Log Output#

Recore writes structured log output to stderr. Use -v / --verbose for debug-level detail.

Standard output (INFO level)#

This is the tutorial run from Tutorial, with timestamps removed:

[INFO recore::parser] Parsed presence/absence: 24 clusters × 12 genomes (core: 14, soft-core 95%: 14)
[INFO recore::pipeline] Gaps: 50 total, 39 after prevalence filter (≥10%), 11 skipped
[INFO recore::pipeline] Core genome (before): 14 / 24 clusters
[INFO recore::pipeline] Loaded GFF annotations for 12 genomes
[INFO recore::search] Prepared search: 39 query sequences across 12 genomes, one minimap2 run per genome
[INFO recore::pipeline] Raw rescues before filtering: 11
[INFO recore::pipeline] Upgraded 3 sequence-only rescues to PSEUDOGENE:internal_stop
[INFO recore::pipeline] Validating synteny...
[INFO recore::pipeline] Synteny: 11 validated, 0 failed, 0 not assessed (no clustered flanking genes, or no annotated copy to compare with)
[INFO recore::pipeline] Final rescues: 11 (DROPPED: 6, MISSING_CDS: 2, SPLIT: 0, FRAGMENTED: 0, ALIGNMENT_SPLIT: 0, START_VARIANT: 0, PARALOG: 0, PSEUDOGENE: 3)
[INFO recore::pipeline] Core genome: 14 → 17 (+3)

Verbose output (DEBUG level, with -v)#

[DEBUG recore::rescue] Synteny skip: no flanking genes mapped to clusters for rescue cluster=3940 genome=75
[DEBUG recore::rescue] Upstream gene 'L1_AS00366_Rv2650c' not found in any Panaroo cluster
[DEBUG recore::pipeline] Top rescues (first 10):
[DEBUG recore::pipeline]   group_2044 in A1_AS00349 → Dropped (id=100.0%, cov=100.0%, conf=1.20, synteny=Some(true))

Common debug messages and their meaning#

Edit this page on GitHub