This resulted in differing files (no singletons) – asking chatGPT, they said that sortmeRNA doesn’t change the filtering parameters for rRNA vs non_rRNA, it just doesn’t pull out paired vs singletons.

So in essence, I can use the “old” files and the “new” files together to assemble a transcriptome using trinity.

creating Trinity environment in conda: conda create -n trinity -c conda-forge -c bioconda trinity

moved all the _non_rRNA and _rRNA files up, and then identified the _out_paired from the first round and renamed those to non_rRNA and moved those too

now identify samples to use for transcriptome assembly (i selected a subset across genotypes and time points, and then looked for ones that had fwd and rev sortmerna files > 1 GB but less than 2 GB).

look for samples from sortmerna with < 1 GB – rerun those (aka the one at 17 k)

samples to rerun (all at 17kb): - Pstr_Jul2023_006 - Pstr_Nov2023_180 - Pstr_Nov2023_181 - Pstr_Nov2023_190 - Pstr_Nov2023_191 - Pstr_Nov2023_195 - Pstr_Nov2023_196

Jan 9, 2026: I reran sortmeRNA on these and it worked, so now they are all ready to go.

This is what ended up working to install and activate trinity on pegasus (although some warnings and errors came up that may cause problems later):

conda update trinity
#to get mamba to work on pegasus:
module load mambaforge/1.5.8
source /share/apps/mambaforge/install/etc/profile.d/conda.sh
source /share/apps/mambaforge/install/etc/profile.d/mamba.sh
mamba install trinity
mamba update trinity
conda activate /nethome/and128/anaconda3/envs/trinity
which Trinity
conda env create --name trinity --file /nethome/and128/anaconda3/envs/trinity/environment.yml
conda env export > environment.yml
conda env create --name trinity --file environment.yml

I ran this, and I keep running into errors:

command="Trinity --seqType fq --left pstr_fwd.fq.gz --right pstr_rev.fq.gz --CPU 10 --max_memory 100G --min_kmer_cov 2 --output /scratch/projects/and_transcriptomics/reciprocaltransplant/raw_seq_files/trinity/output"
bsub -P and_transcriptomics -q bigmem -n 10 -R "rusage[mem=10000]" -W 120:00 -J trin_pstr -e trin_pstr.err -o trin_pstr.out eval ${command}

#!/bin/bash
#BSUB -P and_transcriptomics
#BSUB -q bigmem
#BSUB -n 10
#BSUB -R "rusage[mem=10000]"
#BSUB -W 120:00
#BSUB -J trin_pstr
#BSUB -e trin_pstr.err
#BSUB -o trin_pstr.out

# from pegasus docs
module load miniforge3/24.3.0-0

# Activate Trinity environment
conda activate /nethome/and128/anaconda3/envs/trinity

# Run Trinity
Trinity --seqType fq \
  --left /scratch/projects/and_transcriptomics/reciprocaltransplant/sortmerna/trinity/pstr_fwd.fq.gz \
  --right /scratch/projects/and_transcriptomics/reciprocaltransplant/sortmerna/trinity/pstr_rev.fq.gz \
  --CPU 10 \
  --max_memory 100G \
  --min_kmer_cov 2 \
  --output /scratch/projects/and_transcriptomics/reciprocaltransplant/raw_seq_files/trinity/output

The most recent errors are: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory This Perl not built to support threads Compilation failed in require at /nethome/and128/anaconda3/envs/trinity/bin/Trinity line 5. BEGIN failed–compilation aborted at /nethome/and128/anaconda3/envs/trinity/bin/Trinity line 5.

So i am trying this: conda install -c bioconda -c conda-forge trinity samtools>=1.10 perl-threaded

It still is having issues. I wonder if it is because I have another conda environment with BioPerl.

Jan 23 2026

I was able to run Trinity in the trinity conda environment, and got the job submitted using this directly in the command line:

command="Trinity --seqType fq --left /scratch/projects/and_transcriptomics/reciprocaltransplant/raw_seq_files/trinity/pstr_fwd.fq.gz --right /scratch/projects/and_transcriptomics/reciprocaltransplant/raw_seq_files/trinity/pstr_rev.fq.gz --CPU 10 --max_memory 100G --min_kmer_cov 2 --output /scratch/projects/and_transcriptomics/reciprocaltransplant/raw_seq_files/trinity/trinity"
bsub -P and_transcriptomics -q bigmem -n 10 -R "rusage[mem=10000]" -W 120:00 -J trin_pstr -e trin_pstr.err -o /scratch/projects/and_transcriptomics/reciprocaltransplant/raw_seq_files/trinity/output eval ${command}

However, it quit after a bit and gave me 2 output files, which basically had the error message:

Argument "error" isn't numeric in numeric ne (!=) at /nethome/and128/anaconda3/envs/trinity/bin/Trinity line 3812. Error, need samtools installed that is at least as new as version 1.3 at /nethome/and128/anaconda3/envs/trinity/bin/Trinity line 3813.

Now i’m trying to install samtools again and I’m running into the same issue as before.

Jan 28 2026

samtools: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory

Here is the complete list of what I’m working with data-wise:

#some variables are nested and need to be controlled for in the model, but can't be fixed effects (model becomes too complicated)

# fixed effects for this test: Species, Region 
# random effects: Source Nursery, Site, STRATA, Monitoring Interval (repeated measures)
# I can't really test genotype because there is very few replicates across sites and regions  

# step-wise add fixed effects and then look at top_models and see which fixed effects are important

# Species and region are important

glmm_global <- glmer(Experienced_Predation_num ~ Species + Region + (1|SourceNursery) + (1|Site) + (1|STRATA) + (1|MI_CORRECTED), family = "binomial", data = outplant_monitoring_days_survival_glm)

summary(glmm_global)
# looking at the variance of random effects, MI_Corrected contributes the overwhelming majority
# source nursery contributes very little to the variance

# fit candidate subsets of predictors
options(na.action = "na.fail")  # required for dredge

model_set <- MuMIn::dredge(glmm_global, rank = "AICc", REML = FALSE)

# top models
top_models <- model_set %>% as_tibble() %>% arrange(AICc) %>% slice(1:8)
# best model with lowest AICc is the model that contains all fixed effects (global model) AICc = 12731.33   

model_no_species <- update(glmm_global, . ~ . - Species)
r2_full <- MuMIn::r.squaredGLMM(glmm_global)[, "R2m"]
r2_no_species <- MuMIn::r.squaredGLMM(model_no_species)[, "R2m"]
species_contrib <- r2_full - r2_no_species
# R2 with both species and region is 0.233
# R2 with just region is 0.223
# Species thus contributes only 0.01 to the explained variance


# residual diagnostics (DHARMA)
sim_res <- simulateResiduals(glmm_global, n = 1000)
plot(sim_res)
testUniformity(sim_res) # fits, not significant
testDispersion(sim_res) # fits, not significant
testZeroInflation(sim_res) # fits, not significant

# make summary table 

# fixed effects
fixed_tbl <- tidy(glmm_global, effects = "fixed", conf.int = TRUE)

# random effects
random_tbl <- tidy(glmm_global, effects = "ran_pars", conf.int = TRUE)

# combine
full_tbl <- bind_rows(
  fixed_tbl %>% mutate(effect_type = "fixed"),
  random_tbl %>% mutate(effect_type = "random")
)

# Perform pairwise comparisons to look at the impact of treatments within MCAV
pairwise_emmeans <- emmeans(glmm_global, pairwise ~ Species*Region, adjust = "tukey")

Anova(glmm_global, type = "III")
# significant effect of Species and Region

# View the pairwise comparisons results
print(pairwise_emmeans$contrasts)

Using SortMeRNA for removing rRNA

• Looking online, there seems to be some thoughts that removing rRNA reads is not required when Poly-A selection is performed during sequencing, as preferentially selecting polyA tails automatically creates a bias towards mRNA (see this article and this article).
• However, some suggest that removing rRNA would save computation time and lead to a cleaner assembly (see here).
• This tutorial/paper is a good resource - it says that if GC content is abnormal, it could be a sign of high rRNA content (GC content of over 50% is typical for rRNA) and then SortMeRNA should be used. They also said “if there is any doubt, this step should be performed.”

Based on the multiQC report of my trimmed reads, it looks like the average GC content is not normally distributed and is higher for some samples. To be safe, I’ll follow Brad’s code and run SortMeRNA on it.

#!/usr/bin/env bash

# Define project directories and paths
and="/scratch/projects/and_transcriptomics"
project="and_transcriptomics"
projdir="${and}/reciprocaltransplant"
readsdir="${projdir}/raw_seq_files/trimmed_cutadapt"
logdir="${projdir}/logs/sortmeRNA"

cd "${readsdir}"

# Loop through R1 files
for r1 in *_R1.trimmed.fastq.gz; do
  # Extract sample base name (e.g., "Pstr-Dec2022-202")
  base="${r1%_R1.trimmed.fastq.gz}"

  # Submit the job directly
  bsub <<EOF
#!/usr/bin/env bash
#BSUB -P ${project}
#BSUB -J ${base}_sortmeRNA
#BSUB -e ${logdir}/${base}_sortmeRNA.err
#BSUB -o ${logdir}/${base}_sortmeRNA.out
#BSUB -q bigmem
#BSUB -n 10

cd "${readsdir}"

${and}/programs/sortmerna-4.3.6-Linux/bin/sortmerna \\
  --ref ${and}/programs/sortmerna-4.3.6-Linux/database/smr_v4.3_default_db.fasta \\
  --reads ${readsdir}/${base}_R1.trimmed.fastq.gz \\
  --reads ${readsdir}/${base}_R2.trimmed.fastq.gz \\
  --aligned ${base}_rRNA \\
  --other ${base}_non_rRNA \\
  --fastx \\
  --out2 \\
  --sout \\
  --log \\
  --workdir ${projdir}/raw_seq_files/sortmerna/${base}
EOF
done

#!/usr/bin/env bash

# Define project directories and paths
and="/scratch/projects/and_transcriptomics"
project="and_transcriptomics"
projdir="${and}/reciprocaltransplant"
readsdir="${projdir}/raw_seq_files/trimmed_cutadapt"
logdir="${projdir}/logs/sortmeRNA"

cd "${readsdir}"

# Loop through R1 files
for r1 in *_R1.trimmed.fastq.gz; do
  # Extract sample base name (e.g., "Pstr-Dec2022-202")
  base="${r1%_R1.trimmed.fastq.gz}"

# Write the job script
    cat <<EOF > "${projdir}/scripts/sortmeRNA/${base}_sortmeRNA.job"

#!/usr/bin/env bash
#BSUB -P ${project}
#BSUB -J ${base}_sortmeRNA
#BSUB -e ${logdir}/${base}_sortmeRNA.err
#BSUB -o ${logdir}/${base}_sortmeRsNA.out
#BSUB -q bigmem
#BSUB -n 10
#BSUB -W 48:00
#BSUB -R "rusage[mem=6000]"

cd "${readsdir}"

${and}/programs/sortmerna-4.3.6-Linux/bin/sortmerna --ref ${and}/programs/sortmerna-4.3.6-Linux/database/smr_v4.3_default_db.fasta \
--reads ${readsdir}/${base}_R1.trimmed.fastq.gz \
--reads ${readsdir}/${base}_R2.trimmed.fastq.gz \
--aligned --other --fastx --blast --out2 --sout \
--workdir ${projdir}/raw_seq_files/sortmerna/${base}

EOF

    # Submit the job
    bsub < "${projdir}/scripts/sortmerna/${r1}_sortmerna.job"
done

for f in *.job; do
    bsub < "$f"
done

#!/usr/bin/env bash

# Define project directories and paths
and="/scratch/projects/and_transcriptomics"
project="and_transcriptomics"
projdir="${and}/reciprocaltransplant"
readsdir="${projdir}/raw_seq_files/trimmed_cutadapt"
logdir="${projdir}/logs/sortmeRNA"

cd "${readsdir}"

# Loop through R1 files
for r1 in *_R1.trimmed.fastq.gz; do
  # Extract sample base name (e.g., "Pstr-Dec2022-202")
  base="${r1%_R1.trimmed.fastq.gz}"

# Write the job script
    cat <<EOF > "${projdir}/scripts/sortmeRNA/${base}_sortmeRNA.job"

#!/usr/bin/env bash
#BSUB -P ${project}
#BSUB -J ${base}_sortmeRNA
#BSUB -e ${logdir}/${base}_sortmeRNA.err
#BSUB -o ${logdir}/${base}_sortmeRsNA.out
#BSUB -q normal
#BSUB -n 8
#BSUB -W 48:00
#BSUB -R "rusage[mem=6000]"


# Clean up old SortMeRNA outputs for this sample
rm -rf ${projdir}/raw_seq_files/sortmerna/${base}/kvdb

cd "${readsdir}"

${and}/programs/sortmerna-4.3.6-Linux/bin/sortmerna --ref ${and}/programs/sortmerna-4.3.6-Linux/database/smr_v4.3_default_db.fasta \
--reads ${readsdir}/${base}_R1.trimmed.fastq.gz \
--reads ${readsdir}/${base}_R2.trimmed.fastq.gz \
--aligned --other --fastx --blast --out2 --sout \
--workdir ${projdir}/raw_seq_files/sortmerna/${base}

EOF

    # Submit the job
    bsub < "${projdir}/scripts/sortmerna/${r1}_sortmerna.job"
done

[write_a_read:215] ESC[0;31mERRORESC[0m: Failed deflating readstring:
[write_a_read:215] ESC[0;31mERRORESC[0m: zlib status: -1
[append:328] ESC[0;31mERRORESC[0m: Failed deflating readstring:
[append:328] ESC[0;31mERRORESC[0m: zlib status: -1

#!/usr/bin/env bash

# Define directories
and="/scratch/projects/and_transcriptomics"
project="and_transcriptomics"
projdir="${and}/reciprocaltransplant"
readsdir="${projdir}/raw_seq_files/trimmed_cutadapt"
logdir="${projdir}/logs/sortmeRNA"
sortdir="${projdir}/raw_seq_files/sortmerna"
scriptdir="${projdir}/scripts/sortmerna"

# Make sure scriptdir and logdir exist
mkdir -p "${scriptdir}" "${logdir}"

# Loop through R1 trimmed reads
cd "${readsdir}" || exit

for r1 in *_R1.trimmed.fastq.gz; do
    base="${r1%_R1.trimmed.fastq.gz}"

    # Check if the SortMeRNA out folder exists and has files
    out_folder="${sortdir}/${base}/out"
    if [ -d "$out_folder" ] && [ "$(ls -A "$out_folder" 2>/dev/null)" ]; then
        echo "Skipping ${base} (output already exists)"
        continue
    fi

    # Write the LSF job script
    cat <<EOF > "${scriptdir}/${base}_sortmeRNA.job"
#!/usr/bin/env bash
#BSUB -P ${project}
#BSUB -J ${base}_sortmeRNA
#BSUB -e ${logdir}/${base}_sortmeRNA.err
#BSUB -o ${logdir}/${base}_sortmeRNA.out
#BSUB -q normal
#BSUB -n 8
#BSUB -W 48:00
#BSUB -R "rusage[mem=6000]"


# Clean up old SortMeRNA outputs for this sample
rm -rf ${projdir}/raw_seq_files/sortmerna/${base}/kvdb
cd "${readsdir}"

${and}/programs/sortmerna-4.3.6-Linux/bin/sortmerna \
  --ref ${and}/programs/sortmerna-4.3.6-Linux/database/smr_v4.3_default_db.fasta \
  --reads ${readsdir}/${base}_R1.trimmed.fastq.gz \
  --reads ${readsdir}/${base}_R2.trimmed.fastq.gz \
  --aligned ${sortdir}/${base}/${base}_rRNA \
  --other ${sortdir}/${base}/${base}_non_rRNA \
  --fastx --blast --out2 --sout \
  --workdir ${sortdir}/${base}
EOF

    # Submit the job
    bsub < "${scriptdir}/${base}_sortmeRNA.job"
done

  sortmerna \\
        ${'--ref '+fastas.join(' --ref ')} \\
        $reads_input \\
        --threads $task.cpus \\
        --workdir . \\
        --aligned rRNA_reads \\
        --fastx \\
        --other non_rRNA_reads \\
        $paired_cmd \\
        $out2_cmd \\
        $args

 """
    sortmerna \\
        ${'--ref '+fastas.join(' --ref ')} \\
        $refs_input \\
        $reads_input \\
        --threads $task.cpus \\
        --workdir . \\
        $reads_args \\
        $paired_cmd \\
        $out2_cmd \\
        $args

#!/usr/bin/env bash

# Define directories
and="/scratch/projects/and_transcriptomics"
project="and_transcriptomics"
projdir="${and}/reciprocaltransplant"
readsdir="${projdir}/raw_seq_files/trimmed_cutadapt"
logdir="${projdir}/logs/sortmeRNA"
sortdir="${projdir}/raw_seq_files/sortmerna"
scriptdir="${projdir}/scripts/sortmerna"

# Loop through R1 trimmed reads
cd "${readsdir}" || exit

for r1 in *_R1.trimmed.fastq.gz; do
    base="${r1%_R1.trimmed.fastq.gz}"

    # Check if the SortMeRNA out folder exists and has files
    out_folder="${sortdir}/${base}/out"
    if [ -d "$out_folder" ] && [ "$(ls -A "$out_folder" 2>/dev/null)" ]; then
        echo "Skipping ${base} (output already exists)"
        continue
    fi

    # Write the LSF job script
    cat <<EOF > "${scriptdir}/${base}_sortmeRNA.job"
#!/usr/bin/env bash
#BSUB -P ${project}
#BSUB -J ${base}_sortmeRNA
#BSUB -e ${logdir}/${base}_sortmeRNA.err
#BSUB -o ${logdir}/${base}_sortmeRNA.out
#BSUB -q bigmem
#BSUB -n 10
#BSUB -W 48:00

cd "${readsdir}"

${and}/programs/sortmerna-4.3.6-Linux/bin/sortmerna \
  --ref ${and}/programs/sortmerna-4.3.6-Linux/database/smr_v4.3_default_db.fasta \
  --reads ${readsdir}/${base}_R1.trimmed.fastq.gz \
  --reads ${readsdir}/${base}_R2.trimmed.fastq.gz \
  --aligned ${sortdir}/${base}_rRNA \
  --other ${sortdir}/${base}_non_rRNA \
  --fastx --out2 \
  --workdir ${sortdir}
EOF

    # Submit the job
    bsub < "${scriptdir}/${base}_sortmeRNA.job"
done

I want to go through his steps and follow his code since he constructed a de novo transcriptome for P. strigosa.

The sequencing he performed was “Poly-A capture/enrichment methodology by Novogene (Beijing, China) on the 158 Illumina NovaSeq PE 150bp. The methodology follows a standard workflow of fragmentation, reverse 159 transcription, cDNA synthesis, end-repair and A-tailing.”

3’ RNA-Seq doesn’t necessarily equate to Poly-A selection/capture/enrichment. 3’ RNA-Seq can be done with no selection, meaning that it would just sequence the 3’ends of RNA molecules but isn’t targeting molecules with polyA tails. Poly-A selection/capture/enrichment is performed to enrich mRNA molecules over other types or RNA (typically rRNA) that dominate total RNA content.

The question is, did my reciprocal transplant reads get any sort of selection? Looking back through my notes, I don’t see anything about Poly-A selection.

However, I can’t find a consensus on how many samples from the study to use. I have 192 sequence files - a forward and reverse read for each sample, or which there is 96 of them.

Should I be using all of them? Won’t that be time/resource intensive? Is that even necessary?

At first I thought they were referring to genotypic variation in general, which I did report p-values for because genotype was a factor in the PERMANOVA. But, I think their concern is specifically for within a treatment, the samples of a given genotype are far apart on the PCoA.

So then I asked ChatGPT if there was a test to do that, and of course it had an answer - use “PERMDISP” to assess “multivariate homogeneity of group dispersions.” In the vegan package, there is a function called “betadisper” and you can select the grouping variable you want to assess. It’s kind of like the Levene’s test for multivariate analyses.

So I ran this for genotype for Acer host gene expression first:

First run the PCoA function to make sure it’s making the same PCoA as the manuscript and also to make sure the variables are all loaded.

load("Rdata_files/host/initial_fullddsdesigncountsVsdcounts.RData")
ad.pcoa=pcoa(dist(t(assay(Vsd)),method="manhattan")/1000)
scores=ad.pcoa$vectors
ad.pcoa$values # the % variation explained by each axis is the Relative_eig column
conditions=design

scale_color_manual(values = c("grey", "#FF3333", "#00CCCC"))
#code by group
pdf("PCoA_Treatment_Genotype.pdf")
plot(scores[,1], scores[,2], pch=c(15,17,25)[as.numeric(as.factor(conditions$Genotype))], col=c("#00CCCC", "grey","#FF3333")[as.numeric(as.factor(conditions$Treatment))], xlab="PC1: 20%", ylab="PC2: 11%", main="Treatment")
ordispider(scores[,1:2], conditions$Treatment,col=c("#00CCCC", "grey", "#FF3333"))
legend("topright", legend=c("Control", "Initial", "Variable"), fill = c("#00CCCC", "grey", "#FF3333"), bty="n")
legend("topleft", legend=c("BC-8b", "MB-B", "SI-C"), pch=c(15,17,25), bty="n")

#analysis of variance 
ad=adonis2(t(assay(Vsd))~Genotype+Treatment,data=conditions,method="manhattan", permutations=1e6)
ad
as.data.frame(ad) %>% 
  write_csv("PERMANOVA_Acer.csv")

#analysis of dispersion (within-group variation)

# Test for differences in dispersion by genotype
dist_mat <- (dist(t(assay(Vsd)),method="manhattan")/1000)
dispersion_genotype <- betadisper(dist_mat, group = conditions$Genotype)
anova(dispersion_genotype)
permutest(dispersion_genotype)

# You can also visualize
plot(dispersion_genotype)

TukeyHSD(dispersion_genotype)

# Test for differences in dispersion by genotype within a treatment
conditions$Geno_Treatment <- paste(conditions$Genotype, conditions$Treatment, sep="_")
dispersion_geno_treatment <- betadisper(dist_mat, group = conditions$Geno_Treatment)
anova(dispersion_geno_treatment)
permutest(dispersion_geno_treatment)

# You can also visualize
plot(dispersion_geno_treatment)

TukeyHSD(dispersion_geno_treatment)

When I just run genotype and don’t separate out treatments, it is not significant.

But then when I run the Tukey, while it is still not significant, you can see that is comparing the different groups - the different genotypes. So it isn’t actually assessing within-genotype differences, it’s comparing the dispersion between genotypes.

That’s not what I want. And I guess that’s also the point of the Levene’s test, is that you’re comparing the homogeneity of variation between groups - is the dispersion of one sample group similar to the dispersion of the other?

What the reviewer is asking is, within a sample group, why are the samples so different? Is that statistically significant?

So then I ask ChatGPT again, and this time it tells me to use “anosim” for within-group structure, or substructure, within a grouping variable (in this case, treatment).

But I don’t think that’s what I want either? Because that sounds like then it is testing whether treatment is significant within a given genotype.

The last option that ChatGPT gives me is to do a custom permutation test to evaluate whether a genotype is particularly dispersed, compared to a null expectation. I am wary of trying that because the code it provided me creates a random gene set based on the genes in my data set, and then compares dispersion of my genotypes to the random set. I could do that I guess, but 1) it’s time consuming and 2) I don’t think that will actually prove anything.

Essentially, there is going to be inherent variation between samples. I don’t know how to statistially show that, but that’s the point of having biological replicates so that you get a mean across all the samples. I think if someone was to even sample the same coral at the same time point, you would still get some sort of difference between samples on a PCoA. Obviously you would hope that it wouldn’t be more different than the treatment effect, but there will still be some difference. The samples won’t be right on top of each other.

I think I will attach the PCA graph as another visualization, which shows much less spread and may be what the reviewer is more used to seeing.

To do this, I deleted my .gitignore file on the old computer which had all the files saved locally, and then pushed everything to GitHub and downloaded it all to my new computer. But then I was having trouble removing those files listed in my .gitignore file again.

The code I ended up running (which worked) in the terminal was:

nano .gitignore
git ls-files -i -c --exclude-from=.gitignore -z | xargs -0 git rm --cached
git status
git commit -m "Stop tracking files listed in .gitignore"
git push

Things that I tried that didn’t work: downloading some sort of Box CLI and trying to remotely access it via Pegasus.

curl -L https://downloads.boxcdn.net/boxcli/latest/BoxcliInstaller.run | bash\n
brew install rclone\n
nano ~/.bash_profile
source ~/.bash_profile
ls
brew install rclone\n
rclone config
rclone lsd
rclone lsd Box:/
less ~/.config/rclone/rclone.conf
scp ~/.config/rclone/rclone.conf and128@pegasus.ccs.miami.edu:/scratch/projects/and_transcriptomics/ #tried moving the configuration file from my computer to pegasus

# on Pegasus
module load rclone #rclone is already installed, but it doesn't have Box available
module list
rclone -V
rclone config
mv rclone.conf ~/.config/rclone/
rclone lsd box:
touch ~/.config/rclone/rclone.conf

There is a section about remote transferring on the Pegasus website.

It also still says you can use FileZilla, but when I try going to the FileZilla website on the “UMiamiWireless” wifi on campus, the page is blocked. I need to be on this wifi to access Pegasus, so I don’t know a workaround for that.

(wait can’t I just download FileZilla locally when on a different wifi and then try it)…

Regardless, I think using FileZilla would still require locally downloaded files from Box.

rsync -ah --progress --partial . "/Volumes/2TB/Ch3_RT"

rsync -ah --progress --partial -e ssh "/Volumes/2TB/Ch3_RT" and128@pegasus.ccs.miami.edu:/scratch/projects/and_transcriptomics/reciprocaltransplant/raw_seq_files

file Pstr-Dec2022-164_S46_L001_R1_001.fastq.gz

I got these errors for some of the sequence files when I tried to upload them to NCBI:

And when I emailed NCBI, they said this:

Michael Studivan also ran into this issue when he was uploading files to NCBI for a project. He said what solved it was finding the original files and uploading them directly.

I tried that already from downloading them from the Basespace website to my local drive, then uploading them again. That didn’t help. But what did seem to do something was if i gunzipped the file, then changed the metadata table so the file name was .fastq instead of .fastq.gz, and then uploaded the file. However, now it looks like there is another file that doesn’t work anymore.

What’s annoying is it doesn’t seem to show me the full list of samples. So I really can’t tell if it’s all the Pcli samples, or just the ones listed there.

I’m going to try to download all the Pcli files directly from Basespace using Michael’s command line code (here).

I think Michael’s code is outdated, so I found the BaseSpace CLI tutorial on Illumina here.

# if you have not previously, download BaseSpaceCLI
wget "https://api.bintray.com/content/basespace/BaseSpaceCLI-EarlyAccess-BIN/latest/\$latest/amd64-linux/bs?bt_package=latest" -O $HOME/bin/bs # this didn't work
chmod +x ~/bin/bs

#download via homebrew
brew tap basespace/basespace && brew install bs-cli

export PATH="/opt/homebrew/bin:$PATH"

echo '#!/bin/bash' > downloadReads.sh
echo 'bs download project --concurrency=high -q -n JA23031 -o . | grep "Pcli" ' >> downloadReads.sh
# -n is the BaseSpace project name and -o is the output directory

chmod +x downloadReads.sh

bs --version
# BaseSpaceCLI 1.6.1 -- built on 2024-07-23 at 09:33

bs auth
# has you go to the illumina website and sign in

./downloadReads.sh

The grep Pcli part didn’t work but all the files are downloading very slowly. It looks like you can’t specify which files to download

bs list datasets --filter-term=".*Pcli.*"
#this lists only the basespace files for Pcli

bs download project -n JA23031 --exclude '.*Acer.*'

I also don’t have space on my computer to download them locally. I have about 10 GB left of space on my computer. I can’t decide whether it is worth it to try gunzipping and reuploading all the Pcli ones as fastq files to NCBI, or whether it is better to just try to redownload all the sample files from Illumina Basespace using the command line interface (CLI). I’m going to try to redownload all the files directly to an external hard drive and see how that goes.

Update: I think I successfully downloaded all 96 sequence files (.fastq.gz) onto the 2TB external hard drive on Nov 5. When they download from Basespace, each sequence file is in its own subdirectory. For uploading to NCBI, all the sequence files have to be in one directory, no subfolders. So, I ran these two lines of code from Michael to easily fix that:

find . -name '*.gz' -exec mv {} . \;
rmdir SA*

Tutorials I found that were helpful: 1. https://r.qcbs.ca/workshop07/book-en/mixed-model-protocol.html 2. https://r.qcbs.ca/workshop07/book-en/step-3.-model-validation.html 3. https://r.qcbs.ca/workshop07/book-en/step-4.-interpretation-and-visualization.html 4. https://ourcodingclub.github.io/tutorials/mixed-models/ 5. https://rpubs.com/daharo_calpoly/502695

Some important takeaways: - “If your random effects are there to deal with pseudoreplication, then it doesn’t really matter whether they are “significant” or not: they are part of your design and have to be included.” - When doing model selection and comparing them, you cannot compare lmer models with lm models. Also, you need to make sure any of your mixed effects models are run with “REML=FALSE” so that the AIC can be correctly compared. But, once your final model is selected and it has fixed and random effects, you need to switch it back to “REML=TRUE”. - For post-hoc comparisons,

When I take all the DGEs (p-adjusted<0.05) from this dataset, and input their KO terms into KEGG mapper, I get this list of KEGG pathways:

Then, when you click on estrogen signaling specifically, these are the genes and associated KEGG terms from my list of DGEs.

Which is the same as what I get from the results table linked above. Estrogen signaling doesn’t have the most KO terms in this list, but it ends up being the most significant. The significance test to obtain p-values is the hypergeometric test, which is essentially testing if the number of genes in a given pathway is more than what would be expected by chance. This number is determined based on the “universe,” which is the number of KEGG terms in your dataset from the total number of genes in your dataset. In my dataset, there were 12,344 KO terms from the A.cervicornis genome. Then, of my 819 DGEs, 413 of them had KO terms associated with them. The KO terms and individual genes, however, don’t exactly point to estrogen signaling. The annotations seem more related to heat-shock and cell signaling (i.e. calmodulin, calcium ion binding).

So why is it linking these to the estrogen signaling pathway?

I think perhaps what is happening is that the most significant DGEs in my dataset all happen to be a part of estrogen signaling, but they also are a part of many other pathways. Yes, all of these genes together are significantly enriched in estrogen signaling because they passed the criteria for being more highly expressed than due to chance. But, what gene specifically would point to estrogen signaling.

I think I need a lot more evidence to support the role of endocrine/hormonal signaling in Coral City Camera corals in comparison to the Nursery corals.

Since I’m running low on time to prepare my defense presentation, I’m going to not go down this path any further at this time. But, I compiled a list of papers to review later and look at which genes are specifically linked to endocrine signaling in these papers. If a lot of my genes are also mentioned, then I think that warrants further discussion about the potential role of this pathway. DO THIS LATER!

papers: - https://peerj.com/articles/1982/ - https://www.int-res.com/abstracts/meps/v269/p121-129/ - https://www.sciencedirect.com/science/article/abs/pii/S1095643321000167?via%3Dihub - https://www.sciencedirect.com/science/article/abs/pii/S0048969722071194 - https://www.sciencedirect.com/science/article/abs/pii/S0166445X18305216?casa_token=XQuzXdc9at4AAAAA:QaPEECqFZxjvIwkzwiiaF9pyPDxnmWYYkRDmp2plorfIO4dBzt9s6I7hWEJWlr6h1LLKtqlC - https://www.int-res.com/abstracts/meps/v269/p121-129/ - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8910684/ - https://www.sciencedirect.com/science/article/pii/S0048969721037049#bb0665 - https://ehp.niehs.nih.gov/doi/10.1289/ehp.5233

I found Dr. Ana Palacio’s code on her GitHub that documents really well her scripts for downloading the data from NOAA’s database.

First, I downloaded the .nc file here: https://www.star.nesdis.noaa.gov/pub/sod/mecb/crw/data/5km/v3.1_op/climatology/nc/

Then, I needed to download a package for the terminal that allows me to read .nc files.

brew update
brew install netcdf

ncdump -h ct5km_climatology_v3.1.nc

I can see in the header that I think it includes all lat, lon coordinates.

Now, I read it into R using Ana’s code:

```r extract_oisst_data library(raster) library(parallel) library(ncdf4)

# Read climatology file Climatology_MMM <- “ct5km_climatology_v3.1.nc” MMM.data <- brick(Climatology_MMM) #MMM.data

# KB Nursery lat_KB <- (25.6763) lon_KB <- (-80.0987) extract.pts_KB <- cbind(lon_KB,lat_KB)

#CCC lat.CCC <- (25.766626) lon.CCC <- (-80.144812) extract_CCC <- cbind(lon.CCC, lat.CCC)

#MMM <- raster::extract(MMM.data, extract.pts,method=“bilinear”) MMM <- raster::extract(MMM.data, extract.pts_KB,method=“simple”) MMM_CCC <- raster::extract(MMM.data, extract_CCC,method=“simple”)

write.csv(MMM, “KB_CRW_MMM_1985-2012.csv”, row.names=FALSE)

write.csv(MMM_CCC, “CCC_CRW_MMM_1985-2012.csv”, row.names=FALSE)

So I transformed the new iso2go.tab file so that everything was consistent and named like “Acropora_”.

Today I just realized that in the KOG MWU script and in the annotating DGEs scripts for Acer, I was still using the old files that Michael had made (I realized this because the iso2geneName.tab and iso2kogclass.tab names are Acropora_, not Acervicornis).

Okay but now looking back at his old commit versions on the Acer annotated transcriptome repository on his GitHub, it looks like the naming convention for gene names used to be “Acropora1921” for example, which means it didn’t have an underscore and it also didn’t have zeros before the numbers (which he then added when he changed it to Acervicornis).

Just to be safe, I’m going to redownload all the annotation files and rerun everything that uses those files so that way I can be completely sure its all updated.

I noticed when I was looking at the different time points that six Acer are missing initial FvFm data, and when I look at the csv files that I exported from the IPAM, they have values that I guess weren’t imported into R using the IPAM2R code. I did edit the code because it wasn’t working, maybe that’s why it is importing things in correctly?

Ok, upon further inspection, the Acer that are missing the initial time point fvfm are all from one photo: photo number 4 (“4.csv”) from 2022-03-16. All the Acer that are missing the final time point fvfm are from tank 7 (2022-04-20). For Pclivosa there are several final time point ones that are missing (2022-04-20). Some of them are actually blank in the original IPAM csv files, but some of them have values that should’ve been read in.

So what happened there? I guess let me screenshot to keep record of which ones were missing, and then re-import everything using the IPAM2R code and see if it corrects itself?

UPDATE: I figured out the issue. The IPAM2R code has a line in it that uses the reshape2 function, and some of my csv files had two rows of identical IPAM data instead of one row - one row is what the IPAM2R code expects. I guess if you export your IPAM data twice on accident it will add another row to your csv file. But, when the function in R tries to reshape the data frame after tidying the AOI and other variable info, it basically extracts the wrong information and the Y(II) values end up being incorrect. For the Acer samples that were messed up, they had valuess of 2.00 instead of 0.646 etc that they were supposed to be. So I added a if-then statement within the IPAM2R data which basically removes the second row if there are two rows present in the csv file.

When I looked through the updated dataset that was created in R, I no longer had so many NAs as I did before. I cross-validated the values from the original csv files to make sure they matched, and they did. Yay problem solved!

My overall goals with my Fv/Fm data are to: 1. evaulate whether the variable temperature treatment impacted Fv/Fm in Acer and Pcli 2. evaluate thermal tolerance of temperature-treated versus untreated corals using CBASS and dose-response curve (DRC) modeliong

With that being stated, my move forward should be simple.

I think a couple obstacles that held me up were: 1. Do I / how do I normalize Fv/Fm data? Based on initial measurement (pre-treatment)? Based on pre-CBASS measurement? Do I incorporate number of days and create a rate of change and then statistically compare that? 2. Which statistical model do I use? I want to incorporate fixed (treatment, CBASS temperature) and random (genotype, tank) effects, so Generalized Linear Mixed Models is appropriate. However, I was stuck on using glmmTMB with the “beta” family distribution and the “logit” function. The major assumption of this type of distribution is that data are bound by 0 < x < 1. I originally thought fv/fm data would fit this best because it is a proportion that the IPAM is measuring. But, some of the values equal zero, and when I start doing time normalizations I get values greater than 1 or less than zero. So, for this reason I no longer think glmmTMB is appropriate. The next best GLMM would be the classic continuous response model using lme4::lmer. This is consistent with code I’ve seen from Klepac and Evensen.

See my previous GitHub posts for more confusions: 1. https://github.com/ademerlis/ademerlis.github.io/blob/main/_posts/2023-11-06-TroubleWithFvFmData.md 2. https://github.com/ademerlis/ademerlis.github.io/blob/main/_posts/2023-01-31-GLMMsContinued.md 3. https://github.com/ademerlis/ademerlis.github.io/blob/main/_posts/2023-01-29-GLMMs.md 4. https://github.com/ademerlis/ademerlis.github.io/blob/main/_posts/2023-01-10-LinearMixedModels_StressHardeningExp.md

Moving forward, I’m going to collate notes from Cunning et al 2021, Klepac and Barshis 2020, and Evensen et al 2021 here.

Since my Pegasus environment is now conda, I just ran this in the command line after I logged in:

conda install -c didillysquat -c conda-forge reefmapmaker

conda create --name reefmapmaker_env -c didillysquat -c conda-forge reefmapmaker

conda activate reefmapmaker_env

I downloaded locally the 2021 UNEP WCMC global distrubiton of warm-water coral reefs dataset from here, then unzipped the file. You need this directory to be in the current working directory for the reefMapMaker commands. I’ll create a directory on Pegasus for this and then upload the dataset using scp.

In the installation instructions it says I should be able to run “reefmapmaker” in the directory and it will automatically do it. But i keep getting errors that basically say it can’t find the shape file in the dataset I downloaded.

I tried running this and got “Could not automatically find the reference reef dataset. Please specify the directory of the dataset on the command line using –ref-reef-dir”

reefmapmaker --ref-reef-dir 14_001_WCMC008_CoralReefs2021_v4

Which one is the shape file?

I think it’s this: 01_Data/WCMC008_CoralReef2021_Py_v4_1.shp

I’m going to download their config_sheet.tsv and add my sites to a site_sheet.tsv.

I updated the bounds to fit my data: -84,-80,24,28 and added these sites to my tsv file: KBNursery: 25.6763, -80.0987 PcliCollection 25.77044 -80.15235

attempt #1 to run full code:

reefmapmaker --ref-reef-dir /scratch/projects/and_transcriptomics/programs/reefMapMaker/14_001_WCMC008_CoralReefs2021_v4/01_Data --config-sheet /scratch/projects/and_transcriptomics/programs/reefMapMaker/map/config_sheet.tsv --site-sheet /scratch/projects/and_transcriptomics/programs/reefMapMaker/map/site_sheet.tsv --fig-out-dir /scratch/projects/and_transcriptomics/programs/reefMapMaker/14_001_WCMC008_CoralReefs2021_v4 --bounds=-84,-80,24,28

I think I’m running into issues with the config sheet. I’m going to try running the code without it.

reefmapmaker --ref-reef-dir /scratch/projects/and_transcriptomics/programs/reefMapMaker/14_001_WCMC008_CoralReefs2018_v4/01_Data --site-sheet /scratch/projects/and_transcriptomics/programs/reefMapMaker/map/site_sheet.tsv --fig-out-dir /scratch/projects/and_transcriptomics/programs/reefMapMaker/14_001_WCMC008_CoralReefs2021_v4 --bounds=-84,-80,24,28

Now it’s back to not being able to find the shapefile. I’m going to try to move the zip file onto pegasus first and then unzip it there.

reefmapmaker --ref-reef-dir /scratch/projects/and_transcriptomics/programs/reefMapMaker/14_001_WCMC008_CoralReefs2021_v4_1 --site-sheet /scratch/projects/and_transcriptomics/programs/reefMapMaker/map/site_sheet.tsv --fig-out-dir /scratch/projects/and_transcriptomics/programs/reefMapMaker --bounds=-84,-80,24,28

Ok I found the issue. In the second of code of reefmapmaker.py (lines 439-452), it’s specifically looking for a shape file that starts with the string “WCMC008_CoralReef2018_Py”. However, when you download the dataset from the website link they provide, it’s a 2021 version, so all the file names start with “WCMC008_CoralReef2021”. ugh. I guess I need to manually edit the reefmapmaker.py code.

Can i do this in the command line? Yes, using sed.

First, navigate to where reefmapmaker was installed: /nethome/and128/anaconda3/envs/reefmapmaker_env/lib/python3.10/site-packages/reefmapmaker/

Then, run:

sed -i 's/WCMC008_CoralReef2018_Py/WCMC008_CoralReef2021_Py/g' reefmapmaker.py

#make executable
chmod +x reefmapmaker.py 

Now, navigate to directory which has the WCMC008_CoralReefs2021 data and run:

reefmapmaker --ref-reef-dir /scratch/projects/and_transcriptomics/programs/reefMapMaker/14_001_WCMC008_CoralReefs2021_v4_1 --site-sheet /scratch/projects/and_transcriptomics/programs/reefMapMaker/map/site_sheet.tsv --fig-out-dir /scratch/projects/and_transcriptomics/programs/reefMapMaker --bounds=-84,-80,24,28

This didn’t work, and neither did running the baseline code:

reefmapmaker --ref-reef-dir /scratch/projects/and_transcriptomics/programs/reefMapMaker/14_001_WCMC008_CoralReefs2021_v4_1

this was the log and the errors that came up:

Shape file found: /scratch/projects/and_transcriptomics/programs/reefMapMaker/14_001_WCMC008_CoralReefs2021_v4_1/01_Data/WCMC008_CoralReef2021_Py_v4_1.shp
Drawing annotations on map

Annotations complete

Annotating reference reefs

reading in reference reefs
done
Traceback (most recent call last):
  File "/nethome/and128/anaconda3/envs/reefmapmaker_env/lib/python3.10/site-packages/reefmapmaker/reefmapmaker.py", line 1002, in _add_reference_reefs
    self._handle_multipolygon(r)
  File "/nethome/and128/anaconda3/envs/reefmapmaker_env/lib/python3.10/site-packages/reefmapmaker/reefmapmaker.py", line 1069, in _handle_multipolygon
    for polygon in r.geometry:
TypeError: 'MultiPolygon' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nethome/and128/anaconda3/envs/reefmapmaker_env/bin/reefmapmaker", line 22, in <module>
    mwif.draw_map()
  File "/nethome/and128/anaconda3/envs/reefmapmaker_env/lib/python3.10/site-packages/reefmapmaker/reefmapmaker.py", line 911, in draw_map
    self._add_reference_reefs()
  File "/nethome/and128/anaconda3/envs/reefmapmaker_env/lib/python3.10/site-packages/reefmapmaker/reefmapmaker.py", line 1009, in _add_reference_reefs
    error_count += 1
UnboundLocalError: local variable 'error_count' referenced before assignment

I’m going to give up on this for now because I give up.

Step 1 is installing bioperl. I had trouble doing this, but I didn’t realize I could activate it as a conda environment. It looks like it’s working so far.

# To install Bioperl as a conda environment
conda create -y -n bioperl perl-bioperl

conda activate bioperl

git clone https://github.com/z0on/annotatingTranscriptomes.git

git clone https://github.com/z0on/emapper_to_GOMWU_KOGMWU.git

seq_stats.pl /scratch/projects/and_transcriptomics/genomes/Acer/Locatelli_2023/Acer_Genome/Acropora_cervicornis.mrna-transcripts.fa  > seqstats_Acer.txt

I had several issues running the seq_stats.pl script.

First, I had to update the shebang in the .pl script so that it was calling the right perl (within the bioperl environment).

Then, I also had to remove the line use Bio::Perl; because it was causing issues.

So the seq_stats.pl script beginning looks like this now:

#!/usr/anaconda3/envs/bioperl/bin/perl

use lib "~/anaconda3/envs/bioperl";
use Bio::SeqIO;

Now I run this with the updated script:

seq_stats.pl /scratch/projects/and_transcriptomics/genomes/Acer/Locatelli_2023/Acer_Genome/Acropora_cervicornis.mrna-transcripts.fa  > seqstats_Acer.txt

Results:

36455 sequences. 1678 average length. 65308 maximum length. 61 minimum length. N50 = 2747 61.2 Mb altogether (61163689 bp). 0 ambiguous Mb. (100 bp, 0%) 0 Mb of Ns. (100 bp, 0%)

Next step is to get the uniprot annotations with blast.

# getting uniprot_swissprot KB database
wget ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz

gunzip uniprot_sprot.fasta.gz

Then I created a script for the makedatabase part.

#!/usr/bin/bash
#BSUB -P and_transcriptomics
#BSUB -n 8
#BSUB -W 120:00
#BSUB -o makedatabase.out
#BSUB -e makedatabase.err

module load blast/2.2.29+
makeblastdb -in uniprot_sprot.fasta -dbtype prot

That ran.

Now, I need to split up the fasta file so the blast code runs quickly.

#navigate to wd where splitFasta.pl is located 
#copy fasta file into this directory
cp /scratch/projects/and_transcriptomics/genomes/Acer/Locatelli_2023/Acer_Genome/Acropora_cervicornis.mrna-transcripts.fa Acer2023.fasta

#run split Fasta code
perl splitFasta.pl Acer2023.fasta 200

This created 200 subset.fasta files. Now, need to create a script to submit a job to pegasus that blasts all 200 chunks to uniprot in parallel.

I think I’ll need to create a for-loop so it creates individual jobs for each chunk, so that they can run simultaneously.

This for loop worked:

#BSUB -u your_email@miami.edu

#specify variables and paths

#define variables for directories and files
and="/scratch/projects/and_transcriptomics"
project="and_transcriptomics"
projdir="/scratch/projects/and_transcriptomics/genomes/Acer/annotatingTranscriptomes-master"

cd ${projdir}

data=($(ls subset*))

for file in "${data[@]}" ; do \

#build script
echo "making blast uniprot script for ${file}..."
echo "
#! /usr/bin/env bash
#BSUB -P ${project}
#BSUB -J ${file}_blast_uniprot
#BSUB -e ${projdir}/logs/${file}_blast_uniprot.err
#BSUB -o ${projdir}/logs/${file}_blast_uniprot.out
#BSUB -W 12:00
#BSUB -n 8
#BSUB -q general

cd ${projdir}

module load blast/2.2.29+

blastx -query ${file} -db uniprot_sprot.fasta -evalue 0.0001 -num_threads 4 -num_descriptions 5 -num_alignments 5 -out ${file}.br" > ${projdir}/${file}_blast_uniprot.job

bsub < ${projdir}/${file}_blast_uniprot.job

done

This took ~ l hr to complete.

To make sure it worked, run the following line and you should end up with the same number of queries as there are sequences from the seqstats_Acer.txt file (created from running seq_stats.pl).

grep "Query= " subset*.br | wc -l

I get 36,455, which equals the number of sequences in the seqstats_Acer.txt file.

Next, combine all blast results:

cat subset*br > myblast.br

Then, extract coding sequences and corresponding protein translations using the perl script CDS_extractor_v2.pl.

This is a really long script so I’m not going to paste it in here. I’m first going to try to run it as-is. It looks like it requires Bio::SeqIO so we’ll see how that goes.

I should probably create a job script and submit this as a job just in case it takes awhile.

Michael runs CDS_extractor_v2.pl on a file he made called “Acervicornis_iso.fasta”. It looks like he ran this code on the original fasta file:

cat Acervicornis.fasta | perl -pe 's/>Acropora(\d+)(\S+).+/>Acropora$1$2 gene=Acropora$1/'>Acervicornis_iso.fasta

He had flagged this line of code for only being specific to Trinity-assembled transcriptomes.

The one I’m using was generating using “funannotate.” So, each header of the fasta file looks something like this:

>FUN_000006-T1 FUN_000006

I can try to adapt Michael’s code so it does the same thing, which is basically substituting the trinity header to be named an arbitrary gene name (i.e. Acropora00001).

update: actually i’m not sure I want to do that, because then I would also have to edit the enitre gff3 file (the gene IDs in there are FUN_#). I’m going to leave it as is and change the Acervicornis_seq2iso.tab code to grep and awk FUN instead of Acropora.

Remember the Acer_2023.fasta is the mrna_transcripts.fa file that I just copied over from the original genome download folder.

“Acervicornis_seq2iso.tab” is the necessary file for generating read counts per gene back in the tagSeq bioinformatics pipeline (which is the whole reason i’m running this annotatingTranscriptomes pipeline).

So i should run both lines to get both files: Acervicornis_seq2iso.tab and Acervicornis_iso.fasta.

grep ">" Acer2023.fasta | perl -pe 's/>FUN(\d+)(\S+)\s.+/FUN$1$2\tFUN$1/'>Acervicornis_seq2iso.tab

cat Acer2023.fasta | perl -pe 's/>FUN(\d+)(\S+).+/>FUN$1$2 gene=FUN$1/'>Acervicornis_iso.fasta

Ok the second line doesn’t do anything because the original fasta file is already listed in this exact format. So i’m not sure if any of this code is actually useful.

# extracting coding sequences and corresponding protein translations:
conda activate bioperl #if not active already
echo "perl ~/bin/CDS_extractor_v2.pl Acervicornis_iso.fasta myblast.br allhits bridgegaps" >cds

#! /usr/bin/env bash
#BSUB -P and_transcriptomics
#BSUB -J CDS_extractor
#BSUB -e /scratch/projects/and_transcriptomics/genomes/Acer/annotatingTranscriptomes-master/CDS_extractor.err
#BSUB -o /scratch/projects/and_transcriptomics/genomes/Acer/annotatingTranscriptomes-master/CDS_extractor.out
#BSUB -W 12:00
#BSUB -n 8
#BSUB -q general

cd "/scratch/projects/and_transcriptomics/genomes/Acer/annotatingTranscriptomes-master"

perl CDS_extractor_v2.pl Allyson_Created_Files/Acer2023.fasta Allyson_Created_Files/myblast.br allhits bridgegaps

To run the above job script, I had to manually install the perl module “Bio::DB::Fasta:. I ran this:

cpan Bio::DB::Fasta

But this doesn’t work because there are a ton of other dependencies it needs. I’m going to abandon this pipeline at this step because there is an annotation file that came with the Locatelli 2023 Acer genome and it has GO terms in it. I’ll just move forward with that for right now.

Finally getting the gene counts for Acer samples from Chapter 2

Following the last section of Michael Studivan’s code.

# NOTE: Must have a tab-delimited file giving correspondence between contigs in the transcriptome fasta file and genes

cat Acer/Locatelli_2023/Acervicornis_seq2iso.tab Symbiodinium/Mstudiva_Symbiodinium_annotated_transcriptome_code/Symbiodinium_seq2iso.tab > Host_concat_seq2iso.tab

Job: use sam tools and samcount.pl.

arg1: SAM file (by cluster, contig, or isotig) arg2: a table in the form ‘reference_seqgene_ID’, giving the correspondence of reference sequences to genes. With 454-deived transcriptome, the gene_ID would be isogroup; with Trinity-derived transcriptiome,it would be component.

paths: 1. /scratch/projects/and_transcriptomics/Ch2_temperaturevariability2023/2_trimmed_reads/take_4/trimmed_files/Acer/bowtie2align_LocatelliShoguchi/sam_files/*.sam 2. /scratch/projects/and_transcriptomics/genomes/Host_concat_seq2iso.tab

I need to write a for-loop to each .sam file to do this: print “samcount.pl $file $tab aligner=bowtie2 >$file.counts” but written in bash not perl.

#! /usr/bin/env bash

#define variables for directories and files
and="/scratch/projects/and_transcriptomics"
project="and_transcriptomics"
projdir=

cd "/scratch/projects/and_transcriptomics/Ch2_temperaturevariability2023/2_trimmed_reads/take_4/trimmed_files/Acer/bowtie2align_LocatelliShoguchi/sam_files"

data=($(ls *.sam))

for samp in "${data[@]}" ; do \

#build script
echo "making sam_counts script for ${samp}..."
echo "
#! /usr/bin/env bash
#BSUB -P ${project}
#BSUB -J ${samp}_samcounts
#BSUB -e ${and}/Ch2_temperaturevariability2023/3_genecounts/logs/${samp}_samcounts.err
#BSUB -o ${and}/Ch2_temperaturevariability2023/3_genecounts/logs/${samp}_samcounts.out
#BSUB -W 12:00
#BSUB -n 8
#BSUB -q general

cd \"/scratch/projects/and_transcriptomics/Ch2_temperaturevariability2023/2_trimmed_reads/take_4/trimmed_files/Acer/bowtie2align_LocatelliShoguchi/sam_files\"

module load samtools/1.3

perl samcount.pl ${samp} /scratch/projects/and_transcriptomics/genomes/Host_concat_seq2iso.tab aligner=bowtie2 >${samp}.counts

" > ${and}/Ch2_temperaturevariability2023/3_genecounts/${samp}_samcounts.job

bsub < ${and}/Ch2_temperaturevariability2023/3_genecounts/${samp}_samcounts.job

done

When I ran this though on Pegasus, the counts files came out with nothing in them. Moreover, all the .err files have errors like:

Why didn’t any of this work? It looks like it maybe has to do with the isogroup assignment?

This is what Michael ran on his samples before making the seq2iso.tab file: (link)

# Renaming gene identifiers for ease
sed -i 's/comp/Acropora/g' Acervicornis.fasta
sed -i 's/isogroup/Acropora/g' Acervicornis.fasta

I didn’t initially run this on my fasta file but I’ll try it now and see if it helps.

# Renaming gene identifiers for ease
sed -i 's/comp/Acropora/g' Acropora_cervicornis.mrna-transcripts.fa > Acer2023_edited.fasta
sed -i 's/isogroup/Acropora/g' Acropora_cervicornis.mrna-transcripts.fa > Acer2023_edited.fasta

It doesn’t help because there is nothing labeled “comp” or “isogroup in the fasta file I’m using from Locatelli, which is called Acropora_cervicornis.mrna-transcripts.fa. I looked through the other file types (.gff3, protein.fa, scaffolds.fa) and none of them have the comp or isogroup headers.

I ended up running this code to create a seq2iso.tab file.

grep ">" Acropora_cervicornis.mrna-transcripts.fa | perl -pe 's/>FUN(\d+)(\S+)\s.+/FUN$1$2\tFUN$1/'>Acervicornis_seq2iso.tab

This essentially just extracts the header line from each gene and puts it in a tab-delimited text file.

But then when I run samcount.pl using this seq2iso.tab file, I get no results.

December 20 update:

I had Michael take a look at the seq2iso.tab file, and the issue is that each line starts with “>” but it shouldn’t. It’s messing up the gene names.

So two things I need to do: First, update the Acropora one to have the genes be named “Acropora_000001” instead of “FUN_000001”, and then update the Symbiodinium fasta file to say Symbiodinium_000001 for those genes. Then create the seq2iso.tab files for both species, concatenate them, then re-run the samcount.pl job.

# Renaming gene identifiers for ease

#make new file
cp Acropora_cervicornis.mrna-transcripts.fa Acer_edited.fasta
cp syma_transcriptome_37.fasta Symbiodinium_edited.fasta

sed -i 's/FUN/Acropora/g' Acer_edited.fasta

sed -i 's/comp/Symbiodinium/g' Symbiodinium_edited.fasta
sed -i 's/isogroup/Symbiodinium/g' Symbiodinium_edited.fasta


# making seq2iso.tab files
grep ">" Acer_edited.fasta | perl -pe 's/>Acropora_(\d+)(\S+)\s.+/Acropora_$1$2\t Acropora_$1/'>Acervicornis_seq2iso.tab
grep ">" Symbiodinium_edited.fasta | perl -pe 's/>Symbiodinium(\d+)(\S+)\s.+/Symbiodinium$1$2\t Symbiodinium$1/'>Symbiodinium_seq2iso.tab

# create combo file

cat Acer/Locatelli_2023/Acer_Genome/Acervicornis_seq2iso.tab Symbiodinium/Symbiodinium_seq2iso.tab > Host_concat_seq2iso.tab

Now i should be able to re-run this script with no changes:

Incorrect, the sam files are all named with the original gene names so nothing is going to match up.

I can do one of two things: either rename all the same files to have Acropora instead of FUN, or remake the seq2iso.tab files with the original names and then rename everything after the fact when its in the gene counts matrix.

The sam files are huge so I’m going to try to rename them after the fact.

# making seq2iso.tab files
grep ">" Acropora_cervicornis.mrna-transcripts.fa | perl -pe 's/>FUN_(\d+)(\S+)\s.+/FUN_$1$2\t FUN_$1/'>Acervicornis_seq2iso.tab
grep ">" syma_transcriptome_37.fasta | perl -pe 's/>comp(\d+)(\S+)\s.+/comp$1$2\t comp$1/'>Symbiodinium_seq2iso.tab

# create combo file

cat Acer/Locatelli_2023/Acer_Genome/Acervicornis_seq2iso.tab Symbiodinium/Symbiodinium_seq2iso.tab > Host_concat_seq2iso.tab

Ok I think that worked?! yay

I just ran this in the terminal and it worked fast.

perl expression_compiler.pl *.counts > allcounts.txt

# let's remove those annoying chains of extensions from sample names
cat allcounts.txt | perl -pe 's/\.trim\.sam\.counts//g'> counts.txt

# i also need to remove sample name extensions "i.e. _S25_L001_R1_001"

cat counts.txt | perl -pe 's/(_S\d+_L\d+_R\d+_\d+)//g' > count.txt

#and rename FUN -> Acropora and comp -> symbiodinium
sed -i 's/FUN/Acropora/g' count.txt
sed -i 's/comp/Symbiodinium/g' count.txt

Ok now that it’s all tidied up, I used scp to move it onto my local drive. All done with Pegasus?!!?!!

“Degree Heating Weeks (DHW) is a measure used to assess the thermal stress on coral reefs, which is a significant factor in coral bleaching events. It’s calculated by accumulating the amount of time sea surface temperatures exceed the long-term historical maximum mean temperature for a particular location. Here’s how you calculate DHW:

1. Determine the Historical Maximum Mean Temperature (MMM): This is the highest average monthly sea surface temperature for a given location over a long-term period (usually over several decades). This temperature serves as a threshold.

2. Monitor Daily Sea Surface Temperatures (SST): Obtain daily SST data for your location. This data is often available from satellite observations or oceanographic institutions.

3. Calculate the Temperature Anomaly: For each day, determine how much the SST exceeds the MMM. This is the temperature anomaly. If the SST is lower than the MMM, the anomaly is considered zero for that day.

4. Accumulate the Anomalies Over Time: Degree Heating Weeks are calculated over a rolling 12-week (84-day) period. For each day, add the temperature anomaly to a cumulative sum. Continue this process for each day in the 12-week period.

5. Express the Total as Weeks: The total accumulated temperature anomaly is then divided by 7 to express it in terms of “degree heating weeks.” This unit represents the number of weeks the sea temperature has been above the MMM.

6. Update Regularly: DHW is a rolling measure, so it’s updated regularly (daily or weekly) to reflect ongoing temperature conditions.

DHW is a critical tool for marine scientists and environmental managers in monitoring and responding to the impacts of climate change on coral reef ecosystems. High DHW values indicate prolonged thermal stress, which can lead to coral bleaching and other ecological impacts.

Determining the Historical Maximum Mean Temperature (MMM) for a specific location involves analyzing long-term sea surface temperature data. The MMM is the highest average monthly sea surface temperature typically calculated over a baseline period of several decades. Here’s a step-by-step process to determine the MMM:

1. Collect Long-term Sea Surface Temperature Data: Obtain historical sea surface temperature (SST) data for your location. This data is often available from meteorological organizations, oceanographic institutes, or satellite observations. The dataset should ideally span at least 20-30 years to capture climate variability.

2. Select the Baseline Period: Choose a baseline period for your analysis. This period should be long enough to accurately reflect the range of natural temperature variability. Commonly, a 30-year period is used in climate studies.

3. Calculate Monthly Averages: For each year in the baseline period, calculate the monthly average SST. This process involves averaging all daily temperature readings for each month.

4. Identify the Maximum Monthly Average for Each Month: For each month of the year, identify the highest monthly average SST recorded during the baseline period. This gives you 12 values, each representing the highest monthly average SST for a specific month across the baseline period.

5. Determine the MMM: The MMM is typically the highest of these 12 values. It represents the peak thermal threshold that the location experiences naturally.

6. Consider Spatial Resolution: Depending on your study’s scale, you may need to consider the spatial resolution of your data. For large areas, you might need to calculate the MMM for different zones.

7. Account for Climate Change and Variability: Be aware that climate change can affect the MMM over time. Periodically updating the MMM with recent data can ensure that your analysis reflects current climate conditions.

8. Use of Remote Sensing and Models: In many cases, remote sensing data from satellites and oceanographic models are used to derive SST data, especially for remote or vast ocean areas.

9. Verification and Validation: Validate your calculated MMM with any available in-situ measurements or compare it with results from other studies or datasets to ensure accuracy.

By following these steps, you can determine the Historical Maximum Mean Temperature for a given location, which is crucial for assessing thermal stress on marine ecosystems, particularly coral reefs.”

The NOAA CoralReefWatch has tutorials on all their products.

I think the one I need to use is the NCEI AVHRR Pathfinder SST. The issue is, when I try to download the data files from 1985-2023, they are all .nc extensions. This is a NetCDF4 type file, which is not readable unless you use a code to convert it.

Some people in the field use MATLAB, but I’m going to try to use Python in the terminal.

ChatGPT:

“Working with NetCDF4 data, which is a common format for storing large, multi-dimensional scientific data such as sea surface temperature (SST), can be done using various software tools. Here’s a general guide on how to extract data from a NetCDF4 file into a more readable format:

python3 --version
pip3 install netCDF4 xarray numpy matplotlib #this line did not work because I need other packages (i.e. hdf5)

#install homebrew
 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

#install hdf5
brew install hdf5

#add this to ~/.bash_profile
export HDF5_DIR=$(brew --prefix hdf5)

export PATH=/Users/allysondemerlis/Library/Python/3.9/bin:$PATH #command line told me to do this

curl https://www.ncei.noaa.gov/data/oceans/pathfinder/Version5.3/

python3 #activate python

import netCDF4 as nc
import xarray as xr

dataset = xr.open_dataset('~/Downloads/19811231141339-NCEI-L3C_GHRSST-SSTskin-AVHRR_Pathfinder-PFV5.3_NOAA07_G_1981365_day-v02.0-fv01.0.nc')

sst_data = dataset['sea_surface_temperature'

sst_df = sst_data.to_dataframe()

print(sst_df.head())  # Displays the first 5 row

#longitude boundaries: -80.001, -80.17
#latitude boundaries: 25.60, 25.80

# Define your latitude and longitude range
lat_range = [26, 25]  # Replace with your desired range
lon_range = [-81, -80]  # Replace with your desired range

# Filter the data
filtered_data = sst_data.sel(lat=slice(*lat_range), lon=slice(*lon_range))

print(filtered_data)

import pandas as pd
df = filtered_data.to_dataframe()
df.to_csv('output_filename.csv')