Chain after RepeatFiller with wrong cordinates

[ebioman]

Hi
I am trying RepeatFiller on multiple chain files for hg38->T2T and I am encountering some inconsistent errors.
The official chain file from T2T runs through without anything being reported - seems no repeats are anymore present.
Using self generated ones (from minimap2 and GSAlign) I encounter a weird situation that chain files are generated but then are faulty.

python3 /usr/local/GenomeAlignmentTools/src/RepeatFiller.py  \
  --chain Minimap2_liftover.chain --T2bit hg38_p8_primaryContigs.2bit \
  --Q2bit chm13v2.0.2bit -o Minimap2_liftover.repeatFiltered.chain 

If I try then any kind of command afterwards, e.g.

chainPreNet Minimap2_liftover.repeatFiltered.chain  hg38_p8_primaryContigs.sizes chm13v2.0.sizes stdout 

q end mismatch 242669717 vs 242693499 line 54824 of Minimap2_liftover.repeatFiltered.chain

It fails with that error, other tools such as chainSorter as well. If I use though instead my file Minimap2_liftover.chain then everything goes smoothly. I tried as well flipping target and query in the RepeatFiller command as I was not sure about the definition, and astonishing (and worringly) it actually went through ....
But I get then similarly an incompatible chain file at the end.

I used the latest release from your tools

[MichaelHiller]

I'll ask Katya to have a look as we had this error before and it should be fixed.

In the meantime, could you pls try to compute the chains using https://github.com/hillerlab/make_lastz_chains
which automates all steps, including lastz, axtChain, (should have the latest) RepeatFiller and chainCleaner.
Pls send a ping if this throws the same error.

[ebioman]

Hi
Tried that but run into many new errors, most which I believe are not even related to the input data but purely based technical issues.
Do you have by chance a docker or minimal reproducible example to verify on my side that everything is running as intended ?

[osipovarev]

Hi,
Thank you for reporting the issue.
Could you share the input chains for RepeatFiller (from minimap or GSAlign) where it fails to produce the correct output?
I will try to reproduce the issue and look at what is happening there.
Thanks!

[ebioman]

Hello @osipovarev
Sorry for the delayed reply, I can definitively share them. I will look for the easiest way to make them available.
Thanks :)

[ebioman]

Hi
I put the files here for you link

[osipovarev]

Hi @ebioman ,
Thanks a lot for sharing the files! I now see that the RepeatFiller output chain Minimap2_liftover.repeatFiltered.chain has formatting issues, but I would need the initial input chain to figure out why this happens. Would you mind sharing Minimap2_liftover.chain as well? Or the pipeline you used to get to this file from the data you shared already.

[shuifeng1988]

I have the same error, any solution?

[MichaelHiller]

@osipovarev Katya, could you pls have another look? RepeatFiller works for us internally. Thx !!

[shuifeng1988]

ASM764659v3-chrUn_NKLS02001428.1.filled.sorted.chain.txt
ASM764659v3-chrUn_NKLS02001428.1.filled.chain.txt
ASM764659v3-chrUn_NKLS02001428.1.chain.txt
ASM764659v3-chrUn_NKLS02001428.1.axt.txt

This is my commands for genome alignment,my python version is 3.8.15;
step1: /gpfs/home/huayq/miniconda3/bin/lastzchrUn_NKLS02001428.1.2bit ASM764659v3_genomic.fa K=4500 L=4500 Y=9400 E=30 H=2000 O=400 T=1 --format=axt --ambiguous=n --ambiguous=iupac --output=ASM764659v3-chrUn_NKLS02001428.1.axt 1>&2 2>lastz.err

step2: axtChain -faT -faQ -linearGap=medium -minScore=3000 ASM764659v3-chrUn_NKLS02001428.1.axt chrUn_NKLS02002209.1.fa ASM764659v3_genomic.fa ASM764659v3-chrUn_NKLS02001428.1.chain 1>&2 2>axtChain.err

step3: /gpfs/home/huayq/soft/GenomeAlignmentTools-master/src/RepeatFiller.py -c ASM764659v3-chrUn_NKLS02001428.1.chain -T2 chrUn_NKLS02001428.1.2bit -Q2 ASM764659v3_genomic.fa.2bit -o ASM764659v3-chrUn_NKLS02001428.1.filled.chain 1>&2 2>RepeatFiller.err

step4: chainSort ASM764659v3-chrUn_NKLS02001428.1.filled.chain ASM764659v3-chrUn_NKLS02001428.1.filled.sorted.chain 1>&2 2>ChainSort.err

the step1 and step2 is running successfully, the step3 running completely without error, but after the step3, the step4 can not run and show errors in file "ChainSort.err", The error is "q end mismatch 209850 vs 209891 line 1453 of ASM764659v3-chrUn_NKLS02001428.1.filled.chain". However, when i used the output file of step2 to do the chainSort, it is ok! So, the problem is in the RepeatFiller process! I remmber i can run RepeatFiller successfully before, but the errors occurs suddenly, I don't know why, may be python version?

Also, there is another error in the ChainSort.err file, The error is "Expecting number field 1 line 3746 of ASM764659v3-chrUn_NKLS02002209.1.filled.chain, got chain"
ASM764659v3-chrUn_NKLS02002209.1.filled.sorted.chain.txt
ASM764659v3-chrUn_NKLS02002209.1.filled.chain.txt
ASM764659v3-chrUn_NKLS02002209.1.chain.txt
ASM764659v3-chrUn_NKLS02002209.1.axt.txt

[MichaelHiller]

I have asked @osipovarev to have a look.
Are these all input files to re-run this step? We likely also need the 2bit files I guess.
Thx

[shuifeng1988]

I have asked @osipovarev to have a look. Are these all input files to re-run this step? We likely also need the 2bit files I guess. Thx

Yes, it need 2bit files, because it is too big, almost 1Gb, so i update these to Baidu Netdisk, you can download these files through this link: https://pan.baidu.com/s/1H7ENJQKVW6uf54OUeWbm9g?pwd=bs48, In additional, you may need the Netdisk software to download those 2bit files, https://pan.baidu.com/download! Sorry, i don't know other methods to transfer large file! Thanks a lot!

[osipovarev]

Hi @shuifeng1988 ,
I am now looking into this and I have trouble accessing 2bit files. To get the files form Baidu I need to register, which requires a Chinese phone number: "Registration from overseas and Hong Kong, Macao and Taiwan regions is not currently supported."
Is there any other way I could get/generate 2bit files? The reference seems to be this Bos mutus assembly: https://www.ncbi.nlm.nih.gov/assembly/GCA_007646595.3/ - I could use this to generate the ref 2bit. Any chance I could also get the query from NCBI?

[MichaelHiller]

@shuifeng1988 Does upload to google drive work?
Using the exact 2bit that you use would be better to figure out what the problem is.

@osipovarev do you think a .1 at the end of chrom names is causing problems?
chain 2255979 chrUn_NKLS02002209.1 27572 + 0 27572 VBQZ03000128.1 5497613 - 11645 37929 1

[shuifeng1988]

Hi @shuifeng1988 , I am now looking into this and I have trouble accessing 2bit files. To get the files form Baidu I need to register, which requires a Chinese phone number: "Registration from overseas and Hong Kong, Macao and Taiwan regions is not currently supported." Is there any other way I could get/generate 2bit files? The reference seems to be this Bos mutus assembly: https://www.ncbi.nlm.nih.gov/assembly/GCA_007646595.3/ - I could use this to generate the ref 2bit. Any chance I could also get the query from NCBI?

Yes, i use the Bos mutus assembly you specified to generate my 2bit file, I have uploaded the three 2bit files to my google driver, https://drive.google.com/drive/folders/1mzukarI6fBmGxspH5lbIsZbKpTSgu0fc?usp=sharing, could you try to download from here!

[osipovarev]

@shuifeng1988 yes, I got them, thanks! Will try to recreate the error.