rerun from chainSort

[sqwwww]

hi，

the program failed in the last step(chainSort), is there a way to rerun from the chainSort step?

here are some lines in my log:

1.2 get fills/gaps from tmp.chainCleaner.X6tzroP.net ...
DONE

1.3 get aligning regions from tmp.chainCleaner.X6tzroP.net ...
DONE

1.4 get valid breaks ...
DONE
Remove temporary netfile tmp.chainCleaner.X6tzroP.net
DONE (parsing fills/gaps and getting valid breaks)

2. reading breaking and broken chains from 01.test/test_out/temp_chain_run/zbf.Ha.before_cleaning.chain.gz and write irrelevant chains to 01.test/test_out/temp_chain_run/zbf.Ha.filled.chain__temp.unsorted ...
   DONE

3. reading target and query DNA sequences for breaking and broken chains ...
   DONE

4. loop over all breaks. Remove suspects if they pass our filters and write out deleted suspects to 01.test/test_out/temp_chain_run/removed_suspects.bed ...
   DONE

5. write the (new) breaking and the broken chains to 01.test/test_out/temp_chain_run/zbf.Ha.filled.chain__temp.unsorted ...
   DONE

6. chainSort 01.test/test_out/temp_chain_run/zbf.Ha.filled.chain__temp.unsorted 01.test/test_out/temp_chain_run/zbf.Ha.filled.chain__temp ...
   ERROR: chainSort failed. Command: chainSort 01.test/test_out/temp_chain_run/zbf.Ha.filled.chain__temp.unsorted 01.test/test_out/temp_chain_run/zbf.Ha.filled.chain__temp

slurmstepd: error: Detected 1 oom-kill event(s) in step 1509017.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

[MichaelHiller]

Sounds like out of memory.
chainSort loads all chains into the memory which can be a lot.
Could you run it on a machine with 64 or more GB?
Or configure the server to use disk as virtual RAM?

[sqwwww]

Sounds like out of memory. chainSort loads all chains into the memory which can be a lot. Could you run it on a machine with 64 or more GB? Or configure the server to use disk as virtual RAM?

Hi, MichaelHiller
thanks for your quick reply.

I checked the memory usage， and the process was killed by out-of-memory error.

seff 1509017
Job ID: 1509017
Cluster: cluster
User/Group: suqianwei/zhou3lab
State: OUT_OF_MEMORY (exit code 0)
Cores: 1
CPU Utilized: 00:38:10
CPU Efficiency: 1.43% of 1-20:24:10 core-walltime
Job Wall-clock time: 1-20:24:10
Memory Utilized: 1.95 GB
Memory Efficiency: 99.75% of 1.95 GB

Here I have two questions please,
The first, I used nextflow and slurm job submission system to submit jobs. it seems that the default memory distributed to each job is 2g， how to change the memory distribution from 2g to more? (I know how to change memory settings in one slurm script, but have no idea how to change the memory settings in these automatic running scripts.)
The second, I want to continue running the program at the crash point using the new memory settings，are there any parameters to do this?

It will really help!
many thanks!

[MichaelHiller]

2 GB may not be enough.
I think
#SBATCH --mem-per-cpu=$memoryMb
can be used to submit jobs that require more mem.

There is no way to restart chainCleaner (the step that crashed). But chainCleaner only takes a few min. Run this from scratch and then continue with the downstream steps pls

[sqwwww]

ok, I'll try this, many thanks!