KillMS error with parameter TChunk

[astro-fgao]

Hi there,

I tried to run kMS on some Meerkat data (only 2 mins scan) and I found that if I set the TChunk parameter smaller than the targeted data itself, then there will be en error message complaining about not finding data column (as shown below)

I've tried with different TChunk length and it seems as long as this parameter is smaller than the data, then this error will occur. Currently I'm using a large TChunk and everything is fine, but I can see this becoming a problem e.g. when working on some science targets with long scans.

FYI, the command I used are:

kMS.py --MSName="../../msdir/1579141864_sdp_l0.POLCALIBRATED.ms" --FieldID=17 --SolverType=KAFCA --PolMode=Scalar --BaseImageName=image_DI_Clustered.DeeperDeconv --dt 5 --NCPU 96 --OutSolsName DD0 --NChanSols 20 --InCol CORRECTED_DATA --TChunk 5 --BeamModel FITS --FITSParAngleIncDeg 0.5 --FITSFile='/localwork/feng/data/widefield_pol/fgao/beam-model/meerkat_pb_jones_cube_95channels_$(xy)_$(reim).fits' --CenterNorm 1 --FITSFeed xy --FITSFeedSwap 1 --ApplyPJones 1 --FlipVisibilityHands 1 --NChanBeamPerMS 20

Thank you ~

[cyriltasse]

Well - it's weird - there seem to be a problem reading DATA_DESC_ID in the MS... Or maybe it doesn't see the DATA_DESC_ID in the current chunk?

Also - I don't get it - you seem to be using TChunk=5, but the loaded chunk is only one hour?

How large is the MS? Could you post it somewhere with everything needed so I could reproduce?

[astro-fgao]

Hi Cyril,

Update on this issue:

the .ms data I used above contains multiple fields, and I'm only using one field there. Today I found that if I split out this field, then run ddf/kms on this single field data, kms can finish successfully even if I used a TChunk smaller than the data itself.

Instead, if I don't split the data, then the error would occur.

Currently I can trim the MS down to few hundreds Megabites, and I'm still looking for a place to post everything for you to try.

Also - I don't get it - you seem to be using TChunk=5, but the loaded chunk is only one hour?

Sorry, I copy/paste the wrong command line, which got you confused. In this case above, I was actually using TChunk=1, the total data chunk is like 4 something, this would cause the error message. When I was using TChunk=5, everything is fine.