Stop all motors (:Q#) can crash OAE

[c3n]

Using serial control (via indi or directly with minicom) when issuing a :Q# (stop all motors) command while the scope is stopped or tracking crashes the controller (it stops responding to everything and needs to be power cycled). This is especially annoying since it's so easy to click stop movement when the slew just completed. Also, when using ccdciel the "stop tracking" button actually sends "stop movement". I'm using the newest version of Open Astro Explorer firmware.

[ClutchplateDude]

This does not repro for me. Do you have to have slewed before? Are there any other conditions? I connected, sent :GVP#, then a :Q# and then a :GVP#. and both returned the mount name.

[ClutchplateDude]

Oh, I think I know what might be happening. The documentation for the :Q# was incorrect when it stated that the mount returns 1# when the motors have stopped. It does actually not return anything. So the driver you are using is waiting for an answer that never comes and that is likely blocking any further commands. I'm not sure who maintains the Indi driver as it is not an official driver.

[ClutchplateDude]

I have contacted the author and made him aware of this issue.

[c3n]

I just tried (without indi or anything, just plain serial com) and it crashes only when you've issued a movement command (like start tracking) before. so

:GVP# => "OpenAstroExplorer"
:Q# => no problem

but

:GVP# => "OpenAstroExplorer"
:MT1# => 1
:Q# => crashes

[ClutchplateDude]

I can't repro this, it works fine on my system.
Let's take a look at your system.

• What board?
• What firmware version?
• What add-ons?

[c3n]

It’s the open astro explorer board with the required esp32 and it happens with the current version. No additional plugins, no gps or levelling, just the stock AutoPA

[ClutchplateDude]

Still no repro on my ESP32 system. Can you post your local configuration file on our Discord (or here)?

[c3n]

The only thing I changed from the default OAE config from the wiki is the AZ speed. I also just removed the ESP32 from the board and tested it standalone, connected just to my computer. I compiled it with PIO in VSCode using the "oaeboardv1" setting. Same result :-(

/////////////////////////////////////////////////////////////////////////////////////////////////////////
// This configuration file was generated by the OAT Configurator at https://config.openastrotech.com for
// firmware Latest Official (V1.10.1).
// Save this as Configuration_local.hpp in the folder where you placed the firmware code.
// Unique ConfigKey: FW:O,HS:N,BD:M21,RS:N9,RD:TU,RAP1500:O90:S2:T128:,RMSA3000:V2800:,RT:1,DS:N9,DD:TU,DAP1500:O90:S2:T256:,DMSA3000:V2800:,DT:1,DY:NO,GP:N,LM:N,FC:N,AV:2,ZS:N9,ZD:TU,AP:N,ZAP900:O90:H10:,AH:N

// We live in the Northern Hemisphere
#define NORTHERN_HEMISPHERE 1

// We are using the ESP32 board 

#if defined(BOARD) && BOARD != BOARD_OAE_V1 
#error Selected PIO environment does not match this configuration 
#else 
#define BOARD BOARD_OAE_V1 
#endif 


////////////////////////////////
// RA Stepper configuration 
// See supported stepper values. Change according to the steppers you are using
// Using the NEMA 17, 0.9°/step stepper for RA
#define RA_STEPPER_TYPE STEPPER_TYPE_ENABLE

// Using the TMC2209-UART driver for RA stepper motor
#define RA_DRIVER_TYPE DRIVER_TYPE_TMC2209_UART

// Define some RA stepper motor settings
#define RA_MOTOR_CURRENT_RATING       1200 // mA
#define RA_OPERATING_CURRENT_SETTING  80 // %
#define RA_SLEW_MICROSTEPPING         4
#define RA_TRACKING_MICROSTEPPING     256

#define TRACK_ON_BOOT 0

// TMC2209 Stealth Mode (spreadCycle) - When set to 0, tracking is more precise, but noisy (high-pitched sound). When set to 1, they are silent.
#define RA_UART_STEALTH_MODE          0

// Is it going the wrong way?
#define RA_INVERT_DIR  0

// Define some RA stepper motor settings

#define RA_SLEWING_SPEED_DEGREE 3
#define RA_ACCEL_SecToFullSpeed 1  // Seconds to reach full slewing speed
//#define RA_STEPPER_SPEED 4000 

#define RA_STEPPER_SPEED RA_SLEWING_SPEED_DEGREE * (RA_STEPPER_SPR * RA_SLEW_MICROSTEPPING * (RA_WHEEL_CIRCUMFERENCE / (RA_PULLEY_TEETH * GT2_BELT_PITCH)) / 360)  //3525
#define RA_STEPPER_ACCELERATION (RA_STEPPER_SPEED / RA_ACCEL_SecToFullSpeed)

// (RA_STEPPER_SPR * RA_SLEW_MICROSTEPPING *RA_TRANSMISSION ) 360 * deg

// Using the 16 tooth gear (recommended) for RA belt
#define RA_PULLEY_TEETH 16

////////////////////////////////
// DEC Stepper configuration 
// See supported stepper values. Change according to the steppers you are using
// Using the NEMA 17, 0.9°/step stepper for DEC
#define DEC_STEPPER_TYPE STEPPER_TYPE_ENABLE

// Using the TMC2209-UART driver for DEC stepper
#define DEC_DRIVER_TYPE DRIVER_TYPE_TMC2209_UART

// Define some DEC stepper motor settings
#define DEC_MOTOR_CURRENT_RATING       2000 // mA
#define DEC_OPERATING_CURRENT_SETTING  60 // %
#define DEC_SLEW_MICROSTEPPING         32
#define DEC_GUIDE_MICROSTEPPING        256

// TMC2209 Stealth Mode (spreadCycle) - When set to 0, tracking is more precise, but noisy (high-pitched sound). When set to 1, they are silent.
#define DEC_UART_STEALTH_MODE          1

// Is it going the wrong way?
#define DEC_INVERT_DIR  0

#define DEC_SLEWING_SPEED_DEGREE RA_SLEWING_SPEED_DEGREE
#define DEC_ACCEL_SecToFullSpeed RA_ACCEL_SecToFullSpeed

#define DEC_STEPPER_SPEED DEC_SLEWING_SPEED_DEGREE * (DEC_STEPPER_SPR * DEC_SLEW_MICROSTEPPING / 360 ) // * (DEC_WHEEL_CIRCUMFERENCE / (DEC_PULLEY_TEETH * GT2_BELT_PITCH)) / 360)
#define DEC_STEPPER_ACCELERATION (DEC_STEPPER_SPEED / DEC_ACCEL_SecToFullSpeed)

////////////////////////////////
// Display configuration 
// Define the type of display we are using. Currently: No display
#define DISPLAY_TYPE DISPLAY_TYPE_NONE

////////////////////////////////
// GPS Addon configuration 
// Define whether we have the GPS addon or not. Currently: No GPS
#define USE_GPS 0


// Using the NEMA 17, 0.9°/step stepper for AZ
#define AZ_STEPPER_TYPE STEPPER_TYPE_ENABLED
#define ALT_STEPPER_TYPE STEPPER_TYPE_ENABLED

// Using the TMC2209-UART driver for AZ stepper motor
#define AZ_DRIVER_TYPE DRIVER_TYPE_A4988_GENERIC 
#define ALT_DRIVER_TYPE DRIVER_TYPE_A4988_GENERIC 

////////////////////////////////
// Digital Level Addon configuration 
// Define whether we have the Digital Level or not. Currently: No Digital Level
#define USE_GYRO_LEVEL 0

#define WIFI_ENABLED 0 

///////////////////////
// Debug settings
#define DEBUG_LEVEL (DEBUG_NONE)
//#define DEBUG_LEVEL (DEBUG_STEPPERS)
//#define configCHECK_FOR_STACK_OVERFLOW 1

///////////////////////
// OAE specific defines

#define OAE

#define DEC_PULLEY_TEETH 1

#define RA_STEPPER_SPR (400 * 9)   // change to (200 * 9) for 1.8° stepper
#define DEC_STEPPER_SPR (200 * 50 * 4.5f)  // change "200" to "400" for 0.9° stepper, change "50" to different value depending on worm gear used


// AZ 
#define AZ_STEPPER_SPR 2048 // 28BYJ-48
#define AZ_MICROSTEPPING 1
#define AZ_STEPPER_SPEED 800
#define AZ_STEPPER_ACCELERATION 3000

// ALT
#define ALT_STEPPER_SPR 2048    // 28BYJ-48
#define ALT_MICROSTEPPING 1
#define ALT_STEPPER_SPEED 800
#define ALT_STEPPER_ACCELERATION 3000

[ClutchplateDude]

Will look at this shortly. Can you change the line
#define DEBUG_LEVEL (DEBUG_NONE)
to
#define DEBUG_LEVEL (DEBUG_ANY)
and then recompile and reflash. The connect with a terminal and reboot the board. Let me know what the last few lines are in the terminal.

[c3n]

Is there something else I need to do besides that? Because as soon as i enable DEBUG_ANY or just DEBUG_INFO the serial port doesn't respond at all. When I go back to DEBUG_NONE I don't have any problems (also I needed to disable build cache because changes to configuration local file isn't respected I need to do a clean each time and build cache ignores clean...). I also tried increasing the baud rate but that didn't help either.

btw: I tried the oae-fw branch and HEAD

also: I just tried a different esp32 board (which doesn't fit into the board) and got exactly the same results

[Ekrynox]

Hello,
One time when I tried the debug mode, I got the same issue.
If I recall correctly, you may need to add "#define DEBUG_SERIAL_PORT Serial".

And maybe "#define DEBUG_SEPARATE_SERIAL 1", but for this one, I'm not so sure.

[c3n]

yeah, that worked. apparently it's not really crashing, it's just not receiving any serial data anymore after ":Q#". here's the log with a :Q# that didn't crash (because tracking was off) followed by :MT1# and :Q# with crash:

[37239]{855}ms:313708B: [MOUNT]: Status -> Parked,------,0,0,0,090008,+900000,,
[38364]{1125}ms:313756B: [SERIAL]: ReceivedCommand(2 chars): [:Q]
[38399]{35}ms:313756B: [MEADE]: Received command   ':Q'
[38429]{30}ms:313756B: [MEADE]: Processing command ':Q'
[38459]{30}ms:313756B: [STEPPERS]: stopSlewing: TRK stepper stop()
[38494]{35}ms:313756B: [STEPPERS]: stopSlewing: DEC stepper stop()
[38529]{35}ms:313756B: [STEPPERS]: stopSlewing: RA stepper stop()
[39240]{711}ms:313708B: [MOUNT]: Status -> Parked,------,0,0,0,090008,+900000,,
[41241]{2001}ms:313708B: [MOUNT]: Status -> Parked,------,0,0,0,090008,+900000,,
[41430]{189}ms:313740B: [MOUNT]: checkRALimit: homeRA: 9.0022 (ZeroPos: 9.0022 + TrkHrs: 0.0000)
[41481]{51}ms:313740B: [MOUNT]: checkRALimit: degreePosDec: 0.0000 , RA hourpos : 9.0022)
[41528]{47}ms:313740B: [MOUNT]: checkRALimit: RA hourPos (adjusted): 9.0022
[41568]{40}ms:313740B: [MOUNT]: checkRALimit: DeltaRA: 0.0000 (home:9.0022 - hour:9.0022)
[41616]{48}ms:313740B: [MOUNT]: checkRALimit: deltaRA: 0.0000 => Check against 6.7500
[43242]{1626}ms:313708B: [MOUNT]: Status -> Parked,------,0,0,0,090008,+900000,,
[45243]{2001}ms:313708B: [MOUNT]: Status -> Parked,------,0,0,0,090008,+900000,,
[46662]{1419}ms:313740B: [MOUNT]: checkRALimit: homeRA: 9.0022 (ZeroPos: 9.0022 + TrkHrs: 0.0000)
[46713]{51}ms:313740B: [MOUNT]: checkRALimit: degreePosDec: 0.0000 , RA hourpos : 9.0022)
[46761]{48}ms:313740B: [MOUNT]: checkRALimit: RA hourPos (adjusted): 9.0022
[46801]{40}ms:313740B: [MOUNT]: checkRALimit: DeltaRA: 0.0000 (home:9.0022 - hour:9.0022)
[46848]{47}ms:313740B: [MOUNT]: checkRALimit: deltaRA: 0.0000 => Check against 6.7500
[47244]{396}ms:313708B: [MOUNT]: Status -> Parked,------,0,0,0,090008,+900000,,
[48534]{1290}ms:313756B: [SERIAL]: ReceivedCommand(4 chars): [:MT1]
[48570]{36}ms:313756B: [MEADE]: Received command   ':MT1'
[48601]{31}ms:313756B: [MEADE]: Processing command ':MT1'
[48632]{31}ms:313756B: [MEADE]: Process Move command: [T1]
[48663]{31}ms:313724B: [STEPPERS]: startSlewing: Tracking: Switching RA driver to microsteps(256)
[48717]{54}ms:313772B: [SERIAL]: RepliedWith:  [1]
[49245]{528}ms:313708B: [MOUNT]: Status -> Tracking,--T---,0,0,125,090008,+900000,,
[51246]{2001}ms:313708B: [MOUNT]: Status -> Tracking,--T---,0,0,596,090008,+900000,,
[51894]{648}ms:313740B: [MOUNT]: checkRALimit: homeRA: 9.0031 (ZeroPos: 9.0022 + TrkHrs: 0.0009)
[51945]{51}ms:313740B: [MOUNT]: checkRALimit: degreePosDec: 0.0000 , RA hourpos : 9.0022)
[51992]{47}ms:313740B: [MOUNT]: checkRALimit: RA hourPos (adjusted): 9.0022
[52032]{40}ms:313740B: [MOUNT]: checkRALimit: DeltaRA: 0.0009 (home:9.0031 - hour:9.0022)
[52080]{48}ms:313740B: [MOUNT]: checkRALimit: deltaRA: 0.0009 => Check against 6.7500
[52901]{821}ms:313756B: [SERIAL]: ReceivedCommand(2 chars): [:Q]
[52936]{35}ms:313756B: [MEADE]: Received command   ':Q'
[52965]{29}ms:313756B: [MEADE]: Processing command ':Q'
[52995]{30}ms:313756B: [STEPPERS]: stopSlewing: TRK stepper stop()
[53031]{36}ms:313756B: [STEPPERS]: stopSlewing: DEC stepper stop()
[53066]{35}ms:313756B: [STEPPERS]: stopSlewing: RA stepper stop()
[53247]{181}ms:313708B: [MOUNT]: Status -> Parked,------,0,0,1009,090008,+900000,,
[55248]{2001}ms:313708B: [MOUNT]: Status -> Parked,------,0,0,1009,090008,+900000,,
[57126]{1878}ms:313740B: [MOUNT]: checkRALimit: homeRA: 9.0034 (ZeroPos: 9.0022 + TrkHrs: 0.0012)
[57177]{51}ms:313740B: [MOUNT]: checkRALimit: degreePosDec: 0.0000 , RA hourpos : 9.0022)
[57225]{48}ms:313740B: [MOUNT]: checkRALimit: RA hourPos (adjusted): 9.0022
[57265]{40}ms:313740B: [MOUNT]: checkRALimit: DeltaRA: 0.0012 (home:9.0034 - hour:9.0022)
[57312]{47}ms:313740B: [MOUNT]: checkRALimit: deltaRA: 0.0012 => Check against 6.7500
[57358]{46}ms:313708B: [MOUNT]: Status -> Parked,------,0,0,1009,090008,+900000,,

[c3n]

I added some more code and found out that in Mount.cpp waitUntilAllStopped() doesn't return because _stepperTRK->isRunning() is never false

[c3n]

ok i figured it out:
_stepperTRK is trying to decelerate but in interruptLoop it's only ever called with runSpeed which does not stop when reaching the target position. I changed the TRK/DEC/RA run code in Mount.cpp:2955 like so:

    if (_mountStatus & STATUS_SLEWING)
    {
        if (_mountStatus & STATUS_SLEWING_MANUAL)
        {
            _stepperDEC->runSpeed();
            _stepperRA->runSpeed();
        }
        else
        {
            _stepperDEC->run();
            _stepperRA->run();
        }
    }
    else if (_mountStatus & STATUS_TRACKING)
    {
        _stepperTRK->runSpeed();
    }
    else if (_stepperTRK->isRunning())
    {
        _stepperTRK->run();
    }

and now it works. I don't know if this bug is only present with esp32 or OAE or why you didn't have the same problem in your tests. Also I didn't test it beyond serial port monitoring stuff, so I'm not making a pull request unless you want me to

[ClutchplateDude]

Oh, man, nice. That's some good detective work :-), I changed the logic a bit and will merge it soon.