feat!: support HealthPulse integration

[jkuester]

This branch/PR contains the changes necessary for integrating with the HealthPulse AI app.

sampleImage

The sampleImage logic allows for storing a full resolution image returned from an external app when the returned sampleImage property is true. This is needed for HealthPulse to allow for properly recording images that may be useful in testing/validating the model. Normally images returned from external apps (e.g. the default camera app) are compressed before being sent to the CHT webapp to save on space/bandwidth. However, compressing these sample images would render them useless for testing/validating the AI model.

### duration

Returning the duration value to the CHT form is a workaround to be able to validate/measure time spent in the external app without having to actually update the CHT instance. The proper fix for these changes should be made in cht-core to resolve medic/cht-core#10217.

Primitive list serialization

This allows for serializing an array of primitive values as a space-delimited string value (instead of a JSON array). Sending the values as a string allows us to avoid using a repeat in the form to unpack the array data into the form model. Repeats in Enketo are pretty sketchy, especially for complex workflows.

This change is a great improvement over the existing functionality, but I don't think we can implement it in cht-android without it being a breaking change. Instead, I have just logged medic/cht-core#10508 to implement this in cht-core where it can be made passively.

[jkuester]

This tweak allows us to save the returned image uncompressed when samepleImage is true. This is needed to be able to return the images marked for sampling to Audere (they cannot use compressed versions of the images when training the model).

Note that the difference between 100 quality (full resolution) and 75 quality (compressed) may seem minor, but in my unscientific measurements I was seeing the ultimate size of the png file for the full resolution image to be ~5x greater than the compressed image. For example, when testing with my phone with a 16MP camera, the compressed image size (of the png attachment uploaded to Couch) was 1.4MB, but the uncompressed image size (saved when sampleImage: true) was 5.8MB.

[Benmuiruri]

Hi Josh, a couple of comments.

• I saw a new Healthpulse was released with the changes, I'll be on the look out for that.
• a very common response I am seeing is

result: {
    "interpretation": {
      "concerns": [
        "RDT_NOT_FOUND"
      ],
      "detected_rdts": []
    }
  }

But in the form the output has blanks, will we have the concerns being fed back to use the value to ask the CHW to retake the photo ? (the INTENTion is to prompt them at most two times to retake the photo if the result is not definitively positive or negative)

[jkuester]

But in the form the output has blanks, will we have the concerns being fed back to use the value to ask the CHW to retake the photo ? (the INTENTion is to prompt them at most two times to retake the photo if the result is not definitively positive or negative)

🤔 @Benmuiruri I am not sure I understand the question here. 😓 What do you mean "the output has blanks"? The immediate answer is that, yes, we should be using the concerns (or now the globalConcerns) to trigger prompting the user to try again. Though, the exact form config to make this happen is not something I have worked through....

[Benmuiruri]

@jkuester I was describing what I see in the android_app_launcher form you created

• If there AI correctly inteprets the image as either positive or negative, the output you have in cell 51 of the excel sheet displays the response.
• However, if the AI is not able to intepret the image, the output is blank.

Generally, you did respond to my suggestion that we should use the concerns to prompt back to the user to retry.

Will you be making changes to this draft PR to support that and also in response to the new HealthPulse app ?

[Benmuiruri]

The idea of the workflow is to prompt user to retake the photo with

The photo is not clear due to {{poor lighting}} {{too much blurr}} Please take another photo of the mRDT kit.

However, I am not sure, does healthpulse give back the specific warning, or it is generic ? that will influence what we display whether it's a specific warning or a general warning and prompt to retake the photo.

If the concern is one of the errors we also need to display the general warning and prompt to retake the photo.

[jkuester]

Okay, I just pushed an update here to remove the custom logic around concerns because it is no longer needed. You will need to make sure to update your form to load the top-level concerns as globalConcerns and the concerns nested in the detectedRdt group as concerns. Depending on which version of the HealthPulse app you have been testing with (and the structure of your form), this may resolve some of the unexpected behavior you had...

I think we have 3 distinct error cases, each with different levels of severity:

1. Fatal Error

It is possible for the intent to just totally fail. This could happen if:

• The HealthPulse app is missing
• The license key is invalid or not provided
• The HealthPulse app crashes (e.g. due to a bug).

This is never expected to happen during normal production usage. There are no good recovery scenarios for this (since it is likely the user will not be able to fix the problem.) Also, there is not really any easy way to notify the user in the cht-android app that something went wrong. Instead the user just ends up back in the form page with no output data at all.

2. Global Concern

This is when the intent succeeds, but the RDT classification completely fails (e.g. no RDT was found in the picture). The globalConcerns repeat will be populated with one (or more?) error keys:

• RDT_NOT_FOUND - The AI could not confidently detect an RDT in the image and will not return an interpretation. If this error is returned, always ask the user for a retake.
  • Optional: You can display an error message that asks the user to ensure there is an RDT in the image.
• MULTI_RDT - Multiple RDTs were detected in the image.
  • Optional: You can display an error message that asks the user to ensure that only one RDT is present in the image.

In this case, we do not expect to get any detectedRdt data back. So, there will be no recorded classification.

3. Detected RDT Concerns

This is when the HealthPulse app was able to scan an RDT, but there were problems with the scan that might impact the recorded classification. The detectedRdt.concerns repeat will be populated with one (or more?) error keys:

Non-blocking:

• WARNING - Image quality issues were detected (e.g. poor lighting or too much blur) which could impact AI interpretation accuracy. If present, we recommend asking the user for a retake.
• WRONG_RDT_TYPE - The RDT detected in the image does not match the expected RDT type passed in. An interpretation will be returned for the detected RDT.
  • Optional: Ask the user to retake the image and/or ask the user to confirm the RDT type that they are using.

Blocking:

• RDT_NOT_INTERPRETABLE - The AI could not confidently find an interpretable result area in the image and will not return an interpretation. If this error is returned, always ask the user for a retake.
  • Optional: You can display an error message that asks the user to ensure the RDT is clear and visible in the photo and remind them to follow the photo tips.
• UNSUPPORTED_RDT_TYPE - The RDT found in the image is not supported by the license (non-standard and/or not in the list of supported RDTs). No interpretation will be returned.
• Ask the user to make sure that they are photographing the RDT they selected and to retake the image.

[Benmuiruri]

Hey @jkuester one of the requirements for the workflow is after 3 failed classification attempts hide the healthpulse section end malaria workflow and use the CHP result to manage the child.

I attempted to track in the form how many attempts it has been. I came up with this to only show concerns under these conditions

(count-non-empty(${concern_key}) > 0) and (${attempt_count} < 3) and ${classification} = “”

My idea was to get back the count for attempt_count from cht-android.
ChtExternalAppHandler.txt

Thoughts? Is there another approach possible without tracking the attempt_count in cht-android

[jkuester]

@Benmuiruri I think hacking the attempt_count in here like you have suggested would work, but it feels a bit heavy-handed (needing to use SharedPrefs to store the value smells like we are trying to solve the problem at the wrong place...). Perhaps a "better" place to add support for this functionality would be down in the cht-core code. However, this is not very helpful for us right now since we cannot take a cht-core update for this project.... 😓

However, the good news is that I think we can work around this with no code changes at all! 🤞 I did some testing with the latest version of the android_app_launcher.xlsx form and was able to demonstrate how we can use the requestId as a counter to track the number of times the HealthPulse app is launched. Basically I am just calculating the input requestId like this: 1 + if(${out_request_id} = “”, 0, ${out_request_id}). Then, once we get the intent back from HealthPulse, the out_request_id value will be updated to match the value given in the input. This, in turn, will trigger the calculated input requestId value to be incremented.

With this approach, I think you could could base your re-try logic off of out_request_id (similar to what you were doing for attempt_count)...

[jkuester]

@Benmuiruri @KicaRonaldOkello I have added some logic here based on the issues we are seeing trying to deal with repeat in the android-app-outputs data. Repeats are a particularly weak area in Enketo forms and are prone to bugs.

However, when it comes to storing lists of values, repeats are not our only option. (IMHO they are actually not the best option at all.) In a form data model, it is much easier to represent a list of values as a "node-set" (aka a space-delimited string). This is how values from a select_multiple are stored already. One big advantage of a node-set (besides just not being a repeat with a single field) is that all the ODK select question operations can be used to interact with the node-set data.

So, with this change, globalConcerns and concerns values are no longer being serialized into the form model as repeats, but instead can just be accepted as normal text fields. These text fields will contain space-separated values. See my know example form for what this looks like in the xlsxform, but TLDR is that you no longer have to use repeats...

---

Let me know what you guys think of this or if you have any additional recommendations. 👍

[jkuester]

Okay, for the sake of simplicity I created a new health_pulse_release branch (at the same commit as master) and I have changed this PR to target that branch. I am going to "Squash and merge" this PR into that branch. This will let us keep this health_pulse branch/PR for a historical record, etc AND we will have the health_pulse_release branch with a single commit containing only the necessary changes. This branch can easily be re-based onto future cht-android releases. 👍