Java SDK is unable to get shared key values that were put using the Dart SDK

[akafredperry]

Describe the bug

AtClientImpl uses an empty Initialisation Vector when it encrypts and decrypts "shared keys" (and self keys). So a get works from Java if the put was performed by Java it will not work if the put was performed by the Dart SDK which will use a random IV (which it adds to the key meta data). Correspondingly a get from Dart SDK will work because it supports empty IV on decrypt but a put once the Java SDK has created the shared_key fails.

Steps to reproduce

Use the attached feature file, to enroll 2 APKAMS for the srie and device2 atsigns in the virtual env
This creates 2 shared keys and then repeatedly asserts that it can get the shared keys and the value match what was put
Use the demo dart code in at_demos/demos/at_getting_started APKAMs to get the values for these shared keys...

dart run at_key_get.dart -d vip.ve.atsign.zone -a @srie -n ns -k @srie-app1-device1.atKeys -o @device2
dart run at_key_get.dart -d vip.ve.atsign.zone -a @device2 -n ns -k @device2-app1-device1.atKeys -o @srie

and to attempt to put new values (which should cause the feature file to fail in the expected way

 dart run at_key_put.dart -d vip.ve.atsign.zone -a @device2 -n ns -k @device2-app1-device1.atKeys -o @srie -m xxxxx

Expected behavior

Putting and Getting using the Java and Dart SDK in both "directions" should work

Screenshots

No response

Smartphones

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Were you using an atApplication when the bug was found?

No response

Additional context

No response

[akafredperry]

unable to attach as file, this is the feature file contents for testing

Feature: Hack

  Scenario: :
    Given root server endpoint is vip.ve.atsign.zone:64
    And root server is running
    And atsign keys path is at_demo_data package lib/assets/atkeys
    And atsign keys suffix is .atKeys
    And verbose logging is on
    And key ttl is 1 DAY

    And @srie Activate.onboard with SrieKeys._cramKey from at_demo_apkam_keys.dart in at_demo_data package
    And @srie Activate.otp generates an OTP
    And @srie Activate.enroll for app app1 and device device1 with last OTP and following namespaces
      | Namespace | Access Control |
      | ns        | rw             |
    And @srie Activate.approve for last enrollment
    And AtClient with keys @srie-app1-device1.atKeys for @srie completes enrollment

    And @device2 Activate.onboard with Device2Keys._cramKey from at_demo_apkam_keys.dart in at_demo_data package
    And @device2 Activate.otp generates an OTP
    And @device2 Activate.enroll for app app1 and device device1 with last OTP and following namespaces
      | Namespace | Access Control |
      | ns        | rw             |
    And @device2 Activate.approve for last enrollment
    And AtClient with keys @device2-app1-device1.atKeys for @device2 completes enrollment

    When @srie AtClient.put for SharedKey message.ns shared with @device2 and value "hello from srie"
    And @device2 AtClient.put for SharedKey message.ns shared with @srie and value "hello from device2"

    Then @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"
    And pause for 30 SECONDS
    And @device2 AtClient.get for SharedKey message.ns shared by @srie returns value that matches "hello from srie"
    And @srie AtClient.get for SharedKey message.ns shared by @device2 returns value that matches "hello from device2"

[akafredperry]

There are currently 3 usage scenarios where empty IV is used.

1. AtKeys file read / write - this decrypts / encrypts the pkam and encrypt rsa key pairs with the self key and an empty IV. The fact that I am able to cut the atKeys in java and successfully use them with Dart demo utilities implies that this behaviour is consistent with Dart and so needs to be retained. Correct?
2. put/get for Self Key. this code encrypts / decrypts the value with an empty IV. From what I observe this is inconsistent with the Dart SDK and needs to change. This means adding ivNonce to the Metadata.java, adding support to the UpdateVerbBuilder (for the put). The get already generates an llookup with operation set to all and parses the llookup into a LookupResponse which contains the metadata.
3. put/get for Shared Key. this code encrypts / decrypts the value with an empty IV. Again from what I observe this is inconsistent with the Dart SDK and needs to change and so similar changes as were made for SelfKey. However the 2 get impls (for shared by / shared with) do NOT set operation to all in the llookup command so this will also be required in order to decrypt.

@gkc did I diagnose correctly? Am I correct in thinking that support for empty IV needs to stay because of scenario 1? But maybe removed from EncryptionUtil.java and pushed up into KeysUtil.java? Why do we encrypt the key pairs with a key that is present as plain text in the same file? Is the intention that human being might want to provide the IV contents as another layer of security i.e. something you know (iv) + something you have (at keys file).

Lastly @gkc is the way I am testing Dart / Java interoperability correct / the easiest way to do this?