Use aot-snapshot in pkg-standalone-dev

[ntkme]

As dart-lang/sdk#39973 has been addressed and can be closed, this PR uses dart compile aot-snapshot --enable-asserts for faster dev snapshot experience.

A few clarifications:

1. For kernel snapshot --enable-asserts during compilation does NOT change the behavior of generated snapshot, dart --enable-asserts must be used at the runtime. dart compile kernel --help does not show it as a valid option, it's "valid" only because it's an global option to the dart command for running .dart source file.
2. For aot-snapshot --enable-asserts during compilation does change the behavior of generated snapshot, dartaotruntime --enable-asserts does NOT affect assertion behavior which has been determined during compilation.
3. For both kernel and aot-snapshot, -D variables are determined during compilation, specify variables during runtime does not change any behavior.

Minimum required SDK version is bumped up to 3.8.0.

[ntkme]

@nex3 A few things I would like to discuss:

1. If we bump the minimum dart sdk version to 3.8.0, we can get rid of the "dev kernel snapshot" - and only keep "dev aot-snapshot", "release aot-snapshot", and "release kernel snapshot" (for other platforms) flavors as all platforms supported by 3.8.0 have aot. I kept the "dev kernel snapshot" for now as I don't want to bump dart version requirement too aggressively.
2. In 3.9.0 we have support of cross compile aot-snapshot to any linux target from any 64-bit source. With the current state of cross compiling support, it will increases the code complexity by a lot. As it's not super difficult to do QEMU native compilation on GitHub Actions as we do today in dart-sass. I think this is a low priority, and nice to have feature. Maybe wait for more target platforms to be supported and then we can consider adding this.
3. I don't really like how the packaged snapshots are named today. I understand that the idea is to have a consistent path pattern, however, I think this has become utterly confusing for users to identify what exactly is being built. Thus, I would like to propose to move the naming convention to be fully aligned with current Dart SDK. - e.g. sass.dill (kernel), sass.jit (jit-snapshot), sass.aot (aot-snapshot) - to make it more clear what kind of product is being built.

[nex3]

1. If we bump the minimum dart sdk version to 3.8.0, we can get rid of the "dev kernel snapshot" - and only keep "dev aot-snapshot", "release aot-snapshot", and "release kernel snapshot" (for other platforms) flavors as all platforms supported by 3.8.0 have aot. I kept the "dev kernel snapshot" for now as I don't want to bump dart version requirement too aggressively.

I'm fine bumping the minimum Dart version. Pub has good SDK version solving, so this won't break users who are locked to older SDKs, and the fact that this package is targeted specifically at people who are generally distributing to non-Dart users makes the marginal value of supporting older SDKs lower.

2. In 3.9.0 we have support of cross compile aot-snapshot to any linux target from any 64-bit source. With the current state of cross compiling support, it will increases the code complexity by a lot. As it's not super difficult to do QEMU native compilation on GitHub Actions as we do today in dart-sass. I think this is a low priority, and nice to have feature. Maybe wait for more target platforms to be supported and then we can consider adding this.

I'm assuming the code complexity increase here would be from having separate code paths for cross-compiling Linux but not other operating systems? I agree that this is low-priority. If we get a request for it, maybe we can bump it up, but as it stands I'm happy to keep using QEMU and suggesting others do the same for the time being.

3. I don't really like how the packaged snapshots are named today. I understand that the idea is to have a consistent path pattern, however, I think this has become utterly confusing for users to identify what exactly is being built. Thus, I would like to propose to move the naming convention to be fully aligned with current Dart SDK. - e.g. sass.dill (kernel), sass.jit (jit-snapshot), sass.aot (aot-snapshot) - to make it more clear what kind of product is being built.

I have mixed feelings about this. I think generally it's more important to provide something that makes sense to the end users who are running the applications, and who have no idea what .dill, .jit, or .aot mean—the intention of the single .snapshot file is to clarify that. Maybe a better way to do it would be to call it ${executable}-snapshot.${ext}?

[ntkme]

I'm assuming the code complexity increase here would be from having separate code paths for cross-compiling Linux but not other operating systems?

The cross compile works as the dart compile command downloading the SDK Cache, which consists of cross-compiler and dartaotruntime for the target platforms:

Dart SDK hard codes the download path from storage.googleapis.com, meaning on dart-musl/dart-android it will not be able to download the correct cross-compiler. However, these files can be download manually:

• https://storage.googleapis.com/dart-archive/channels/${channel.name}/${stage.name}/$version/$path
• https://github.com/dart-musl/dart/releases/download/$version/$path
• https://github.com/dart-android/dart/releases/download/$version/$path

The "path" part looks like:

• dartaotruntime_${target_os}_${target_arch}
• gen_snapshot_${host_os}_${host_arch}_${target_os}_${target_arch}

These files are downloaded to:

• $HOME/.dart/dartdev/sdk_cache/$version/$path

The biggest problem here is the dart-gnu and dart-musl literally have the conflicting filenames, because the Dart SDK itself cannot tell the difference. We can to download and overwrite those files as needed, but it introduces possible race conditions for concurrent compilation and may cause unintended side effect on which one becomes the default cross compile without using cli_pkg. I can think of a few solutions but non of them are perfect:

1. To not use dart compile aot-snapshot at all, and download and directly invoke gen_snapshot_*. Something like this: https://github.com/dart-lang/sdk/blob/c40f1f7dfe8ad0f6fef098ce5c43c2d8e152ccd7/pkg/dartdev/lib/src/commands/compile.dart#L662
2. Download sdk cache files to $TMPDIR/.dart/dartdev/sdk_cache/$version/$path, symlink $HOME/.pub-cache to $TMPDIR/.pub-cache, and then execute HOME="$TMPDIR" dart compile aot-snapshot. - This would trick dart compile to use a different location for SDK cache and thus avoiding the path conflict.

Maybe a better way to do it would be to call it ${executable}-snapshot.${ext}?

Looks good to me.

[ntkme]

I'm fine bumping the minimum Dart version.

I bumped it to 3.8.0 and updated the dev builds to always be aot-snapshot.

As for the snapshot naming, I decided to not change that for now in this PR. I realized that when cross-compiling gets supported at a later time, the aot-snapshot must have platform identifier in their grinder task and artifact name (at least for the intermediate artifacts on disk). So it's probably better to rename them later when we implement cross-compiling.

This PR is ready for review. In my local test, switching to AOT cuts the dart test time for dart-sass 1.90.0 down from 49 seconds to 22 seconds on my M1 MacBook Pro.

[nex3]

The cross compile works as the dart compile command downloading the SDK Cache, which consists of cross-compiler and dartaotruntime for the target platforms:

Dart SDK hard codes the download path from storage.googleapis.com, meaning on dart-musl/dart-android it will not be able to download the correct cross-compiler. However, these files can be download manually:

• https://storage.googleapis.com/dart-archive/channels/${channel.name}/${stage.name}/$version/$path
• https://github.com/dart-musl/dart/releases/download/$version/$path
• https://github.com/dart-android/dart/releases/download/$version/$path

The "path" part looks like:

• dartaotruntime_${target_os}_${target_arch}
• gen_snapshot_${host_os}_${arch_arch}_${target_os}_${target_arch}

These files are downloaded to:

• $HOME/.dart/dartdev/sdk_cache/$version/$path

The biggest problem here is the dart-gnu and dart-musl literally have the conflicting filenames, because the Dart SDK itself cannot tell the difference. We can to download and overwrite those files as needed, but it introduces possible race conditions for concurrent compilation and may cause unintended side effect on which one becomes the default cross compile without using cli_pkg. I can think of a few solutions but non of them are perfect:

1. To not use dart compile aot-snapshot at all, and download and directly invoke gen_snapshot_*. Something like this: https://github.com/dart-lang/sdk/blob/c40f1f7dfe8ad0f6fef098ce5c43c2d8e152ccd7/pkg/dartdev/lib/src/commands/compile.dart#L662
2. Download sdk cache files to $TMPDIR/.dart/dartdev/sdk_cache/$version/$path, symlink $HOME/.pub-cache to $TMPDIR/.pub-cache, and then execute HOME="$TMPDIR" dart compile aot-snapshot. - This would trick dart compile to use a different location for SDK cache and thus avoiding the path conflict.

Yeah, that's not worth the hassle right now. Can you file an issue against the Dart SDK requesting a CLI flag to pass in a specific path for the SDK cache?

[nex3]

Since we're no longer supporting Dart 3.7.x, let's also simplify CliPlatform.useNative to remove the IA32 check (since there are no longer any Dart SDKs that support IA32).

[ntkme]

The ia32 Abi classes are not removed from SDK yet, because dart is technically still support the kernel/JIT runtime on ia32. As far as I understand, it's for Google/Flutter's internal use and no ia32 sdk/binary is published for external users. Therefore we still need to keep the logic to filter out ia32.

[nex3]

But we shouldn't need the && !arch.isIA32 check in CliPlatform.useNative, since there aren't actually any IA32 SDKs and so we won't have any IA32 binaries.

[ntkme]

Thanks. Updated.

[ntkme]

Can you file an issue against the Dart SDK requesting a CLI flag to pass in a specific path for the SDK cache?

I did a bit more testing and found that if we only support aot-snapshot then we don't need any sdk support, but if we want to support exe as well then we do need a change: dart-lang/sdk#61374