CarbonData API 404 error

[mishagrol]

Dear team of CarbonPlan, first of all, thanks for your project!

I tried to reproduce some results and understand database structure, but met some problems:

Example 1.

I tried to run FIA downloading - https://carbonplan.org/data

import os
from intake import open_catalog

os.environ['CARBONPLAN_DATA'] = 'https://carbonplan.blob.core.windows.net/carbonplan-data'
cat = open_catalog(
    "https://carbonplan.org/data/api/intake/fia.yaml"
)
cat["raw_table"](name="plot").to_dask()

But received this error:

ClientResponseError: 404, message='Not Found', url=URL('https://carbonplan.org/data/api/intake/fia.yaml')

Is there any full manual how to deal with data in carbonplan? Thanks!

[jhamman]

Hi @mishagrol - Something seems to have changed upstream with how intake/fsspec handle directory style Parquet datasets over http. We can look into this but in the meantime, this seems to work for me:

pd.read_parquet('gs://carbonplan-data/raw/fia/plot.parquet')

Is there any full manual how to deal with data in carbonplan? Thanks!

Not really. We haven't been working directly on this project much lately. Feel free to open issues if you have more questions though.

[mishagrol]

@jhamman

Thank you for your reply! I tried to reproduce your code in Google Colab,

import pandas as pd
df = pd.read_parquet('gs://carbonplan-data/raw/fia/plot.parquet')

But again I met a problem with the availability of storage

_request out of retries on exception: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7fb53a0d62d0>)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/credentials.py", line 111, in refresh
    self._retrieve_info(request)
  File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/credentials.py", line 88, in _retrieve_info
    request, service_account=self._service_account_email
  File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/_metadata.py", line 234, in get_service_account_info
    return get(request, path, params={"recursive": "true"})
  File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/_metadata.py", line 187, in get
    response,
google.auth.exceptions.TransportError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7fb53a0d62d0>)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/gcsfs/retry.py", line 115, in retry_request
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/gcsfs/core.py", line 368, in _request
    headers=self._get_headers(headers),
  File "/usr/local/lib/python3.7/dist-packages/gcsfs/core.py", line 347, in _get_headers
    self.credentials.apply(out)
  File "/usr/local/lib/python3.7/dist-packages/gcsfs/credentials.py", line 169, in apply
    self.maybe_refresh()
  File "/usr/local/lib/python3.7/dist-packages/gcsfs/credentials.py", line 164, in maybe_refresh
    self.credentials.refresh(req)
  File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/credentials.py", line 117, in refresh
    six.raise_from(new_exc, caught_exc)
  File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7fb53a0d62d0>)

---------------------------------------------------------------------------

TransportError                            Traceback (most recent call last)

[/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/credentials.py](https://localhost:8080/#) in refresh(self, request)
    110         try:
--> 111             self._retrieve_info(request)
    112             self.token, self.expiry = _metadata.get_service_account_token(

25 frames

TransportError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7fb53a0d62d0>)


The above exception was the direct cause of the following exception:

RefreshError                              Traceback (most recent call last)

<decorator-gen-116> in _request(self, method, path, headers, json, data, *args, **kwargs)

/usr/local/lib/python3.7/dist-packages/six.py in raise_from(value, from_value)

RefreshError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7fb53a0d62d0>)