Data Export and Import Questions

[jerrac]

Background

I have a standalone Greptime 0.17.2 instance running in Docker via Docker Compose. There area few other services on that vm, and I have caused some of them to crash when hitting Greptime with too large of a query. (I'm still figuring out how to not do that...)

Due to that, I thought moving Greptime over to a vm on its own would help ensure I don't take other things down.

At the same time I thought I'd set up data storage on an S3 compatible target. I think if I ever need to cluster, having the data on S3 already would help.

My problem is that the data export is really really slow. As in my metrics table is something like 27GB, and my export has only created 19GB worth of data in over 20 hours. The docs mention watching system resources, but not really anything on how to adjust things based on what you see.

I tried exporting a second table at the same time, but that caused the docker container to run out of resources and get killed.

This is inside our data center, so networking should not be an issue. I think we're on 10G at the slowest.

My target storage is an nfs mount with lots of space. I initially started exporting the data the wrong way and it was getting sent to the local container storage, and that export was moving just as slow. So the speed issue should not be caused by NFS, I hope.

I do have the docker service limited with mem_limit: "8GB" and cpus: 4.0 settings. Which I'm guessing might be where the problem is? Specifically docker stats tells me that the container is using well over 200% cpu, and is using up almost all of it's available RAM.

Questions:

• Would increasing the memory and cpu limits help?
• Can the data exports be interrupted and then resumed, or do you have to start over from scratch? The current export was initially interrupted, but when I restarted it, I couldn't tell if it was overwriting the existing data, or if it had somehow known to resume from where it left off.
• Is there another way to get the data transferred? As in could I start using my new instance of Greptime, and at the same time get the old data pushed into it little by little?
• Any other tips? Places I probably misunderstood something?

Thanks!

[fengjiachun]

Thank you for the detailed context and questions. Here are a few suggestions that may help:

1. Prefer splitting the export by time ranges first
   For a 27 GB metrics table, doing one huge “all history” export is usually the slowest and most fragile option. The export tool supports time-range based export, so it’s better to break the data into several windows (for example: per month, or even per week/day if needed) and run multiple exports:

# Example: export a single database within a time range
greptime cli data export \ 
  --addr <your_greptime_addr> \ 
  --output-dir /path/to/backup/metrics-2024-01 \ 
  --database <your_database_name> \ 
  --start-time "2024-01-01 00:00:00" \ 
  --end-time "2024-01-31 23:59:59"

You can repeat this for different time windows until the full history is covered. This has two benefits:

Each export job is smaller and less likely to hit resource limits or fail halfway.

If one job fails, you only need to re-run that specific time range, not the whole 27 GB again.

2. About CPU / memory limits

Increasing the memory limit is the most effective way to avoid the container being OOM-killed during export.
The export process needs additional working memory for scanning data, buffering rows, encoding, and writing files. With only 8 GB available, a large table export (such as 27 GB of metrics) can easily exhaust the container’s memory and get terminated by Docker.

If possible, raising the memory limit (for example to 16–32 GB, depending on your VM capacity) will significantly reduce the risk of OOM and make the export more stable.
CPU limits can also be raised, but memory is the more critical factor for preventing crashes.

As in could I start using my new instance of Greptime, and at the same time get the old data pushed into it little by little?

Yes — as long as your new writes do not use timestamps that overlap with the historical data you are importing, you can absolutely start writing to the new Greptime instance while gradually importing the old data.

This approach is safe and commonly used when migrating from one instance to another without downtime. The only thing to avoid is writing new data that falls into the same timestamp ranges as the historical batches you are importing.

If your system always writes strictly increasing timestamps, then you can migrate history “little by little” without any interruption.

[jerrac]

Thanks! Doing small exports plus just pointing at the new instance for new data makes a lot of sense. I just wasn't sure I could do that exports/imports piecemeal. I'll get started on that next week, other stuff came up today.

[jerrac]

I just ran into the issue described here: #7442

And that got me wondering, for importing data, is the value of the --import-dir expected to be a directory local to the server you are importing data to?