pull the data

[haozidede]

Hi cwerner,

I found this project interesting.
And I have already installed the required enviroment. However, when I use 'dvc pull createdataset' to download the data, it goes wrong. "ERROR: unexpected error - The config profile (ifu-s3) could not be found".
Could you please share me the data? I would appreciate it.

yours sincely Wei
Best wishes

[cwerner]

Hi Wei 👋

Sorry about that. The reason you see this is that we use a private on-premise S3 storage to host our data by default.

I added the internal config settings for a co-worker, however you are missing the required access keys (that make up the AWS profile settings "ifu-s3") for them to work.

.dvc/config:

[core]
    remote = ifu-s3
[cache]
    dir = /glacier2/dvc-cache
['remote "ifu-s3"']
    url = s3://deadtrees/dvc
    endpointurl = https://s3.imk-ifu.kit.edu:8082
    profile = ifu-s3

However, the source data is freely accessible. So if you download the raw data, do an initial preprocessing step 0, and then run the full dvc preprocessing chain (using dvc repro ...you should be good to go.

If you check the README.md file under scripts/ you see the source for the data and the required steps to get from raw data to the beginning of the dvc pipeline.

Let me know if this works for you (you would need some smallish extra vector data but that I could provide you via email).

Cheers,
C

[haozidede]

Great thanks for your reply. Following your advice, I've pulled down the "2017: RGB Image, NIR Image".
However, when I excute
“gdal_translate -of GTiff -co "COMPRESS=LZW" -co "PREDICTOR=2" -co "TILED=YES" -co "BIGTIFF=YES" -co "ALPHA=NO" -colorinterp_1 "red" -colorinterp_2 "green" -colorinterp_3 "blue" allbands.vrt ortho_ms_${YEAR}.tif”. It goes wrong,and returned as:
"
Input file size is 295000, 415000
0ERROR 1: Marker is not compliant with its position

ERROR 1: opj_decode() failed
ERROR 1: ortho2017_RVB_pays.jp2, band 1: IReadBlock failed at X offset 0, Y offset 0: opj_decode() failed
"
Could you please figure out my falt?

[cwerner]

Hi Wei.

I think this is caused by your version of GDAL. Pretty sure you are using a JPG2000 driver (most likely OpenJPEG) that cannot handle these big jp2 files...
See:
https://trac.osgeo.org/gdal/ticket/7092
https://lists.osgeo.org/pipermail/gdal-dev/2017-October/047397.html

You either need to use another jp2 driver or try to upgrade the OpenJPEG lib... I think our GIS people might have used the (commercial) JP2KAK driver...

A list of gdal drivers is here: https://gdal.org/drivers/raster/ (depending on your system not all of them are installed by default)

Hope this works for you?