API commands

On this page you will learn how to upload data using API. Make sure you are both registered and have created API Token. This page contains terminal commands that will create dataset, upload files to dataset. The commands are color coded as shown in the color coding section. There is also a command to remove a dataset. The contents are listed in order of increasing difficulty.

Color coding:

terminal command

variable definition or $variable invoke

file or path and file

string or path

To add new dataset to root repository using json file and API token use the following curl command:

curl -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST “https://mxrdr.icm.edu.pl/api/dataverses/:root/datasets” –upload-file file.json

To add new dataset and store dataset id in the variable mxrdrid that can be utilized elsewhere in your scripts use the following curl command:

mxrdrid=$( curl -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST “https://mxrdr.icm.edu.pl/api/dataverses/:root/datasets” –upload-file file.json | grep -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep -o -P ‘(?=[0-9]).’ )

To get identifiers of published dataverses in the repository use the following curl command:

curl –silent -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” “https://mxrdr.icm.edu.pl/api/search?q=*&type=dataverse&per
_page=1000” | tr ‘,’ “\n” | grep -o -P ‘(?=”identifier”:”).*(?=”)’ | replace ‘”identifier”:”‘ ”

Then one can use those identifiers to upload datasets directly to specific dataverse. Using variable dataverse to upload:

mxrdrid=$( curl -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST “https://mxrdr.icm.edu.pl/api/dataverses/$dataverse/datasets” –upload-file file.json | grep -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep -o -P ‘(?=[0-9]).’ )

To add file to an already existing dataset use the following curl command:

curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@file.zip‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”

One can use a text file key.txt as an API token, the file should contain one line: X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx and can be used with the following curl command:

curl –silent –retry 10 -H @key.txt -X POST -F ‘file=@file.zip‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”

With the previously given basic commands it is possible to build more complicated scripts to upload multiple datasets and files. For example:

originalpath=$(pwd) # store current path in a variable ‘originalpath’
dataverse1=ABC
dataverse2=DEF
date=$( date ) # store current date in a variable ‘date’

mxrdrid1=$( curl –retry 5 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST “https://mxrdr.icm.edu.pl/api/dataverses/$dataverse1/datasets” –upload-file file1.json | grep -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep -o -P ‘(?=[0-9]).’ )
mxrdrid2=$( curl –retry 5 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST “https://mxrdr.icm.edu.pl/api/dataverses/$dataverse1/datasets” –upload-file file2.json | grep -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep -o -P ‘(?=[0-9]).’ )
mxrdrid3=$( curl –retry 5 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST “https://mxrdr.icm.edu.pl/api/dataverses/$dataverse2/datasets” –upload-file file3.json | grep -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep -o -P ‘(?=[0-9]).’ )

mxrdrid=$mxrdrid1
cd /path/to/files/1/
file=file1.jpeg
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@’$file’‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”
file=file1.zip
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@’$file’‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”
file=file2.zip
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@’$file’‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”
cd
cd $originalpath
echo $mxrdrid $file $date >> mxrdridlog.txt

mxrdrid=$mxrdrid2
cd /path/to/files/2/
file=file1.jpeg
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@’$file’‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”
file=file1.zip
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@’$file’‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”
file=file2.zip
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@’$file’‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”
cd
cd $originalpath
echo $mxrdrid $file $date >> mxrdridlog.txt

mxrdrid=$mxrdrid3
cd /path/to/files/3/
file=file1.jpeg
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@’$file’‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”
file=file1.zip
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@’$file’‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”
file=file2.zip
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” -X POST -F ‘file=@’$file’‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”}}’ “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid/add”
cd
cd $originalpath
echo $mxrdrid $file $date >> mxrdridlog.txt

To remove a dataset use the following commad:

curl -H @key.txt -X DELETE “https://mxrdr.icm.edu.pl/api/datasets/$mxrdrid“

To create new jpeg image based on a raw diffraction image use the following command that uses adxv:

adxv -jpeg_quality 100 -sa image.cbf image.jpeg

To create new jpeg image based on a raw diffraction image that could be utilized in a more complicated script use (here we have used this as an example):

…
cd /path/to/files/1/
fileraw=$ (ls -1 *.cbf | head -1)
file=”${fileraw%.*}“
adxv -jpeg_quality 100 -sa $fileraw $file.jpeg
…