{"id":76,"date":"2020-11-17T13:05:28","date_gmt":"2020-11-17T12:05:28","guid":{"rendered":"https:\/\/mxrdr-info.ocean.icm.edu.pl\/?page_id=76"},"modified":"2021-10-29T14:59:04","modified_gmt":"2021-10-29T12:59:04","slug":"data-uploading","status":"publish","type":"page","link":"https:\/\/info.mxrdr.icm.edu.pl\/index.php\/data-uploading\/","title":{"rendered":"API commands"},"content":{"rendered":"\n

On this page you will learn how to upload data using API. Make sure you are both registered<\/a> and have created API Token<\/a>. This page contains terminal commands that will create dataset, upload files to dataset. The commands are color coded as shown in the color coding section<\/a>. There is also a command to remove a dataset<\/a>. The contents are listed in order of increasing difficulty. <\/p>\n\n\n\n

Color coding:<\/p>\n\n\n\n

terminal command<\/span><\/p>\n\n\n\n

variable<\/span> definition or $variable invoke<\/span><\/p>\n\n\n\n

file or path and file<\/span><\/p>\n\n\n\n

string or path<\/span><\/p>\n\n\n\n

To add new dataset to root repository using json file and API token use the following curl command:<\/p>\n\n\n\n

curl -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST “https:\/\/mxrdr.icm.edu.pl\/api\/dataverses\/:root<\/span>\/datasets” –upload-file file.json<\/span><\/p>\n\n\n\n

To add new dataset and store dataset id in the variable mxrdrid<\/em> that can be utilized elsewhere in your scripts use the following curl command:<\/p>\n\n\n\n

mxrdrid<\/span>=$( curl<\/span> -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST “https:\/\/mxrdr.icm.edu.pl\/api\/dataverses\/:root<\/span>\/datasets” –upload-file file.json<\/span> | grep<\/span> -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep<\/span> -o -P ‘(?=[0-9]).’ )<\/p>\n\n\n\n

To get identifiers of published dataverses in the repository use the following curl command:<\/p>\n\n\n\n

curl<\/span> –silent -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> “https:\/\/mxrdr.icm.edu.pl\/api\/search?q=*<\/span>&type=dataverse<\/span>&per
_page=1000<\/span>” | tr ‘,’ “\\n” | grep -o -P ‘(?=”identifier”:”).*(?=”)’ | replace ‘”identifier”:”‘ ”<\/p>\n\n\n\n

Then one can use those identifiers to upload datasets directly to specific dataverse. Using variable dataverse<\/em> to upload:<\/p>\n\n\n\n

mxrdrid<\/span>=$( curl<\/span> -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” <\/span>-X POST “https:\/\/mxrdr.icm.edu.pl\/api\/dataverses\/$dataverse<\/span>\/datasets” –upload-file file.json<\/span> | grep<\/span> -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep<\/span> -o -P ‘(?=[0-9]).’ )<\/p>\n\n\n\n

To add file to an already existing dataset use the following curl command:<\/p>\n\n\n\n

curl<\/span> –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST -F ‘file=@file.zip<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”<\/p>\n\n\n\n

One can use a text file key.txt<\/em> as an API token<\/a>, the file should contain one line: X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx<\/em> and can be used with the following curl command:<\/p>\n\n\n\n

curl<\/span> –silent –retry 10 -H @key.txt<\/span> -X POST -F ‘file=@file.zip<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”<\/p>\n\n\n\n

With the previously given basic commands it is possible to build more complicated scripts to upload multiple datasets and files. For example:<\/p>\n\n\n\n

originalpath<\/span>=$(pwd<\/span>) # store current path in a variable ‘originalpath’
dataverse1<\/span>=ABC<\/span>
dataverse2<\/span>=DEF<\/span>
date<\/span>=$( date<\/span> ) # store current date in a variable ‘date’
<\/p>\n\n\n\n

mxrdrid1<\/span>=$( curl<\/span> –retry 5 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST “https:\/\/mxrdr.icm.edu.pl\/api\/dataverses\/$dataverse1<\/span>\/datasets” –upload-file file1.json<\/span> | grep<\/span> -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep<\/span> -o -P ‘(?=[0-9]).’ )
mxrdrid2<\/span>=$( curl<\/span> –retry 5 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” <\/span>-X POST “https:\/\/mxrdr.icm.edu.pl\/api\/dataverses\/$dataverse1<\/span>\/datasets” –upload-file file2.json<\/span> | grep<\/span> -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep<\/span> -o -P ‘(?=[0-9]).’ )
mxrdrid3<\/span>=$( curl<\/span> –retry 5 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST “https:\/\/mxrdr.icm.edu.pl\/api\/dataverses\/$dataverse2<\/span>\/datasets” –upload-file file3.json<\/span> | grep<\/span> -o -P ‘(?=”id”:).(?=,”persistentId”:)’ | grep<\/span> -o -P ‘(?=[0-9]).’ )
<\/p>\n\n\n\n

mxrdrid<\/span>=$mxrdrid1<\/span>
cd<\/span> \/path\/to\/files\/1\/<\/span>
file<\/span>=file1.jpeg<\/span>
curl<\/span> –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST -F ‘file=@’$file’<\/span>‘ <\/span>-F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”
file<\/span>=file1.zip<\/span>
curl<\/span> –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST -F ‘file=@’$file’<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”
file<\/span>=file2.zip<\/span>
curl<\/span> –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST -F ‘file=@’$file’<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”
cd<\/span>
cd<\/span> $originalpath<\/span>
echo<\/span> $mxrdrid $file $date<\/span> >> mxrdridlog.txt<\/span>
<\/p>\n\n\n\n

mxrdrid<\/span>=$mxrdrid2<\/span>
cd<\/span> \/path\/to\/files\/2\/<\/span>
file<\/span>=file1.jpeg<\/span>
curl<\/span> –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST -F ‘file=@’$file’<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”
file<\/span>=file1.zip<\/span>
curl<\/span> –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST -F ‘file=@’$file’<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”
file<\/span>=file2.zip<\/span>
curl<\/span> –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” <\/span>-X POST -F ‘file=@’$file’<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”
cd
cd<\/span> $originalpath<\/span>
echo<\/span> $mxrdrid $file $date <\/span>>> mxrdridlog.txt<\/span>
<\/p>\n\n\n\n

mxrdrid<\/span>=$mxrdrid3<\/span>
cd<\/span> \/path\/to\/files\/3\/<\/span>
file<\/span>=file1.jpeg<\/span>
curl<\/span> –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST -F ‘file=@’$file’<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”
file<\/span>=file1.zip<\/span>
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST -F ‘file=@’$file’<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”
file<\/span>=file2.zip<\/span>
curl –silent –retry 10 -H “X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”<\/span> -X POST -F ‘file=@’$file’<\/span>‘ -F ‘jsonData={“description”:””,”termsOfUseAndAccess”:{“termsType”:”LICENSE_BASED”,”license”:“CC BY Creative Commons Attribution License 4.0”<\/span>}}’ “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>\/add”
cd
cd<\/span> $originalpath<\/span>
echo<\/span> $mxrdrid $file $date<\/span> >> mxrdridlog.txt<\/span><\/p>\n\n\n\n

To remove a dataset use the following commad:<\/p>\n\n\n\n

curl<\/span> -H @key.txt<\/span> -X DELETE “https:\/\/mxrdr.icm.edu.pl\/api\/datasets\/$mxrdrid<\/span>“<\/p>\n\n\n\n

To create new jpeg image based on a raw diffraction image use the following command that uses adxv<\/em>:<\/p>\n\n\n\n

adxv<\/span> -jpeg_quality 100 -sa image.cbf image.jpeg<\/span><\/p>\n\n\n\n

To create new jpeg image based on a raw diffraction image that could be utilized in a more complicated script use (here we have used this<\/a> as an example):<\/p>\n\n\n\n

…
cd<\/span> \/path\/to\/files\/1\/<\/span>
fileraw<\/span>=$ (ls<\/span> -1 *.cbf<\/span> | head<\/span> -1)
file<\/span>=”${fileraw%.*}<\/span>“
adxv<\/span> -jpeg_quality 100 -sa $fileraw<\/span> $file<\/span>.jpeg<\/span>
…<\/p>\n","protected":false},"excerpt":{"rendered":"

On this page you will learn how to upload data using API. Make sure you are both registered and have created API Token. This page contains terminal commands that will create dataset, upload files to dataset. The commands are color coded as shown in the color coding section. There is also a command to remove … <\/p>\n

Continue reading “API commands”<\/span><\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"_links":{"self":[{"href":"https:\/\/info.mxrdr.icm.edu.pl\/index.php\/wp-json\/wp\/v2\/pages\/76"}],"collection":[{"href":"https:\/\/info.mxrdr.icm.edu.pl\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/info.mxrdr.icm.edu.pl\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/info.mxrdr.icm.edu.pl\/index.php\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/info.mxrdr.icm.edu.pl\/index.php\/wp-json\/wp\/v2\/comments?post=76"}],"version-history":[{"count":48,"href":"https:\/\/info.mxrdr.icm.edu.pl\/index.php\/wp-json\/wp\/v2\/pages\/76\/revisions"}],"predecessor-version":[{"id":600,"href":"https:\/\/info.mxrdr.icm.edu.pl\/index.php\/wp-json\/wp\/v2\/pages\/76\/revisions\/600"}],"wp:attachment":[{"href":"https:\/\/info.mxrdr.icm.edu.pl\/index.php\/wp-json\/wp\/v2\/media?parent=76"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}