Home

Import Data Using The Rub Importer

⚠️ Note: This page is within the system namespace/directory because as of 2024-09-20, the RUB importer in ReSeeD only supports importing from a subdirectory of the same S3 share ("share" meaning "parent of an S3 bucket") that is also used by ReSeeD to store the imported data (configured using the S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, and S3_REGION variables in the .env file). The name of the S3 bucket from which the RUB importer imports the data is configured in the S3_FILE_UPLOAD_BUCKET variable in the .env file. Because access to this S3 share requires sysadmin access anyway, this page belongs into the system directory of the wiki for the time being.

About this process

This process uses the Bulkrax CSV from S3 parser to do imports. Metadata is prepared in CSV files, data for each dataset is provided in distinct folders (see below).

Prepare the data

The data to be imported needs to have the following file structure

• There should be a file called metadata.csv

  • The format of the columns in the metadata.csv file is explained in The metadata CSV format section (below)
  • General csv remarks
    • Commas are used to separate columns, and semi-colons used (in some cases) to separate values within a single column.
    • All text containing commas or semi-colon not meant to be interpreted as separators (e.g. in description or when listing contributors by "LAST_NAME, FORENAME(S)") needs to be wrapped in quotation marks.
    • Encoding: UTF-8 without BOM is advised.
  • The CSV file should contain one row for each dataset to be imported
    • The row should mention the path to the dataset relative to the directory containing the metadata.csv in the column dataset_path.

• Within each dataset path, there should be a directory named data where all the data for the dataset is placed.

  An example data structure for 2 datasets is shown below

  cl-reseed_import/set1/
  ├── dataset1
  │   └── data
  │       └── 1529
  │           ├── folder_1
  │           │   ├── another_file.exe
  │           │   └── some_other_file.json
  │           ├── my_software.exe
  │           └── mydata.json
  ├── dataset2
  │   └── data
  │       ├── AV02CP07GI0
  │       │   ├── anat
  │       │   │   └── sub-AV02CP07GI0_T1w.nii
  │       │   └── func
  │       │       └── sub-AV02CP07GI0_task-rest_bold.nii
  │       ├── CHANGES
  │       ├── README
  │       ├── dataset_description.json
  │       ├── participants.json
  │       └── participants.tsv
  └── metadata.csv
  

• The example zip file Example_RUB_import_data.zip has the datasets and the metadata.csv structured as needed.

Steps to run an import

1. Upload the data you want to import (for example: the unzipped data in Example_RUB_import_data.zip) into the S3 bucket that ReSeed has access to

   For example: cl-reseed_import

   This bucket name needs to be filled in the form for Specify a bucket name with prefix

2. Log into ReSeeD as an administrator.

3. On the dashboard you should see the options Importers and Exporters. Click on Importers.

   

4. In the importers page, click on New on the top left corner. This would open the importer form

   

5. Fill in the Importer form with the values as shown in the screenshot above and click on Create and import

   Field name	Value	Note
   Name	Any identifiable name for the import
   Administrative Set	RUB publication workflow	This will apply this workflow to all imported datasets
   Frequency	Once	We are running a one off import
   Limit	0 or leave blank	This will import all records in the metadata.csv file
   Parser	CSV from S3 - ReSeed CSV parser for work (Datasets) from local S3	This will choose the parser for ReSeed
   Visibility	Private	The workflow will need all datasets to be private until published
   Rights statement	Leave blank	It will pick up the rights statement from the csv file
   Specify a bucket name with prefix	cl-reseed_import	The bucket name with the prefix. You could also add a path within the bucket, for example: cl-reseed_import/set1

The metadata CSV format