Admin Guide: Import Data Using The Rub Importer

⚠️ Note: This page is within the system namespace/directory because as of 2026-03-31, the RUB importer in ReSeeD only supports one set of credentials for accessing S3 buckets for any data on the system as well as data to be imported (configured using the S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, and S3_REGION variables in the .env file). We also suggest to use the S3 bucket configured in the S3_FILE_UPLOAD_BUCKET variable in the .env file for data ingest. Because access to to S3 used by the ReSeeD service requires sysadmin access anyway, this page belongs into the system directory of the wiki for the time being.

About this process

This process uses the Bulkrax CSV from S3 parser to do imports. Metadata is prepared in CSV files, data for each dataset is provided in distinct folders (see below).

Prepare the data

The data to be imported needs to have the following file structure

• Metadata for all datasets to be imported is read from a single metadata.csv

  • The format of the columns in the metadata.csv file is explained in The metadata CSV format section (below). ⚠️ Make sure to take into account any known issues mentioned there!
  • General csv remarks
    • Commas are used to separate columns, and semicolons used (in some cases) to separate values within a single column. Make sure to set the field separator accordingly.
    • All text containing commas or semicolon not meant to be interpreted as separators (e.g. in description or when listing contributors by "LAST_NAME, FORENAME(S)") needs to be wrapped in quotation marks.
    • Encoding: UTF-8 without BOM is advised.
  • The CSV file needs to contain one row for each dataset to be imported
    • Each row needs to mention the path to the dataset relative to the directory containing the metadata.csv in the column dataset_path.

• Within each dataset path, there needs to be a directory named data where all the data for the dataset to be ingested is placed.

  An example data structure for 2 datasets is shown below

  cl-reseed_import/set1/
  ├── dataset1
  │   └── data
  │       └── 1529
  │           ├── folder_1
  │           │   ├── another_file.exe
  │           │   └── some_other_file.json
  │           ├── my_software.exe
  │           └── mydata.json
  ├── dataset2
  │   └── data
  │       ├── AV02CP07GI0
  │       │   ├── anat
  │       │   │   └── sub-AV02CP07GI0_T1w.nii
  │       │   └── func
  │       │       └── sub-AV02CP07GI0_task-rest_bold.nii
  │       ├── CHANGES
  │       ├── README
  │       ├── dataset_description.json
  │       ├── participants.json
  │       └── participants.tsv
  └── metadata.csv
  

• The example zip file Example_RUB_import_data.zip has the datasets and the metadata.csv structured as needed.

Steps to run an import

1. Upload the data you want to import (for example: the unzipped data in Example_RUB_import_data.zip) into any S3 bucket that ReSeed has access to. We advise to use the bucket configured in the S3_FILE_UPLOAD_BUCKET variable in the .env file.

   For example: cl-reseed_import - make sure to make exact note of this as you will need the exact bucket name to start the import.

2. Log into ReSeeD as an administrator.

3. On the dashboard you should see the options Importers and Exporters. Click on Importers.

   

4. In the importers page, click on New on the top left corner. This would open the importer form

   

5. Fill in the Importer form with the values as indicated in the table below and click on Create and import

   ⚠️ CREATE AND VALIDATE and CREATE are not supported at the moment.

   ⚠️ The RUB importer will always display Error: NoMethodError - undefined method complex_subjects'`. ⚠️ You may safely ignore this because the complex_subjects methods is only part of the CRC Importer. Look at Sidekiq to see if the importer is running succesfully.

The metadata CSV format