Download & Rclone part removed due to server shutting down. Now this is just a TLDR of ROM update.

This assumes everything is already configured to match No-Intro/Redump romset, only minimal config example is provided in #Config.

TLDR

1. Download newer DATs via browser/script
   • No-Intro DAT Files, all in one pack (table d’hôte)
   • Redump DAT Files, pick & download (à la carte)
     • browser: redump-highlight.user.js
     • script: update-dat.sh
   • 模拟游戏中文数据站
2. Unpack DATs to ROMVault\DATRoot
   • No-Intro > pwd, aka. DATRoot
   • Redump > DATRoot\Redump
3. Prepare selected DATs via init-retool-dat.sh (rsync)
   • Only include necessary romset to speed up retool
   • Order matters, see script comment
4. Filter DAT in retool
   1. Add DAT files recursively from quick import folder
   2. Remove CHS DATs
   3. Process DAT Files
5. Update database in RomVault
   1. Update DATs
   2. Scan ROMs
   3. Find Fixes (do NOT fix ROMs yet)
6. Fix romset
   • Round One
     1. Select romset with missing ROMs, right click and choose Save fix DATs
     2. Convert ROMVault fix DATs to human/machine readable ROM list via fixDat.nu
     3. Get missing ROMs manually, and save to ROMVault\ToSort¹
     4. RomVault: Scan Quick -> Find Fixes -> Fix ROMs
   • Round Two
     1. Repeat 6.1-6.3 with a newer romset² somewhere else
     2. Get last few missing ROMs somehow
     3. Check ROMVault\ToSort again and clean up ROMs
7. Update playlist in RetroArch
   1. Refresh playlist
   2. Clean playlist
8. Merge/Sync new romset

TLDR²

1. Get newer DATs
   1. No-Intro DAT Files, unpack to ROMVault\DATRoot
   2. update-dat.sh
2. init-retool-dat.sh
3. Retool
   1. Add DAT files recursively from quick import folder
   2. Remove CHS DATs
   3. Process DAT Files
4. See point 5-8 in #TLDR

Config

RomVault -> Settings:

• RomVault Settings -> Core Settings
  • DATRoot: DATRoot\Retool, NOT DATRoot!
  • Filenames not to remove: *BIOS*, *-CH*, *中文游戏合集
• Directory Mappings
  • Main ROM directory: RomVault <- path\to\RetroArch\roms
  • Sub collection: RomVault\No-Intro\Nintendo - Nintendo 3DS (Decrypted) <- E:\external\disk\path\roms\No-Intro\Nintendo - Nintendo 3DS (Decrypted)

Retool:

• Paths -> output folder Replace input DAT files
• Exclusions: AabdepPrv, aka. Applications, Audio, Bad dumps, Demos, Educational, Preproduction, Promotional, Pirate, Video
• Options
  • Compilation handling: Prefer individual titles
  • Output options: originalheader, reprocess
• quick import folder: $SCOOP\persist\romvault\DATRoot\Retool

RetroArch playlist:

• scan_dat_file_path & scan_filter_dat_content are not enabled as DAT files have timestamp suffix. While it can be easily renamed, I’ve already dedup ROMs in RomVault so using it here would not have much benefit
• "scan_overwrite_playlist": false ensures ROMs are not deleted as soon as I refresh playlist. This is used as sometimes I don’t want/have full copy of a gigantic romset

// /path/to/RetroArch/playlists/Sega - 32X.lpl
{
  "version": "1.5",
  "default_core_path": "/path/to/RetroArch/cores/picodrive_libretro.so",
  "default_core_name": "Sega - MS/GG/MD/CD/32X (PicoDrive)",
  "base_content_directory": "/home/redacted/Game",
  "label_display_mode": 0,
  "right_thumbnail_mode": 0,
  "left_thumbnail_mode": 0,
  "thumbnail_match_mode": 0,
  "sort_mode": 0,
  "scan_content_dir": "/path/to/RetroArch/roms/No-Intro/Sega - 32X",
  "scan_file_exts": "",
  "scan_dat_file_path": "",
  "scan_search_recursively": true,
  "scan_search_archives": true,
  "scan_filter_dat_content": false,
  "scan_overwrite_playlist": false,
  "items": [
    {}
  ]
}

• Simple Migration/Mirroring to another platform
  1. Change path
  2. Refresh playlist
  3. Clean playlist

Structure

• No-Intro files (unar)
  • No-Intro
  • Non-Redump
  • Source Code
  • Unofficial
  • index.txt
• Redump & CHS (wget)
  • dat-*.list
  • update-dat.sh
• Retool (rsync)
  • exclude-*.txt
  • include-*.txt
  • init-retool-dat.sh

$ cd $SCOOP/persist/romvault/DATRoot
$ tree -L 1 ./
./
├── .bak # backup of previous dat
│   ├── CHS
│   ├── No-Intro
│   └── Redump
├── CHS # dat
├── No-Intro
├── Non-Redump
├── Redump
├── Source Code
├── Unofficial
├── Retool # retool processed dat
│   ├── CHS # simple rsync
│   ├── No-Intro
│   └── Redump
├── dat-chs-cm.list # wget list
├── dat-chs-ol.list
├── dat-chs.list -> dat-chs-cm.list # symlink
├── dat-redump.list
├── exclude-nointro.txt # rsync rule
├── exclude-redump.txt
├── include-nointro.txt
├── include-redump.txt
├── index.txt # file tree
├── init-retool-dat.sh # rsync script
├── update-dat.sh # wget script

Script

#!/usr/bin/env nu

# Usage: convert ROMVault fix DATs to sensible ROM list
# Tested environment: Linux, Windows, WSL

def dump-list [dat: path] {
    let output = ($dat | path basename
        # rm redundant prefix
        | str replace '[Unnumbered] Chinese Game Dat (Simplified) - ' ''
        # rename extension
        | str replace '.dat' '.txt')

    # parse fixDat file
    open $dat --raw
    # clean escape symbol
    | str replace --all '&' '&'
    | str replace --all '"' '"'
    | str replace --all '&lt;' '<'
    | str replace --all '&gt;' '>'
    # match via regex instead of useless "from xml"
    # match "<game name", but not "cloneof" or "<rom name"
    | parse --regex '<game name="(?<name>[^"]+)"'
    | get name
    # append .zip suffix
    | each { |it| $"($it).zip" }
    | uniq
    | sort
    | save --force $output
}

# iterate
ls fixDat_*.dat | get name | each { |it|
    dump-list $it
    # rm $it
}