ℹ️ Note

A more thorough introduction/analysis is available as RomVault, Retool, fixDat & Digital Preservation.

I’ve mentioned in 更新强迫症#游戏 that rsync server in linked post shut down in April. It took me a while to try out different ideas for sufficient romset update, and while this post is more updated, the theory and analysis in ROM is still valid. Sadly, I’m still not in the mood to talk about digital preservation and this is only a TLDR of ROM update.

This assumes everything is already configured to match No-Intro/Redump romset, only minimal config example is provided in #Config.

TLDR

1. Download newer DATs via browser/script
   • No-Intro DAT Files, all in one pack / table d’hôte
   • Redump DAT Files, pick & download / à la carte
     • browser > redump-highlight.user.js
     • script > update-dat.sh
   • 模拟游戏中文数据站
2. Unpack DATs to ROMVault\DatRoot
   • No-Intro > pwd, aka. DatRoot
   • Redump > DatRoot\Redump
3. Prepare selected DATs via init-retool-dat.sh (rsync)
   • Only include necessary romset to speed up retool
   • Order matters, see script comment
4. Filter DAT in retool
   1. Add DAT files recursively from quick import folder
   2. Remove CHS DATs
   3. Process DAT Files
5. Update database in RomVault
   1. Update DATs
   2. Scan ROMs
   3. Find Fixes (do NOT fix ROMs yet)
6. Fix romset
   • Round One
     1. Select romset with missing ROMs, right click and choose Save fix DATs
     2. Get missing ROMs via fixDat.sh (rclone) to ROMVault\ToSort¹
     3. RomVault: Scan Quick -> Find Fixes -> Fix ROMs
   • Round Two
     1. Repeat 6.1-6.3 with a newer romset² somewhere else
     2. Get last few missing ROMs somehow
     3. Check ROMVault\ToSort again and clean up ROMs
7. Update playlist in RetroArch
   1. Refresh playlist
   2. Clean playlist
8. Merge/Sync new romset

TLDR²

1. Get newer DATs
   1. No-Intro DAT Files, unpack to ROMVault\DatRoot
   2. update-dat.sh
2. init-retool-dat.sh
3. Retool
   1. Add DAT files recursively from quick import folder
   2. Remove CHS DATs
   3. Process DAT Files
4. See point 5-8 in #TLDR

Config

RomVault:

• Set DatRoot to DatRoot\Retool
  • NOT DatRoot!
  • NOT DatRoot!
  • NOT DatRoot!
• DirMapping
  • Main ROM directory: map path\to\RetroArch\roms as RomVault
  • Sub collection: map E:\external\disk\path\roms\No-Intro\Nintendo - Nintendo 3DS (Decrypted) as RomVault\No-Intro\Nintendo - Nintendo 3DS (Decrypted)
• Add *BIOS*, *中文游戏合集 to IgnoreFiles

Retool:

• output: path\to\retool-win-x86-64
• quick import folder: ROMVault\DatRoot\Retool

RetroArch playlist:

• scan_dat_file_path & scan_filter_dat_content are not enabled as DAT files have timestamp suffix. While it can be easily renamed, I’ve already dedup ROMs in RomVault so using it here would not have much benefit
• "scan_overwrite_playlist": false ensures ROMs are not deleted as soon as I refresh playlist. This is used as sometimes I don’t want/have full copy of a gigantic romset

// /path/to/RetroArch/playlists/Sega - 32X.lpl
{
  "version": "1.5",
  "default_core_path": "/path/to/RetroArch/cores/picodrive_libretro.so",
  "default_core_name": "Sega - MS/GG/MD/CD/32X (PicoDrive)",
  "base_content_directory": "/home/redacted/Game",
  "label_display_mode": 0,
  "right_thumbnail_mode": 0,
  "left_thumbnail_mode": 0,
  "thumbnail_match_mode": 0,
  "sort_mode": 0,
  "scan_content_dir": "/path/to/RetroArch/roms/No-Intro/Sega - 32X",
  "scan_file_exts": "",
  "scan_dat_file_path": "",
  "scan_search_recursively": true,
  "scan_search_archives": true,
  "scan_filter_dat_content": false,
  "scan_overwrite_playlist": false,
  "items": [
    {}
  ]
}

• Simple Migration/Mirroring to another platform
  1. Change path
  2. Refresh playlist
  3. Clean playlist

Structure

• No-Intro files (unar)
  • No-Intro
  • Non-Redump
  • Source Code
  • Unofficial
  • index.txt
• Redump & CHS (wget)
  • dat-*.list
  • update-dat.sh
• Retool (rsync)
  • exclude-*.txt
  • include-*.txt
  • init-retool-dat.sh

$ cd /path/to/ROMVault/DatRoot
$ tree -L 1 ./
./
├── .bak (backup of previous dat)
│   ├── CHS
│   ├── No-Intro
│   └── Redump
├── CHS
├── dat-chs-cm.list (wget)
├── dat-chs-ol.list (wget)
├── dat-chs.list -> dat-chs-cm.list
├── dat-redump.list (wget)
├── exclude-nointro.txt (rsync)
├── exclude-redump.txt (rsync)
├── include-nointro.txt (rsync)
├── include-redump.txt (rsync)
├── index.txt
├── init-retool-dat.sh (rsync)
├── No-Intro
├── Non-Redump
├── Redump
├── Retool (retool processed dat)
│   ├── CHS (simple rsync)
│   ├── No-Intro
│   └── Redump
├── Source Code
├── Unofficial
├── update-dat.sh (wget)

Script

TBA

Download

Many ways to download missing ROMs from remote server:

1. HTTP: most obvious one, download via browser
2. FTP: similar to HTTP, server down with Rsync
3. Rsync, fixDat4rsync.sh, see ROM#Rsync
4. Wget, fixDat4http.sh, blocked by server, HTTP 303 See Other redirect
5. #Rclone (recommeded), works just like rsync, but easier
6. Bad boy behavior:³ fake UA, headless browser automation
7. …

Rclone

Showcase only, figure out how to setup yourself:

# prepare
cd $SIMPATH/RetroArch
ROMSET="Apple-Bandai - Pippin (Floppies)"

# list remote roms
rclone lsf "remote:No-Intro/${ROMSET}"

# dumb way to download
rclone copy "remote:No-Intro/${ROMSET}/Replace With Real ROM Name (Japan) (v1.03).zip" "roms/No-Intro/${ROMSET}"

# sync w/ remote the smart way
rclone --dry-run sync "remote:No-Intro/${ROMSET}/" "roms/No-Intro/${ROMSET}/"
# or more well-rounded, better than `--include-from` & `--exclude-from`
rclone -v --ignore-existing --update --max-size="200m" --no-update-modtime --filter-from="fixDat.list" sync "remote:No-Intro/${ROMSET}/" "roms/No-Intro/${ROMSET}/"

Example fixDat.list:

+ [BIOS] 32X M68000 (USA).zip
- [BIOS] 32X SH-2 Master (USA).zip
- [BIOS] 32X SH-2 Slave (USA).zip