Editing Multiple HEX Files At Once v3

Use this forum to discuss hex-editing or otherwise altering any Three Kingdoms game, and to promote modded titles and patches.

Editing Multiple HEX Files At Once v3

Unread postby Lezard » Fri Dec 14, 2012 10:51 pm

Note: I'll take ROTK XI as an example (Scenario files specifically), but the idea presented here applies to any HEX files.

The Problem
As everybody knows (or is going to soon find if new here) editing can become a tedious task, specially when you want to edit Scenario files, you have to apply the same changes over and over again for each file.

The Solution in Theory
One way around this problem would be batch searching and replacing occurrences in the files (make X different replacements, of Y occurrences, in Z different files). This works pretty much as with any text files, let's say you want to change "The cat climbed the tree and could not get down" for "The werewolf climbed the tree and could not get down", and you don't want to change other phrases that also have cat on then. Well, it's obvious, you search for the entire phrase and replace for the altered entire phrase. When you want to change a couple of stats of an officer across many scenarios in hex format, all you have to do is select a bigger chunk of bytes and replace for an slightly different chunk (how many officers have the same APTs, Stats, Ability Persistence and Skill — note they are next to each other), if the chunk is big enough it's probably unique (and if it wasn't everything sided like that you could use wildcards, but I'll skip that).
EDIT (2nd solution): Another way around the problem would be to specify an offset, insert a new string and delete the old string. This way you're sure to change values (no need to match strings), but is not sure of which were the previous ones (unless you go and check them in every file). This solution isn't easily available to everyone tough.

The Solution in Practice
Many programs offer batch search and replace features, but not so many have hex search and replace and when they do, they're usually paid. Here are some of them (which are paid):
Hex Editor Neo ("Find in Files" feature — popular here in the-scholars)
Batch Hex Editor 5.0 (the name says it all)
PowerGrep (very powerful tool, have the feature as a bonus — very expensive)
and so on (there are many paid editors out there with this feature).

They maybe worth the price for some, and surely have a bunch of advantages and other features, but still are paid and probably overkill for the basic need pointed above shared by the ROTK enthusiasts. So here are some free editors that supposedly do that (source https://en.wikipedia.org/wiki/Comparison_of_hex_editors):
AviHex
Be.HexEditor
XVI32 (can be called from console and allows internal scripting — good for who can write scripts but I don't think it have the feature in question)
MadEdit (have the feature, but you have to use the "opened files" option because the "folder" one doesn't work — tremendous amount of supported encodes)
wxHexEditor (still beta ... I couldn't even run it)
iBored (focus on Mac)

Free editors that supposedly don't do that:
1Fh
HxD
Emacs
Frhed (Free Hex Editor)
Hex Edit
Hexplorer ("powerful editor for images")
Okteta
tiny hexer

Note that I have limited myself to Windows since the game runs on Windows. Unfortunately many are bare bones beta versions.

Other alternative is to go for command-line based programs and scripts. I'll follow this post with some scripts made by me.

Scripts - Preparation
First things first: all the solutions below are FREE and for Windows, they are in order of flexibility given to the end user (you — who may not find use in a regex feature without some reading, hence the ordination).

For those asking themselves, "how much I need to know?" I'd say not much really. You must learn to navigate in powershell and run the scripts. All the magic have been already written by me, you don't need to learn how to program. I'll point out if something else is needed to make a worth usage (such as regex).
- To navigate in the Powershell use cd, dir or ls commands.
- To run a script type it's path with file name and press <enter> (example: "C:\path\script_name.ps1"). If any parameters are required I'll point out.
-More on Navigation:
http://secretgeek.net/powershell_beginner.asp
http://ss64.com/ps/set-location.html
http://ss64.com/nt/cd.html
Note: the last isn't specifically for powershell, but works on most part due to alias (read first link — secretgeek)

Now basic software requirements for my scripts:
- Windows Powershell 2.0 or later (comes with Windows 7 or above in place of CMD, but can be installed on earlier versions too)
- Not necessary, but I recommend Notepad++ to handle the scripts (instead of Notepad).
- Others will be pointed with the scripts themselves.
- Note: If you want to further develop the scripts (for those "advanced"). Powershell-plus is a possible free IDE choice (though Notepad++ is just fine).

If you've never run a script in Powershell, you'll have to unlock scripts usage:
go start > execute > type powershell <enter>. The black screen opens and you'll be looking to your powershell terminal: type Set-ExecutionPolicy RemoteSigned <enter>. Click here for detail about this.

Actual Scripts

Solution 1: GSAR
Extra knowledge: none
Extra software: GSAR
Read the huge comment that starts my script (begin with <# and end with #>), for notes on GSAR installation and script usage (parameters and user input).
Copy-Paste the code below to notepad and save it as whatever.ps1
Code: Select all
<#
.SYNOPSIS
   This is a PowerShell Script that relies on GSAR (General Search And Replace) to batch replace hex strings (can make X different replacements, of Y occurrences, in Z different files).
   
.DESCRIPTION
   This script helps you by coordinating multiple calls to GSAR (allowing the said X different replacements) and also by providing a convenient input format: it gets all the changes to be made from a file, a list written by you.
   GSAR is simple and fast, but inflexible. No wildcards or regular expressions are allowed in search string. Also you can't skip a digit when replacing (if nothing is written as a replacement, the digit is deleted). GSAR always replaces all occurrences of searched strings in each file (say Y ocurrences), it's not optional.
   GSAR can be found here: http://gnuwin32.sourceforge.net/packages/gsar.htm
   Setup: put gsar.exe in C:\WINDOWS, keep gsar-man.pdf somewhere accessible and setup VC6RedistSetup_deu.exe if you lack msvcrt.dll and msvcp60.dll in your system folder (Windows\System or WINDOWS\system32). You can throw remaining files away.

.PARAMETER gsarImportPath
   First parameter (no need to write "-gsarImportPath") expects the path to a properly written txt file with a list of changes to be done ("C:\Path\GSARimport.txt"). This file is supposed to have each line in the following format (and is only a bunch of these lines):
   "NAME", "FIND", "REPLACE"
   Where:
   - NAME = A name/description for your convenience (so you remember what that line was about).
   - FIND = Hex String to search for.
   - REPLACE = Hex String to use as replacement.
   Example:
   "Yellow", "46 61 63 65 64 20 77 69 74 68 20", "65 6C 6C 6F 77 54 75 72 62 61 6E"
   Note: Follow this exact format, don't add/remove any quote, comma or blank space, if you do (add/remove) the line will be skipped and no change applied (or you'll get a gsar error report, but hardly worse than that).

.PARAMETER filesPath
   Second parameter (no need to write "-filesPath") expects the path to a folder containing the hex files to be changed ("C:\Path"). Note: Changes will apply to all files in the specified folder that match the extension given (see next parameter).

.PARAMETER ext
   Third parameter (no need to write "-ext") expects an extension, without the dot ("s11"). This helps filtering unwanted files a bit.

.PARAMETER noBackup
   Optional parameter (must write "-noBackup"), allows you to skip the auto backup.

.EXAMPLE
   ... Usage:
   1- Create a file with lines formated as stated above (1st parameter)
   2- Create a folder and put the files you want changed there (2nd parameter)
   3- All files must have same extension (3rd parameter)
   4- Call this script via powershell:
      a) Example with backup (put parameters in this order):
         .\ROTK_XI_gsar_v3.ps1 "C:\Path\GSAR Import.txt" "C:\Path\Test" "s11"
      b) Example without backup (put parameters in this order):
         .\ROTK_XI_gsar_v3.ps1 "C:\Path\GSAR Import.txt" "C:\Path\Test" "s11" -noBackup
   5- Check the generated gsarLog saved in the same folder of the input files (2nd parameter). That's important as it'll allow you to easily check whether or not everything was changed, or if something wasn't changed, or even if something that should not be changed was so. The log tells you the number of matches of change "Whatever" (the name you gave for the change in your file will be displayed) in each file (tells you nothing when no matches are found). Also tells you if a line could not be read and was skipped.
#>

# Definition of parameters taken as input by this script
[CmdletBinding()]
Param(
   [Parameter(Mandatory=$True,Position=1)] [string]$gsarImportPath,
   [Parameter(Mandatory=$True,Position=2)] [string]$filesPath,
   [Parameter(Mandatory=$True,Position=3)] [string]$ext,
   [Parameter(Mandatory=$FALSE)] [switch] $noBackup
)

#-----------------------------------------------------------------------------------------------

# Function equivalent to PAUSE from DOS (pauses execution until a key is pressed)
function WaitKey
{
    param( [String] $strPrompt = "Press any key to continue ... ")
    Write-Host
    Write-Host $strPrompt -NoNewline
    $key = [Console]::ReadKey($true)
    Write-Host
}

#-----------------------------------------------------------------------------------------------

# Function that backup files (only those that'll be changed)
function BackupFiles ($SourceFolder)
{
   # Create backup folder
   $date = (Get-Date).tostring("yyyy-MM-dd_HH-mm-ss")
   $DestinationFolder = "$SourceFolder\Backup_" + $date
   if(!(test-path "$DestinationFolder" -pathType container))
   {
      New-Item -ItemType directory -Path $DestinationFolder
   }
   
   # Copy-Rename Files
   $aux = $SourceFolder + "\" + "*.s11"
   $filenames = @(Get-ChildItem $aux)
   foreach ($file in $filenames)
   {
      $fileLeaf = Split-Path "$file" -Leaf
      $bakfile = $fileLeaf + ".bak"
        Copy-Item $file "$DestinationFolder"
      Rename-Item "$DestinationFolder\$fileLeaf" $bakfile
   }
}

#-----------------------------------------------------------------------------------------------

# GSAR requires the hex strings to be written in an specific format
# This function does the convertion
function ConvertToGsarInput ($gsar_input)
{
   $gsar_input = $gsar_input -replace ' ', ':x'
   return ("$gsar_input")
}

#-----------------------------------------------------------------------------------------------
# Main
#-----------------------------------------------------------------------------------------------

# Display error and exit script if 2nd parameter points inexistent folder path
if(!(test-path "$filesPath" -pathType container))
{
   Write-Host "Error: source folder does not exist!" -ForegroundColor "Red"
   Write-Host "Exiting script"
   WaitKey "Press any key to exit ... "
   exit
}

# Actual backup
if (-not $noBackup)
{
   BackupFiles ($filesPath)
}

# Create LOG file
$enddate = (Get-Date).tostring("yyyy-MM-dd_HH-mm-ss")
#$filename = "FileName{0:yyyyMMdd-HHmm}" -f (Get-Date)
$logname = "gsarLog_"+$enddate+".txt"
Set-Content -path "$filesPath\$logname" -value "`r`n===============`r`n-GSAR LOG FILE-`r`n==============="

# For each change in the list txt call gsar (applying them to all hex files)
# And, for each call, register changes to LOG file
$lines = Get-Content ($gsarImportPath)
#$lines = (Get-Content ($gsarImportPath) | Measure-Object –Line ).Lines;
for ($i=0; $i -lt $lines.count; $i++)
{
   $ffile = (Get-Content "$gsarImportPath")[$i]
   if ($ffile -like "`"*`", `"*`", `"*`"")
   {
      $line = $ffile -split ", "
      $l1 = $line[0]
      $l2 = $line[1]
      $l3 = $line[2]
      Add-Content -path "$filesPath\$logname" -value "`r`n`r`n---$l1---"
      $a = ConvertToGsarInput (" $l2")
      $b = ConvertToGsarInput (" $l3")
      Push-Location $filesPath
      gsar "-s$a" "-r$b" "-o" "*.$ext" | Add-Content "$filesPath\$logname"
      Pop-Location
   } else
   {
      if ($ffile -eq "")
      {
         Add-Content -path "$filesPath\$logname" -value "`r`n`r`n---Incorrect input line skipped---`r`n<empty line>"
      }else
      {
         Add-Content -path "$filesPath\$logname" -value "`r`n`r`n---Incorrect input line skipped---`r`n$ffile"
      }
   }
}
Add-Content -path "$filesPath\$logname" -value "`r`n`r`n==============`r`n-GSAR LOG END-`r`n=============="

#-----------------------------------------------------------------------------------------------
# END

Sat Jan 05, 2013 — Small correction, it now checks if input folder exists even if you don't backup
Sat Jan 05, 2013 — Improved correctness check over input lines from the txt list of the user

That's it! I hope I haven't forgotten anything important. I'm NOT a programmer myself.
THE END

Questions :?:
Last edited by Lezard on Tue Feb 04, 2014 7:21 pm, edited 14 times in total.
"They envied the power that I wield, and that envy led them to an early end". —Lezard Valeth
User avatar
Lezard
Retired Account
 
Posts: 445
Joined: Sun Nov 25, 2012 10:33 pm

Re: Editing Multiple HEX Files At Once

Unread postby Zyzyfer » Tue Dec 18, 2012 4:11 pm

That's waaaaaaaaaaaaay over my head. :lol: I appreciate you taking the time to write all that up but I generally don't download too many programs and I'm kind of a dunce at computer programming.

By the way, I unlocked the trial version of Hex Editor Neo and tried to do the Find in Files command. Had some problems. It started off all right for a couple of officers, but kept creating backup files and temporary files and things even though I turned off auto backup files, and then just kind of lost the plot. It doesn't save time inputting the actual stat changes but it saves some time looking up officers that I want to change and doing the repetitive copy/paste. I will be done soon with the major scenarios Lord Cao Cao posted but there are some other ones that I want to edit, so I'd like to take advantage of the feature while it's free. But hell if I can figure out what I was doing wrong. :|
Gamefaqs: KongZhou
Steam: heinous_won
User avatar
Zyzyfer
Scholar of Shen Zhou
 
Posts: 3208
Joined: Thu May 08, 2008 1:17 pm
Location: South Korea

Re: Editing Multiple HEX Files At Once

Unread postby kiko94 » Tue Dec 18, 2012 9:07 pm

Can't wait to get my hands on the pc to test this out on friday, I usually (for finding officers) just move 152 bytes (or how many bytes was that again ? ) it will also be helpful for editing city stats I guess.
My first collection of RTK XI duels (among many to come) here
Duel Collection Number 2 this time in HD
User avatar
kiko94
Sage
 
Posts: 292
Joined: Mon Aug 06, 2012 12:04 am

Re: Editing Multiple HEX Files At Once

Unread postby Lezard » Wed Dec 19, 2012 4:25 pm

That's waaaaaaaaaaaaay over my head. :lol: I appreciate you taking the time to write all that up but I generally don't download too many programs and I'm kind of a dunce at computer programming.

I'm graduated in Computer Science (though have never been very fond of programing nor worked with it). So I may not be very clear at times (to make things worse I was asleep when posting — did some grammar corrections to the post now). If you have any question just ask and I will do my best to answer. :D

By the way, I unlocked the trial version of Hex Editor Neo and tried to do the Find in Files command. Had some problems. It started off all right for a couple of officers, but kept creating backup files and temporary files and things even though I turned off auto backup files, and then just kind of lost the plot. It doesn't save time inputting the actual stat changes but it saves some time looking up officers that I want to change and doing the repetitive copy/paste. I will be done soon with the major scenarios Lord Cao Cao posted but there are some other ones that I want to edit, so I'd like to take advantage of the feature while it's free. But hell if I can figure out what I was doing wrong. :|

Hmmmm ... I've never used Hex Editor Neo, so I can't help you with it's usage (same for Batch Hex Editor 5.0). I have PowerGREP (but unless you're an IT pro your not gonna buy it — it's ridiculously pricey). Though complicated, I'd say the script is the way to go. You don't need to know much to use what I posted, just got to add those blocks of code in the item "d" with the strings that you want changed you see (of course it may take some time to get used with the powershell terminal, but all you really need to know is the cd command to navigate). I guess that in order to write a script you will spend about double the time it would take to edit one file (a bit more actually ~2,5x). So if you have 3 or more files to change, batch replace might be a better choice (and you can save the work for future use).

Can't wait to get my hands on the pc to test this out on friday, I usually (for finding officers) just move 152 bytes (or how many bytes was that again ? ) it will also be helpful for editing city stats I guess.

Yep, 152 bytes for an officer, but I didn't get what you mean by 'move' ... about city stats, well, I guess so, but I must remark that if there are bytes that change from file to file (like the deceased/unavailable officer byte) you must use wildcards, or take separate, smaller, chunks of bytes (the search part would fail with a byte that changes), or use the address-insert-delete solution (you could sophisticate it a bit more too if you want to keep the mutable byte).
"They envied the power that I wield, and that envy led them to an early end". —Lezard Valeth
User avatar
Lezard
Retired Account
 
Posts: 445
Joined: Sun Nov 25, 2012 10:33 pm

Re: Editing Multiple HEX Files At Once

Unread postby Killaz916 » Thu Dec 20, 2012 1:01 pm

Outstanding guide there, wish I had the patience to write a guide like that lol.
There's a feature in one program where you can make bookmarks every selected bytes, forgot the name though.

Also I like how you chose the name Lezard, Valkyrie Profile was a great series. I actually have some amazing art books of it too. I'll send you some pics if you want.

Anyways if I remember the name of the hex editor I'll tell you guys.
User avatar
Killaz916
Scholar
 
Posts: 323
Joined: Wed Jul 09, 2008 5:51 am
Location: Michigan

Re: Editing Multiple HEX Files At Once

Unread postby Lezard » Thu Dec 20, 2012 3:38 pm

Thank you! It took me some time to develop the script (my first time around powershell) and write it all down. :lol:
I just wish I could make it look less intimidating (because it's not hard really).

Valkyrie Profile series is one of my favorites, I tried gathering artworks in the past, but failed, so if you could share it 'd be highly appreciated :D

Funny, coming to think of it, my reaction to ROTK portraits were more or less the same as when I first played VP (PS1). And then I learned about Shunya Yamashita searching for VP2 material ... well better stop reminiscing :lol:
"They envied the power that I wield, and that envy led them to an early end". —Lezard Valeth
User avatar
Lezard
Retired Account
 
Posts: 445
Joined: Sun Nov 25, 2012 10:33 pm

Re: Editing Multiple HEX Files At Once

Unread postby Killaz916 » Thu Dec 20, 2012 4:09 pm

Yeah no problem either today or this weekend when I'm out of work I'll upload the art book to you. The scans are high detail. I actually liked the voice actor for Lezard in VP2 since he does a lot of anime series too.
User avatar
Killaz916
Scholar
 
Posts: 323
Joined: Wed Jul 09, 2008 5:51 am
Location: Michigan

Re: Editing Multiple HEX Files At Once v2

Unread postby Lorilee96 » Mon Aug 26, 2013 4:58 am

It does highlighting and line numbers, these were my first reasons for using it. Notepad ++ has been my best friend for writing scripts and HTML files.
Lorilee96
Lorilee96
Tyro
 
Posts: 2
Joined: Mon Aug 26, 2013 4:46 am


Return to Three Kingdoms Game Editing & Mods

Who is online

Users browsing this forum: No registered users and 1 guest

Copyright © 2002–2008 Kongming’s Archives. All Rights Reserved