Page 1 of 3

Salem Layer Util

PostPosted: Tue Mar 26, 2013 9:08 am
by bmer
[see boshaw's post below]

Re: Salem Layer Util

PostPosted: Wed Mar 27, 2013 5:43 am
by boshaw
I'd just like to say that the link you posted is a bit outdated, and sadly i can't update it anymore than i don't seem to have access anymore.

However, here's a basic rundown of everything currently in salem-layer-util with respect to salem only:

Last Update: 4/3/13, using salem-client from ~3/10/13
if you notice a layer no longer working message me and i'll check to see if loftar updated a layer,etc

=Known Issues=

=Obtaining=
You can find the latest source @ https://github.com/boshaw/salem-layer-util
If you don't want to use git, they offer a zip file of the branch: https://github.com/boshaw/salem-layer-u ... master.zip
If you are running x64 you can use the following precompiled package: http://www.mediafire.com/download/zq87y ... -18-13.rar

=Usage=
There's multiple ways you can use salem-layer-util. Previously you'd have to install a CommonLisp compiler and set it up yourself, however, at the request of someone from HnH I added in support for ABCL by default. This means as long as you have a java JVM you'll easily be able to run salem-layer-util. You're still free to set it up with SBCL or any CL compiler though if you perfer it that way.

1) [Easy Method] simply download the branch/source, and use run-abcl.{bat,sh}
This will automatically load in the files and set you at a REPL once done where you can then do your work.

2) If you already have a CL compiler or don't mind downloading one you can set it up as follows:
2.1) download quicklisp ( http://beta.quicklisp.org/quicklisp.lisp )
2.2) run your CL compiler and load in quicklisp.lisp : `(load "path-to-quicklisp/quicklisp.lisp")'
2.3) follow the instructions with quicklisp to install it on your compiler
2.4) go into ~/quicklisp/local-projects
2.5) either unzip your salem-layer-util source code to a folder name salem-layer-util or if you have git type: git clone git://github.com/boshaw/salem-layer-util.git
2.6) run your CL compiler
2.7) type: (ql:quickload "salem-layer-util") to load salem-layer-util

Now that you're at the REPL there's only one thing you really need to know in order to run salem-layer-util:

(slu:run KEYS) or (salem-layer-util:run KEYS) either will work
KEYS are as follow:
Code: Select all
:mode -> :d          Decodes file(s) given throguh ARGS key by list of pathnames
                      default output folder: dout/
      -> :e          Encodes file(s) given through ARGS key by list of pathnames
                      default output folder: dres/
      -> :da         Decodes a set of files within a specified directory into another
      -> :ea         Encodes a set of files within a specified directory into another
      -> :cache->res  Renames a set of files within a specified directory from *.cached to *res

:skip-old -> t       Skip processing old files
          -> nil     (Default-value: t)

:verbose -> t        Be verbose when decoding/encoding
         -> nil      (Default-value: t)

:print-skip -> t     Print skipped files
            -> nil   (default-value: nil)

:layers "path-to-layers" Tells salem-layer-util what layers to load in for decoding/encoding
                          (default-value: "layers/salem")

:args -> '(...)      A list of arguments based on given mode
if :d or :e, :args -> '(file1 file2 file3 ...)
if :da or :ea, :args -> '(input-folder output-folder)
if :cache->res -> '(input-folder)
Note: pathnames to files or folders can be absolute or relative

Note: acceptable values for "layers" key are: "layers/salem" or "layers/hnh" depending on what game you are using it for


=Example Usage=
;;Decode two files res/(1.2).res into dout
Code: Select all
(slu:run :mode :d
         :args '("res/1.res" "res/2.res"))



;;Decode all res/cache files from res folder into dec folder
;; also doesn't skip old files and stays quit mostly
Code: Select all
(slu:run :mode :da
         :skip-old nil
         :verbose nil
         :args '("salem/res" "salem/dec"))

;;does the same as above, but for hnh resource files
Code: Select all
(slu:run :mode :da
         :skip-old nil
         :verbose nil
         :layers "layers/hnh"
         :args '("salem/res" "salem/dec"))



;;Encode only new res/cache files form dec folder into res folder
Code: Select all
(slu:run :mode :ea
         :args '("salem/dec" "salem/new-res"))


If you need help running it just pm me and i'll help you get if you're having an issue understanding the syntax.

=Layer Info=
Note: When i refer to CString or String, I mean a valid UTF-8 string. otherwise you'll encounter errors with both salem-layer-util, and salem-client (see pic below)
Image
The ß displayed in that picture is UTF-8 encoded -> 0xC3 0x9F
The box right after it is ß, but in UTF-16/32 -> 0xDF
The client reads strings in with UTF-8 encoding, thus you get artifacts like that.

So Salem shares some layers with HnH and i'd rather not have rewrite them out, so, for layers:
action,anim,images,tile,tileset,pagina,tooltip,audio,midi, and code
refer to http://www.havenandhearth.com/forum/vie ... 27&t=23458 (under Layer Structure/Info)


=audio2 Layer=
Code: Select all
uint8 - version
if version == 1 || version == 2:
  CString - ID
  If version == 2:
    uint16 - bvol (base volume?)
  Rest of data buffer is the audio clip (ogg format?)


Note: if version != 1 or 2, client will error when loading.

=boneoff Layer=
Code: Select all
CString - NM (name)
Until data buffer empty do:
  uint8 - opcode
  if opcode = 0: (position data -> glTranslate?)
    float - x
    float - y
    float - z
  if opcode = 1: (angle data -> glRotate?)
    float - ang
    float - ax
    float - ay
    float - az
  if opcode = 2:
    CString - bonenm (bone name)
  if opcode = 3:
    float - rx1
    float - ry1
    float - rz1
    CString - orignm (original name)
    CString - tgtnm (target name?)


Note: if opcode is not 0,1,2, or 3 it'll error when loading in the client.

=codeentry Layer=
Code: Select all
Until data buffer empty do:
  uint8 - t
  if t = 1:
    while(true):
      CString - en (widget type string?)
      CString - cn (class constructor?)
      if en.length -> 0 => exit while(true)
  if t = 2:
    while(true):
      CString - ln
      if ln.length -> 0 => exit while(true)
      uint16 - version


Note: if t is not 1 or 2 it'll error when loading.

=font Layer=
Code: Select all
uint8 - version
if version = 1:
  uint8 - type
  if type = 0:
    Rest of data buffer -> truetype-font data


Note: if version != 1 and type != 0, it'll error when loading

=light Layer=
Code: Select all
int16 - id
floatx4 - amb (GL_AMBIENT)
floatx4 - dif (GL_DIFFUSE)
floatx4 - spc (GL_SPECULAR)
Until data buffer empty:
  uint8 - t
  if t = 1:
    float - ac
    float - al
    float - aq
  if t = 2: (positional data?)
    float - x
    float - y
    float - z
  if t = 3:
    float - exp


Note: if t != 1,2, or 3, it'll error when loading

=mat Layer=
Code: Select all
uint16 - id
Until data buffer empty:
  CString - thing
  if thing = "col":
    floatx4 - amb (GL_AMBIENT)
    floatx4 - dif (GL_DIFFUSE)
    floatx4 - spc (GL_SPECULAR)
    float - shine (GL_SHININESS)
    floatx4 - emi (GL_EMISSION)
  if thing = "linear":
    no data, but it makes linear = true -> sets minfilter in TexGL to GL_LINEAR_MIPMAP_LINEAR
  if thing = "mipmap":
    no data, but it makes mipmap = true -> sets magfilter in TexGL to GL_LINEAR
  if thing = "nofacecull":
    no data, but it adds nofacecull as a state (GL_CULL_FACE)
  if thing = "tex":
    uint16 - id
  if thing = "texlink":
    CString - nm (name)
    uint16 - version
    uint16 - id
  if thing = "light":
    CString - light


Note: if thing isn't one of the above, client will error on loading
if light isn't pv,pp,vc,pc or n, client will error on loading
pv => vlights
pp => plights
vc => vcel
pc => pcel
n => no light
The above are different light settings for salem based on shader frag/vert files loftar has made, there used to be a command to change lighting in game, but i'm unaware if it's still there.

=mesh Layer=
Code: Select all
uint8 - fl
uint16 - num
uint16 - matid
if (fl&2) != 0:
  uint16 - id
if (fl&4) != 0:
  uint16 - ref
if (fl&~7) != 0 : ERROR
for 0 -> (num * 3)-1:
  uint16 - ind[i]


Note: & => bitwise and, ~ => bitwise negation
If you change fl, make sure the rest of the data file is correct based on the above conditions of fl

=neg Layer=
Code: Select all
uint16 - cc(x)
uint16 - cc(y)
SKIP OFFSET 4->15
uint8 - en
for 1 -> en-1:
  uint8 - epid
  uint16 - cn
  for 1 -> cn-1:
    uint16 - ep[epid][o][x]
    uint16 - ep[epid][o][y]


=rlink Layer=
Code: Select all
uint8 - t
if t = 0:
  CString - meshnm (mesh name)
  uint16 - meshver
  int16 - meshid
  CString - matnm (mat name)
  uint16 - matver
  int16 - matid
if t = 1:
  CString - nm (name)
  uint16 - version


Note: if t != 0 or 1, it'll error when loading...

=skan Layer=
Code: Select all
uint16 - id
uint8 - fl
uint8 - mode
float - len
if (fl&1) != 0:
  float - nspeed
Until data buffer empty:
  CString - bnm (bone name?)
  uint16 - frames.length
  for 1 -> frames.length-1:
    float - tm
    floatx3 - trans (translation?)
    float - rang    (gl_rotate angle?)
    floatx3 - rax   (gl_rotate specifications?)


Note: mode must be 0,1,2, or 3
0 => WarpMode.ONCE
1 => WarpMode.LOOP
2 => WarpMode.PONG
3 => WarpMode.PONGLOOP

=skel Layer=
Code: Select all
Until data buffer empty:
  CString - bnm (bone name?)
  floatx3 - pos (position)
  floatx3 - rax (gl_rotate specification?)
  float - rang ( gl_rotate angle?)
  CString - bp


=vbuf Layer=
Code: Select all
uint8 - fl
uint16 - num
Until data buffer empty:
  uint8 - id
  if id = 0:
    for 0 -> (num*3)-1:
      float - vertex
  if id = 1:
    for 0 -> (num*3)-1:
      float - normal
  if id = 2:
    for 0 -> (num*2)-1:
      float - texels (tex coordinates?)
  if id = 3:
    uint8 - mba
    while(true):
      CString - bone
      if bone = "": exit from while(true)
      while(true):
        uint16 - run
        uint16 - st
        if run = 0: exit from while(true)
        for 0 -> run-1:
          float - w


Note: id must be 0,1,2 or 3


A good way to figure out what a layer really does, pick some resource that you have access to, change up some things and see how it changes. Some of the layers are more obvious than others.

Re: Salem Layer Util

PostPosted: Thu Mar 28, 2013 2:14 am
by Hans_Lemurson
No longer content to live in the shadows, his great craft known only in whispered secrets, Boshaw emerges and shows himself!

Anyways, thanks for making this utility. However out of date the website Bmer first linked, it did at least connect me to your github page and I was able to download and (after a fair amount of troubleshooting) use your program.

I wrote down some notes as I went about what the necessary steps were to access the gfx files (which is what I was interested in) for editing:
How to run the damn thing.txt wrote:To get a fresh and up to date copy of salem-res.jar, download from:
http://plymouth.seatribe.se/java/salem-res.jar

Extract the files to the main directory of the SLU. (winrar ftw!)

Run the ABCL thing, and then to get started, type
(slu:run)
parentheses included!

To decode the contents of the gfx folder, type the following:
(slu:run :mode :da :args '("res/gfx/" "decoded/gfx/") )
mind the parantheses at the end!
A folder named "decoded" will now contain all of the decoded graphics.

To re-pack/encode the files you modified, type the following:
(slu:run :mode :ea :args '("decoded/gfx/" "custom_res/gfx/") )
Now, a folder named "custom_res" will contain all of your modified and re-packed files. This can be pasted directly into the "[Username]\Salem\" folder that is used by Ender's Client.


It is also apparently necessary for the string of any new file-path to end in a /, otherwise the program has an error. If all the named directories already exist though, then this is not required.

Re: Salem Layer Util

PostPosted: Thu Mar 28, 2013 3:36 am
by boshaw
Hans_Lemurson wrote:It is also apparently necessary for the string of any new file-path to end in a /, otherwise the program has an error. If all the named directories already exist though, then this is not required.


I over looked that, basically i do assume you supply the / at the end otherwise Common Lisp (ensure-directories-exist "...") won't create the directory; However, this should no longer be an issue as I just patched it so it automatically concatenates "/" onto the end of the arguments incase you forget when doing :da and :ea.

Also I forgot to mention something you had stated, which was (slu:run) with no arguments, which displays documentation-string for the function which has all the information needed to basically run it. Thanks for mentioning that.

I added in some scripts to make an executable of the essentials out of the code, this assumes you downloaded sbcl and setup quicklisp. Just run make-sbcl.{bat,sh} and it'll produce you an executable that you can use from the command line, but it does take up quite a bit of space (~50mb on windows). when running it from the command line the arguments are all the same with 2 slight changes:
1) instead of :d :e :da :ea, you type d e da ea (without the ':' basically)
2) :args value is now just a string where items are separated by spaces (ie: '(".test/salem/blah.res" "../blah2.res") => ".test/salem/blah.res ../blah2.res") try not to use pathnames with spaces or you'll get undesirable results since it parses by spaces

If you make an executable, the only folders you then need to keep is the layers folder, specifically layers/salem and layers/hnh (if you do hnh stuff). The rest of the lisp files and folders can be deleted.

Re: Salem Layer Util

PostPosted: Thu Mar 28, 2013 7:43 pm
by Hans_Lemurson
For now, so long as run-abcl.bat "makes it go" I'm content. The most important feature of a program or utility is a "Make it go" button.

Another question: How do you run the :e mode? Your target for the source files is a folder, so how do you make sure to designate this properly? I tried to test a modification to the beach texture, and it produced a 10 byte beach.res file which, needless to say, caused a client crash.

I used
Code: Select all
(slu:run :mode :e :args '("dout/beach.res") )

and it said
Code: Select all
IN:dout/beach.res
OUT:dres//beach.res
Encoding dout/beach.res

Given that the folder itself was called beach.res, might this have caused some ambiguity? Also, what does the // in the "OUT" destination signify?

Re: Salem Layer Util

PostPosted: Thu Mar 28, 2013 9:00 pm
by boshaw
Hans_Lemurson wrote:For now, so long as run-abcl.bat "makes it go" I'm content. The most important feature of a program or utility is a "Make it go" button.

Another question: How do you run the :e mode? Your target for the source files is a folder, so how do you make sure to designate this properly? I tried to test a modification to the beach texture, and it produced a 10 byte beach.res file which, needless to say, caused a client crash.

I used
Code: Select all
(slu:run :mode :e :args '("dout/beach.res") )

and it said
Code: Select all
IN:dout/beach.res
OUT:dres//beach.res
Encoding dout/beach.res

Given that the folder itself was called beach.res, might this have caused some ambiguity? Also, what does the // in the "OUT" destination signify?


The issue was actually the same as :ea and :da, :ea technically doesn't use :e, they are slightly different. I added in the implicit "/" ending for :e arguments in case someone forgets, so your above code should work now with the latest source files (mainly just salem-layer-util.lisp).

The "//" is just the result of concatenating an extra "/" onto the OUT folder, nothing special to say the least.

Re: Salem Layer Util

PostPosted: Sun Mar 31, 2013 7:52 am
by Hans_Lemurson
Any idea what that the "flavprob" variable means in: res\gfx\tiles\apath.res\tileset

(It's the same in all the tile resources as I'm sure you know, but that's a concrete example)

Also, your description about the Layers in your first post on this thread seems to be missing the Tiles and Tileset portion. I looked there for clarification, but found them missing.

Re: Salem Layer Util

PostPosted: Sun Mar 31, 2013 4:29 pm
by EnderWiggin
Hans_Lemurson wrote:Any idea what that the "flavprob" variable means in: res\gfx\tiles\apath.res\tileset
I believe it is probability of flavor objects (like some random grass bits on plains) to spawn on this tile set. Those flavor objects are spawned client-side.

Re: Salem Layer Util

PostPosted: Sun Mar 31, 2013 5:19 pm
by boshaw
Hans_Lemurson wrote:Any idea what that the "flavprob" variable means in: res\gfx\tiles\apath.res\tileset

Well in salem sources it appears to be used only once in MCache:

Code: Select all
if(rnd.nextInt(set.flavprob) == 0){
    Resource r = set.flavobjs.pick(rnd);
    double a = rnd.nextDouble() * 2 * Math.PI;
    Gob g = new FlavObj(c.add(tc).mul(tilesz).add(tilesz.div(2)), a);
    g.setattr(new ResDrawable(g,r));
    Coord cc = c.div(cutsz);
    fo[cc+x + (cc.y * cutn.x)].add(g);
}

Which is located in a `makeflavor' function

so what Ender said seems to be the answer.


Hans_Lemurson wrote:Also, your description about the Layers in your first post on this thread seems to be missing the Tiles and Tileset portion. I looked there for clarification, but found them missing.

those two can be found at http://www.havenandhearth.com/forum/vie ... 27&t=23458
tileset is below anim and above pagina
tile is below image and above neg

Re: Salem Layer Util

PostPosted: Mon Apr 01, 2013 2:28 am
by Hans_Lemurson
What do you need to do to actually get your modified graphics recognized by Ender's client?

I have been playing around with beach.res for my testing, and I found that no graphical change I made showed up on the beaches when I loaded the game. I also investigated Chiprel's LowGFX pack and found that it's beach textures didn't work either. In fact, about half of the textures in his pack are non-functional and display the default instead.

UserNM's 11x11 texture pack does however work for all tiles. Replacing it's beach texture with an 8x8 test pattern worked perfectly without problem.

I cannot for the life of me figure out what the difference is between these, why the client accepts Usernm's .res files but rejects most of those provided by Chiprel. The biggest obvious difference is that UserNM's stuff uses single image tilesets, but would that really make so huge a difference?

Do I need to modify the "tileset" data somehow?
What about the META INF files?

Also, what do you do about all the textures that are NOT in salem-res.jar? The current Dirt and Clay textures, among others, are nowhere to be found. Usernm was able to include a dirt.res which worked, but Chiprel did not include anything, presumably because he couldn't find the existing texture to reduce.