ARZ Extractor

http://forum.xentax.com/viewtopic.php?f=10&t=12037
https://gitlab.com/atom0s/grimarz

To extract database.arz

grimarz.exe <path for database.arz>

The extracted files are in \extracted

-------------------------------------------------------------------------------------------------

struct GRIMDAWN_ARZ_V3_HEADER
{
    unsigned int Unknown; // Assumed version check.
    unsigned int RecordTableStart;
    unsigned int RecordTableSize;
    unsigned int RecordTableEntryCount;
    unsigned int StringTableStart;
    unsigned int StringTableSize;
};

values are big-endian, otherwise they make no sense

02000300 unknown / version

08DA4002 -> 37,804,552 (240DA08)
8DD60D00 -> 906,893    (24EB095) [37,804,552 + 906,893 = 38,711,445, so records are right in front of strings]
445A0000 -> 23,108

95B04E02 -> 38,711,445 (24EB095)
C87F2B00 -> 2,850,760  (27A305D) [38,711,445 + 2,850,760 = 41,562,205, so 16 bytes less than file size]

38,711,445 + 2,850,760 = 41,562,205 < 41,552,221 (database.arz size)
makes sense as the last 16 bytes are some 'random garbage' while the ones before it are strings

String Table Format

0x00000000    unsigned int    Number Of Strings In Table
    ->        unsigned int    String Length
    ->        char[]          String
    ->        unsigned int    String Length
    ->        char[]          String

... etc until number of strings is met.

So the first 4 bytes are the number of strings and then it is the well known 4 bytes length
followed by the string and so on, confirmed

A0EC0000 -> 60,576 strings

Record Table Format

0x00000000    unsigned int    File Path String Id
0x00000004    unsigned int    String Length
0x00000008    char[]          String (Record Type) [String is not null terminated!]
0x000000??    unsigned int    Record Data Position
0x000000??    unsigned int    Record Data Length (Compressed)
0x000000??    unsigned int    Record Data Length (Decompressed)
0x000000??    unsigned int64  Record File FILETIME

File Path String Id = String index in String table (0 is first string). 
A '/' in the string is a directory separator for the file path

Record Data Position ?
00000000
E5010000 -> 485
A0020000 -> 672
A7F1164D681BD001

Once the data has been decompressed, the format is

0x00000000    unsigned short  Data Type [See Below]
0x00000002    unsigned short  Value Count
0x00000004    unsigned int    String Key Id
0x00000008    unsigned int    Value [To be read as int/float/String Key ID/bool as needed.]
...

Data Type can be

-> 0x0000 == Int32 Value
-> 0x0001 == Float Value
-> 0x0002 == String Value [Value will be index inside of string table of the string.]
-> 0x0003 == Boolean Value

If Count > 1, then additional values follow the String Key ID, all of the same Data Type

===============================================================================================================================================

ARC Extractor

http://forum.xentax.com/viewtopic.php?f=10&t=12041
https://gitlab.com/atom0s/grimarc

file
{
    header
    file_data
    file_record_table
    file_table_of_contents
}

File Header

/**
* @brief ARC File Header (v3)
*/
struct ARC_V3_HEADER
{
    unsigned int Magic; // ARC
    unsigned int Version;
    unsigned int NumberOfFileEntries; // total number of files this .arc contains -> number of strings, number of ToCs
    unsigned int NumberOfDataRecords; // total number of file parts within the second part of this .arc file (NumberOfDataRecords * 12) = RecordTableSize
    unsigned int RecordTableSize;     // total size of the file parts table. (Each entry is 12 bytes long.)
    unsigned int StringTableSize;     // total size of the string table
    unsigned int RecordTableOffset;   // offset to the file parts table
	                                     -> String offset = file part offset + file part table size
										 -> ToC offset    = String offset + string table size
};

After the header is the file data parts. This chunk is handled by the next two parts of the file.

The record table defines the file parts in the chunk of data before it. The format is

struct ARC_V3_FILE_PART
{
    unsigned int PartOffset;
    unsigned int CompressedSize;
    unsigned int DecompressedSize;
};

Each entry points to a part of a file. (Or a whole file if its not parted.) If the compressed size matches the decompressed size,
the part is not considered compressed and can be dumped as-is

Following this table, is the string table. Each string is the name of a file within the archive. The string table is just a block
of strings, null terminated in a row. The position of the string is the index of the file within the archive.
So the first entry in the ToC table uses the first string in the string table, the second uses the second and so on. 

The last block of data in the file is the files table of contents. This is the major part of the file that is used to read and
understand the rest of the file. The ToC attaches to the record table to know which blocks of data to read. The ToC is formatted
like this

struct ARC_V3_FILE_TOC_ENTRY
{
    unsigned int        EntryType;
    unsigned int        FileOffset;
    unsigned int        CompressedSize;
    unsigned int        DecompressedSize;
    unsigned int        Unknown0001; // Possible timestamp?
    unsigned __int64    FileTime;
    unsigned int        FileParts;
    unsigned int        FirstPartIndex;
    unsigned int        StringEntryLength;
    unsigned int        StringEntryOffset;
};

===============================================================================================================================================

Player and Stash format / code

http://www.lost.org.uk/grimdawn/

===============================================================================================================================================

Crystal Clear Icon Set

http://www.softicons.com/system-icons/crystal-project-icons-by-everaldo-coelho
http://www.crystalxp.net/galerie/en.id.4538-crystal-project-png-everaldo-icons-a-png.htm

LGPL
http://www.gnu.org/licenses/lgpl.html

===============================================================================================================================================

LZ4 Compression

https://github.com/jpountz/lz4-java/
https://github.com/jpountz/lz4-java/releases
http://repo1.maven.org/maven2/net/jpountz/lz4/lz4/
http://mvnrepository.com/artifact/net.jpountz.lz4/lz4

Apache License
https://github.com/jpountz/lz4-java/blob/master/LICENSE.txt

add lz4-1.3.0.jar to the project
- right click the project
- Properties - Libraries, tab Compile
- Add JAR/Folder

===============================================================================================================================================

DDS Reader

https://github.com/npedotnet/DDSReader
https://github.com/npedotnet/DDSReader/releases

MIT
http://opensource.org/licenses/mit-license.php
https://github.com/npedotnet/DDSReader/blob/master/LICENSE


DWORD               dwMagic; = 0x20534444 (reversed to 0x53444420)
DDS_HEADER          header;

If the DDS_PIXELFORMAT dwFlags is set to DDPF_FOURCC and dwFourCC is set to "DX10" an additional DDS_HEADER_DXT10
structure will be present to accommodate texture arrays or DXGI formats that cannot be expressed as an RGB pixel
format

DWORD               dwMagic;
DDS_HEADER          header;
DDS_HEADER_DXT10    header10;

A pointer to an array of bytes that contains the main surface data.

BYTE bdata[]

A pointer to an array of bytes that contains the remaining surfaces such as; mipmap levels, faces in a cube map,
depths in a volume texture.

BYTE bdata2[]

typedef struct {
  DWORD           dwSize;
  DWORD           dwFlags;
  DWORD           dwHeight;
  DWORD           dwWidth;
  DWORD           dwPitchOrLinearSize;
  DWORD           dwDepth;
  DWORD           dwMipMapCount;
  DWORD           dwReserved1[11];
  DDS_PIXELFORMAT ddspf;
  DWORD           dwCaps;
  DWORD           dwCaps2;
  DWORD           dwCaps3;
  DWORD           dwCaps4;
  DWORD           dwReserved2;
} DDS_HEADER;

struct DDS_PIXELFORMAT {
  DWORD dwSize;
  DWORD dwFlags;
  DWORD dwFourCC;
  DWORD dwRGBBitCount;
  DWORD dwRBitMask;
  DWORD dwGBitMask;
  DWORD dwBBitMask;
  DWORD dwABitMask;
};

===============================================================================================================================================

http://www.dafont.com/vinque.font?psize=xs

===============================================================================================================================================

Expression Evaluator

http://projects.congrace.de/exp4j/index.html

Apache
http://projects.congrace.de/exp4j/license.html

===============================================================================================================================================

Windows.ini file support

http://ini4j.sourceforge.net/

===============================================================================================================================================

Convert PNG to Icon, for Excelsior JET installer

http://icoconvert.com/

===============================================================================================================================================

Image Thumbnails (for resized component overlays)

https://github.com/coobird/thumbnailator
https://code.google.com/p/thumbnailator/

MIT
http://opensource.org/licenses/mit-license.php
https://github.com/coobird/thumbnailator/blob/master/LICENSE

BufferedImage newImage = Thumbnails.of(otherImage)
                             .size(SMALL_SIZE, SMALL_SIZE)
                             .asBufferedImage();
	


/**
* Resizes an image using a Graphics2D object backed by a BufferedImage.
* @param srcImg - source image to scale
* @param w - desired width
* @param h - desired height
* @return - the new resized image
*/
private BufferedImage getScaledImage(BufferedImage src, int w, int h){
    int finalw = w;
    int finalh = h;
    double factor = 1.0d;
    if(src.getWidth() > src.getHeight()){
        factor = ((double)src.getHeight()/(double)src.getWidth());
        finalh = (int)(finalw * factor);                
    }else{
        factor = ((double)src.getWidth()/(double)src.getHeight());
        finalw = (int)(finalh * factor);
    }   

    BufferedImage resizedImg = new BufferedImage(finalw, finalh, BufferedImage.TRANSLUCENT);
    Graphics2D g2 = resizedImg.createGraphics();
    g2.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BILINEAR);
    g2.drawImage(src, 0, 0, finalw, finalh, null);
    g2.dispose();
    return resizedImg;
}

-----

    private float xScaleFactor, yScaleFactor = ...;
    private BufferedImage originalImage = ...;

    public void paintComponent(Graphics g) {
        Graphics2D g2 = (Graphics2D)g;
        int newW = (int)(originalImage.getWidth() * xScaleFactor);
        int newH = (int)(originalImage.getHeight() * yScaleFactor);
        g2.setRenderingHint(RenderingHints.KEY_INTERPOLATION,
                            RenderingHints.VALUE_INTERPOLATION_BILINEAR);
        g2.drawImage(originalImage, 0, 0, newW, newH, null);
    }

===============================================================================================================================================

Java Databases

http://www.oracle.com/technetwork/java/javadb/overview/index.html

Java DB is Oracle's supported distribution of the Apache Derby open source database. Java DB is included in the JDK.

Outdated compared to the latest Derby release

https://db.apache.org/derby/
https://db.apache.org/derby/derby_downloads.html
https://db.apache.org/derby/manuals/index.html

http://hsqldb.org/
http://hsqldb.org/web/hsqlDocsFrame.html

HSQLDB supports the widest range of SQL Standard features seen in any open source database engine.
HSQLDB has been constantly developed over 12 years and is used as a database and persistence engine in over 1700
Open Source Software projects and many commercial products.

Used by Open Office.

http://www.h2database.com/html/main.html

By same developer as HSQLDB, but somwhat faster, esp. in embedded scenarios with sequential access (but not random),
see http://www.h2database.com/html/performance.html

http://sourceforge.net/projects/db4o/

Object DB


http://docs.oracle.com/javase/tutorial/jdbc/

===============================================================================================================================================

Version Management

http://mercurial.selenic.com/

http://hgbook.red-bean.com/read/

https://netbeans.org/kb/docs/ide/mercurial.html

===============================================================================================================================================

http://www.miglayout.com/
GPL / BSD

Look & Feel

https://seaglass.googlecode.com/svn/doc/index.html
Apache
http://www.apache.org/licenses/LICENSE-2.0

http://www.jgoodies.com/downloads/libraries/
BSD

A custom look and feel for Swing.
http://grepcode.com/project/repo1.maven.org/maven2/com.github.insubstantial/substance/
Substance customizations to some SwingX components.
http://grepcode.com/project/repo1.maven.org/maven2/com.github.insubstantial/substance-swingx
Substance customizations to some Flamingo components.
http://grepcode.com/project/repo1.maven.org/maven2/com.github.insubstantial/substance-flamingo
A Ribbon component and other supporting components.
http://grepcode.com/project/repo1.maven.org/maven2/com.github.insubstantial/flamingo
Trident Animation Library.
http://grepcode.com/project/repo1.maven.org/maven2/com.github.insubstantial/trident
A swing component enhancement framework.
http://grepcode.com/project/repo1.maven.org/maven2/com.github.insubstantial/laf-plugin
Swing component enhancements.
http://grepcode.com/project/repo1.maven.org/maven2/com.github.insubstantial/laf-widget

http://www.pagosoft.com/projects/pgslookandfeel/

https://java.net/projects/liquidlnf
http://sourceforge.net/projects/liquidlnf/
http://sourceforge.net/projects/oalnf/

http://www.randelshofer.ch/quaqua/
LGPL / BSD, but limited to Apple because of the L&F copying Apple's

===============================================================================================================================================

SwingX

https://swingx.java.net/
https://java.net/projects/swingx/downloads/directory/releases

===============================================================================================================================================

Formats

-------------------------------------------------------------------------------------------------

Items are in

records\items

Class
  defines type of item
 
  AreaOfInterest
  ArmorJewelry_Amulet
  ArmorJewelry_Medal
  ArmorJewelry_Ring
  ArmorProtective_Chest
  ArmorProtective_Feet
  ArmorProtective_Hands
  ArmorProtective_Head
  ArmorProtective_Legs
  ArmorProtective_Shoulders
  ArmorProtective_Waist
  ItemArtifact
  ItemArtifactFormula
  ItemEnchantment
  ItemRelic                  " Component
  ItemTransmuter
  ItemUsableSkill
  OneShot_Food
  OneShot_Gold
  OneShot_PotionHealth
  OneShot_PotionMana
  OneShot_Sack
  OneShot_Scroll
  QuestItem
  WeaponArmor_Offhand        " Focus item
  WeaponArmor_Shield
  WeaponHunting_Ranged1h
  WeaponHunting_Ranged2h
  WeaponMagical_Staff
  WeaponMelee_Axe
  WeaponMelee_Axe2h
  WeaponMelee_Dagger
  WeaponMelee_Mace
  WeaponMelee_Mace2h
  WeaponMelee_Scepter
  WeaponMelee_Sword
  WeaponMelee_Sword2h
  
  LootRandomizer (completion bonus, prefix, suffix, modifier)
  LootRandomizerTable (set of bonuses, from blacksmith, per faction)

armorClassification
  Type of armor
  
  Caster
  Light
  Heavy
  null

augmentSkillLevel1
  + skill level (int)

augmentSkillName1
  skill for which the level is increased

  records/skills/playerclass03/curse1.dbr
  
  in that file, find tag buffSkillName (records/skills/playerclass03/curse1_buff.dbr)
  in that file, find tag skillDisplayName (tagClass03SkillName06A), the text is in tags_skills.txt

bitmap
  specifies path to the item bitmap (.tex)
  
  items/gearhead/bitmaps/a02_head002.tex
  items/geartorso/bitmaps/a00_torso001.tex
  items/gearlegs/bitmaps/a02_legs02.tex
  items/gearfeet/bitmaps/a04_feet01.tex
  items/gearweapons/swords1h/bitmaps/a00_sword001.tex
  items/gearaccessories/medals/a04_medal_ribbon.tex
  items/gearaccessories/waist/a06_waist_leatherbelt.tex
  items/misc/potion_healtha01up.tex
  
artifactBitmap
  bitmap for artifacts

  items/gearrelics/tier1/tier1_relic_06.tex  

relicBitmap
  bitmap for components

  items/craftingparts/components/component_a48_flintcorebolts_b.tex

bonusTableName
  completion bonuses sets for components and artifacts

  records/items/lootaffixes/completion/completionbonus_a003_weapons.dbr
  records/items/lootaffixes/completionrelics/completionbonus_a004_tier1defense.dbr
  
  in that dbr file, the tags randomizerName## (randomizerName1 and up) define the available completion bonuses
  each bonus is a dbr file again, e.g. records/items/lootaffixes/completion/a34b_piercedmg.dbr

conversionInType
  damage type that gets converted (weapons)

  Physical
  
conversionOutType
  damage type that gets converted to (weapons)

  Chaos
  
conversionPercentage
  percentage of damage type that gets converted (float)

characterDefensiveAbility  
  + Defensive ability (float)

characterOffensiveAbility
  + Offensive ability (float)
  
defensiveProtection
  Armor value (int)
  
dexterityRequirement
  min Dex required (int)

hidePrefixName
  hide prefix (bool, 0 = false)
  
  not in artifacts and components 

hideSuffixName
  hide suffix (bool, 0 = false)

  not in artifacts and components 

intelligenceRequirement
  min Int required (int)

itemClassification
  item rarity

  Common
  Rare " Monster Infrequent
  Epic


  Items
    null
    Broken
    Common
    Magical
    Rare
    Epic
    Legendary
    Quest
  
  Affixes
    null
    Broken
    Magical
    Rare
    Epic

artifactClassification
  additional classification for artifacts

  Lesser
  Greater
  Divine
  null
  
itemLevel
  level of item (int)

itemNameTag
  item name, tag is found in text_en.arc, tags_items.txt

  tagTorsoA000

itemSetName
  Set the item belongs to
  
  records/items/lootsets/itemset_c002.dbr

description
  itemNameTag for artifacts / components / potions / notes / blueprints
  
  tagRelicB003
  tagCompA032Name

itemQualityTag
  item style (prefix of item name, e.g. Exalted ...), tag is found in text_en.arc, tags_items.txt

  Assault
  Crude
  Exalted
  Fortified
  Imbued
  Imperial
  Improvised
  Iron
  Raider
  Reinforced
  Savage
  Scrapmetal
  Solid
  Splintered
  Steel
  Tarnished
  Weathered
  null

itemStyleTag
  item style (prefix of item name, e.g. Worn Cloth), tag is found in text_en.arc, tags_items.txt

  not in artifacts and components 
  
  tagStyle001

  Apprentice
  Assault
  Brigandine
  Bronze
  Chain
  Chainmail
  Cloth
  Combat
  Embroidered
  Engraved
  Exalted
  Field
  Frontier
  Fur-lined
  Gilded
  Heavy
  Hide
  Hinterland
  Imperial
  Infantry
  Leather
  Novice
  Opulent
  Ornate
  Padded
  Patchwork
  Plate
  Plated
  Quilted
  Raider
  Reinforced
  Restored
  Salvaged
  Savage
  Scalemail
  Scrapmetal
  Spiked
  Splinted
  Steel
  Studded
  Worn
  Woven
  null

levelRequirement
  required level for item to drop / equip ? (int)

offensivePhysicalMax
  max physical damage (float)
  
  offensive<Damage Type>Max with <DamageType> in Aether, Chaos, Cold, Fire, Lightning, Physical, Pierce, Poison
  but not Vitality !
  
  Confusion, Convert, Fear, Knockdown, Life, PercentCurrentLife, PierceRatio, SlowAttackSpeed, SlowBleedingDuration, SlowBleeding,
  SlowColdDuration, SlowCold, SlowDefensiveAbilityDuration, ...

offensivePhysicalMin
  min physical damage (float), same damage types as for Max

offensivePierceRatioMin
  pierce % for weapons

quest
  quest item  (bool, 0 = false) ? not set in quest items either
  Quest itms have a tag requiredQuestFile (string) and showQuestItemTag (bool)
  
strengthRequirement
  min Str required (int)

-------------------------------------------------------------------------------------------------

Sets are in

records\items\lootsets

setName
  itemNameTag for item sets (records in records\items\lootsets), tag is found in text_en.arc, tags_items.txt

setDescription
  Set description for item sets (records in records\items\lootsets), tag is found in text_en.arc, tags_items.txt
  
setMembers
  List of items contained in the set for item sets (records in records\items\lootsets)
  
  records/items/gearhead/c007_head.dbr,records/items/geartorso/c007_torso.dbr,records/items/gearlegs/c004_legs.dbr,records/items/gearfeet/c005_feet.dbr

-------------------------------------------------------------------------------------------------


Affixes are in

records\items\lootaffixes

records\items\lootaffixes\crafting\completionbonus_*.dbr are the crafting bonuses of the different factions

completionbonus_apprentice.dbr         -> Duncan (elemental damage, elemental resist)
completionbonus_homestead.dbr          -> Black Legion (chaos + aether resist)
completionbonus_kymonschosen.dbr       -> Kymon's Chosen (fire + lightning damage)
completionbonus_master.dbr             -> Angrim (physical damage)
completionbonus_orderofdeathsvigil.dbr -> Order of Death's Vigil (cold + vitality damage)

randomizerName1,records/items/lootaffixes/crafting/ad01_fireresist.dbr,
randomizerName2,records/items/lootaffixes/crafting/ao11_firedmg.dbr,
randomizerName3,records/items/lootaffixes/crafting/ao13_lightningdmg.dbr,

the other files in this dir are the individual bonuses

itemClassification
  affix rarity
  
  Magical
  Rare
  Epic

  itemClassification,Magical,

levelRequirement
  required level for item to equip

  levelRequirement,70,

lootRandomizerName
  affix name, tag is found in text_en.arc, tags_items.txt

  lootRandomizerName,tagPrefixAA005,


===============================================================================================================================================

Linking items together

baseName     records/items/gearhead/a04_head001.dbr                      -> entry in database.arz
prefixName   records/items/lootaffixes/prefix/b_ar013_ar.dbr             -> entry in database.arz
suffixName   records/items/lootaffixes/suffix/a004b_ch_att_cunphy_02.dbr -> entry in database.arz
modifierName records/items/lootaffixes/crafting/ad09_aetherresist.dbr    -> entry in database.arz
reiicName    records/items/materia/compa_polishedemerald.dbr             -> entry in database.arz
relicBonus   records/items/lootaffixes/completion/a01a_physicaldmg.dbr   -> entry in database.arz

-> search in string table for the corresponding string
   search in records table for the record with strID = index of the string found

===============================================================================================================================================

http://thegrimcompendium.co.uk/forum/index.php
