Update nacp parsing

[Dimensional]

Recently a game was updated to version 1.9.0 on the Switch, and this update added a 17th language to what it supports. This of course breaks the current structure for reading control.nacp, however what was found out is that now the supported languages are compressed into a small zlib binary blob at the start of the file, while retaining the remainder of the standard data structure, ie everything after 0x3000 is still pretty much the same.

By reading offset 0x3215, the reserved byte is now 1, which may be a compression flag. Reading 2 bytes at 0x0 gives a possible compressed size, and if you read from 0x2+size, running it through zlib decompress with a wbits of -15, you'll decompress into memory 0x6000 bytes worth of data that has the same structure as classic nacp, allowing for 32 language slots instead of the original 16.

Certain tools rely on libnx for reading the nacp file, so this is an issue/enhancement to have this patched, now that there are 2 apparent versions of title lists.

Another formerly reserved flag was at offset 0x3404, which now has 4 non-zero bytes. I thought those were language flags, but I'm unable to determine as it's not matching up with what is in the new structure.

[yellows8]

What game is this?

[Dimensional]

Breath of the Wild. Updated to 1.9.0 the other day on Switch 1.

[DarkMatterCore]

For what it's worth, I have updated my title cache library to add support for these edge cases, but I've had to override multiple libnx values and do gruesome pointer operations to achieve it.

Actual support for compressed title areas and Polish + Thai languages within libnx would be much appreciated.

[Dimensional]

From the looks of it, Polish comes before Thai in the language listing. Looking over the title structure shows 16 titles in local languages, then the 17th is empty, then the 18th has data.

[masagrator]

Since HOS is reading it correctly it must mean we miss some ns function similar to nsGetApplicationDesiredLanguage that instead of pointer returns uncompressed NacpLanguageEntry or somehow force nsGetApplicationControlData2/something else to return uncompressed one.

nsGetApplicationControlData2 still returns compressed NacpLanguageEntrys with flag1 0xFF and flag2 0.

I bet such function will discard last 0x3000 bytes of uncompressed data since they are not supported on S1

Edit:

• 0x0 + 0x0 returns the same data as 0xFF + 0x0
• 0x0 + 0x1 returns Nacp with compressed data for Switch 2. If game doesn't have one, it returns error. Also after decompressing is 0x6000 bytes.

It seems like it doesn't matter what we put in first flag in terms of returned data. If we put anything else than 0 or 1 to second flag, it returns error.

[DarkMatterCore]

Since HOS is reading it correctly it must mean we miss some ns function similar to nsGetApplicationDesiredLanguage that instead of pointer returns uncompressed NacpLanguageEntry or somehow force nsGetApplicationControlData2/something else to return uncompressed one.

There must likely exist such a call, I agree. There are references to some specific zlib error strings within ns.

What's interesting is that, even if HOS doesn't support these new languages under Switch 1, the NACP SupportedLanguage bitflag reflects them (e.g. bits 16 and 17 being used in BotW and SMP Jamboree) and extra icons are being added for them (e.g. /icon_Thai.dat).

Edit:

• 0x0 + 0x0 returns the same data as 0xFF + 0x0
• 0x0 + 0x1 returns Nacp with compressed data for Switch 2. If game doesn't have one, it returns error. Also after decompressing is 0x6000 bytes.

I think this perfectly demonstrates that the second integer in nsGetApplicationControlData2 represents the Application Control Data (ACD) Index you wish to request, which is indeed a NACP field that was recently introduced.

Still no idea about the first integer, though. But thanks for looking into this.

[masagrator]

There are references to some specific zlib error strings within ns.

I checked that and they exist in my 21.1.0 dump, but they don't in 20.1.1.

With 21.0.0 they introduced 5 new functions to IReadOnlyApplicationControlDataInterface, so one of them may be it.

[masagrator]

__int64 __fastcall sub_710008B310(unsigned __int64 a1, unsigned __int16 *a2)
{
  _BYTE *v4; // x8
  __int64 v5; // x9

  if ( *((_BYTE *)a2 + 0x3215) )
  {
    if ( *((_BYTE *)a2 + 0x3215) == 1 )
    {
      if ( (Decompress(a1, 0x6000LL, a2 + 1, *a2, a1 + 0x6000, 7160LL) & 1) != 0 )
        return 0LL;
      else
        return 0x7010LL;
    }
    else
    {
      return 0x6E10LL;
    }
  }
  else
  {
    sub_7100284970(a1, a2, 0x3000uLL);
    v4 = (_BYTE *)(a1 + 0x3000);
    v5 = 16LL;
    do
    {
      *v4 = 0;
      --v5;
      v4[512] = 0;
      v4 += 768;
    }
    while ( v5 );
    return 0LL;
  }
}

So yeah, 0x3215 is the offset Switch looks at.

[masagrator]

So qlaunch seems to use cmd 19 which later uses that function ^

Per 21.0.0 (since 21.1.0 is broken on yls8 bot):

      19:    {"vt":  0xB8, "lr": 0x7100169330, "inbytes":  0x10, "outbytes":   0xC, "buffers": [6]},

Which seems to be another version of NsGetApplicationControlData. Got enough time to try it by just replacing cmdid in nsGetApplicationControlData2 to 19 + using struct for Out with correct 0xC size and with flags 0xFF and acd index 0 it returns error. I will continue tomorrow.

[Maufeat]

Per 21.0.0 (since 21.1.0 is broken on yls8 bot):

      19:    {"vt":  0xB8, "lr": 0x7100169330, "inbytes":  0x10, "outbytes":   0xC, "buffers": [6]},

Which seems to be another version of NsGetApplicationControlData. Got enough time to try it by just replacing cmdid in nsGetApplicationControlData2 to 19 + using struct for Out with correct 0xC size and with flags 0xFF and acd index 0 it returns error. I will continue tomorrow.

I've partially reversed it via qlaunch since FW 21.0.0 it calls cmd 19 - So I tried to reverse it and got it to work to display the data correctly. So maybe it helps: https://git.eden-emu.dev/eden-emu/eden/pulls/3153

[masagrator]

Ok, got it to work, but it still returns compressed nacp. flag2/acd_idx seems to do nothing: 0, 1, 2, 0xFF - all return Switch 1 data. Testing further.

Edit: another discovery - BIT(0) in flag1 returns smaller icon, used by qlaunch when pressing + on app.
Also cmd 19 now accepts struct of size at least 0x1D000 instead of 0x24000, reducing space for icon.

Edit2: It seems like cmd 19 doesn't decompress nacp when controldata is already in cache while using Storage option. StorageOnly returns also only compressed nacp.

Edit3: cmd 18, 20, 21 and 22 are also variations of NsGetApplicationControlData, by default all of them return compressed nacp.

22 returns Switch 1 data + smaller icon if flag1 = 0, if flag1 = 1 it returns Switch 2 Edition data with full icon.
21 returns only Switch 1 data + smaller icon.
20 works similar to 19.
18 returns only Switch 1 data + full icon.

Edit4: After forcing sysmodule to go through decompressing, homebrew still returns compressed data. it seems it decompresses data and stores title + publisher somewhere else while not parsing that data anywhere for return.

So yeah, I think we need to implement zlib decompressor manually or figure out how to retrieve title from that new cache.

Edit5: Yeah, qlaunch gets compressed ControlData too

Thread 6 "DefaultWorker" hit Breakpoint 50, 0x0000002d396d68a4 in ?? ()
=> 0x0000002d396d68a4:  d101c3ff        sub     sp, sp, #0x70
(gdb) p/x $x2
$32 = 0x2d3cee41a0
(gdb) p/x $x3
$33 = 0x1d000
(gdb) x/8x $x2
0x2d3cee41a0:   0x00000000      0x00000000      0x00000000      0x00000000
0x2d3cee41b0:   0x00000000      0x00000000      0x00000000      0x00000000
(gdb) b *$lr
Breakpoint 52 at 0x2d39a38dc8
(gdb) c
Continuing.

Thread 6 "DefaultWorker" hit Breakpoint 52, 0x0000002d39a38dc8 in ?? ()
=> 0x0000002d39a38dc8:  34000200        cbz     w0, 0x2d39a38e08
(gdb) x/8x 0x2d3cee41a0
0x2d3cee41a0:   0xd6ed0145      0x60c24ad1      0x1ef1c014      0x97473ce5
0x2d3cee41b0:   0x4257475d      0x4ae09dd0      0x0f621441      0x4b213730
(gdb) x/8b 0x2d3cee41a0
0x2d3cee41a0:   0x45    0x01    0xed    0xd6    0xd1    0x4a    0xc2    0x60
(gdb) x/32b 0x2d3cee41a0
0x2d3cee41a0:   0x45    0x01    0xed    0xd6    0xd1    0x4a    0xc2    0x60
0x2d3cee41a8:   0x14    0xc0    0xf1    0x1e    0xe5    0x3c    0x47    0x97
0x2d3cee41b0:   0x5d    0x47    0x57    0x42    0xd0    0x9d    0xe0    0x4a
0x2d3cee41b8:   0x41    0x14    0x62    0x0f    0x30    0x37    0x21    0x4b
(gdb)

[masagrator]

Ok, it seems that qlaunch has zlib decompressor implemented inside. So I guess it uses its own internal cache instead of something handled by NS.

Thread 6 "DefaultWorker" hit Breakpoint 57, 0x0000002d39a38dc8 in ?? ()
=> 0x0000002d39a38dc8:  34000200        cbz     w0, 0x2d39a38e08
(gdb) p/x $38
$40 = 0x2d3cee41a0
(gdb) x/32x $38
0x2d3cee41a0:   0x45    0x01    0xed    0xd6    0xd1    0x4a    0xc2    0x60
0x2d3cee41a8:   0x14    0xc0    0xf1    0x1e    0xe5    0x3c    0x47    0x97
0x2d3cee41b0:   0x5d    0x47    0x57    0x42    0xd0    0x9d    0xe0    0x4a
0x2d3cee41b8:   0x41    0x14    0x62    0x0f    0x30    0x37    0x21    0x4b
(gdb) rwatch *(short*)0x2d3cee41a0
Hardware read watchpoint 58: *(short*)0x2d3cee41a0
(gdb) c
Continuing.

Thread 6 "DefaultWorker" hit Hardware read watchpoint 58: *(short*)0x2d3cee41a0

Value = 325
0x0000002d396d6d64 in ?? ()
=> 0x0000002d396d6d64:  91401a64        add     x4, x19, #0x6, lsl #12
(gdb) p/x 0x0000002d396d6d64 - $main
$41 = 0x2d6d64
(gdb)

__int64 __fastcall sub_71002D6D38(__int64 a1, unsigned __int16 *a2)
{
  _BYTE *v4; // x8
  __int64 v5; // x9

  if ( *((_BYTE *)a2 + 0x3215) )
  {
    if ( *((_BYTE *)a2 + 0x3215) == 1 )
    {
      if ( (Decompress(a1, 0x6000LL, a2 + 1, *a2, a1 + 0x6000, 7160LL) & 1) != 0 )
        return 0LL;
      else
        return 0x7010LL;
    }
    else
    {
      return 0x6E10LL;
    }
  }
  else
  {
    sub_7100484BF0(a1, a2, 0x3000LL);
    v4 = (_BYTE *)(a1 + 0x3000);
    v5 = 16LL;
    do
    {
      *v4 = 0;
      --v5;
      v4[512] = 0;
      v4 += 768;
    }
    while ( v5 );
    return 0LL;
  }
}

[masagrator]

If someone wants helper, this is what I'm using to decompress data. It will discard last 0x3000 bytes.
In combination with this PR: #716 and switch-zlib from pacman

I didn't make it fully on stack because I'm using this in non main thread which has very small stack.

// Returns true if decompressed correctly.
bool nacp_decompress(NsApplicationControlData* appControlData)
{	
	Bytef* temp_buffer = (Bytef*)calloc(32, sizeof(NacpLanguageEntry));
	if (!temp_buffer)
		return false;

    z_stream strm = {0};

    if (inflateInit2(&strm, -15) != Z_OK) {
		free(temp_buffer);
        return false;
	}

    strm.next_in   = (Bytef*)appControlData->nacp.lang_data.compressed_data.buffer;
    strm.avail_in  = appControlData->nacp.lang_data.compressed_data.buffer_size;
    strm.next_out  = temp_buffer;
    strm.avail_out = sizeof(NacpLanguageEntry[32]);

    int ret = inflate(&strm, Z_FINISH);
	bool res = false;
    if (ret == Z_STREAM_END) {
		memcpy(&appControlData->nacp, temp_buffer, 0x3000);
		res = true;
	}

    inflateEnd(&strm);
	free(temp_buffer);
    return res;
}

[yellows8]

Haven't tried using this but this should get the decompressed entry: https://switchbrew.org/wiki/NS_services#Cmd421