enh(ci): allow to ignore certain preprocessor define directive when attempting to compile shaders

Description

We (and be we I really mean I) sometimes write shaders with exclusive preprocessor #definecombinations that are not allowed to coexist (this is used as a way to select a path in certain complex shaders, like in volume rendering).

Typical code looks like this:

//-----------------------------------------------------------------------------
// Sanity checks

#if defined(SIGHT_PATH_A)

    #if defined(SIGHT_PATH_B)
        #error Invalid configuration. Expected at most one of SIGHT_PATH_A, SIGHT_PATH_B.
    #endif

#elif defined(SIGHT_PATH_B)

    #if defined(SIGHT_PATH_A)
        #error Invalid configuration. Expected at most one of SIGHT_PATH_A, SIGHT_PATH_B.
    #endif

#else
    #error Missing preprocessor define. Expected SIGHT_PATH_A or SIGHT_PATH_B.
#endif

// ...

Such code fails to pass glslang-validator stage in the CI, so currently files with such code must be added to .sheldonignore. And this is not good for code quality check.

Side note: the way we parse shader files names to get the stage doesn't work well with recommendations of glslang. Since the fix is only 7 lines long, I propose to add it along the way.

Proposal

Allow to ignore certain preprocessor #define directive when attempting to compile shaders

• Parse comments, in the style of cspell:
  • Edit all_pp_defines_combinations in hooks/glslang_validator.py so it also looks for comment matching a certain format.
    • Format: // sheldon: glslang require-exclusive(define_a, define_b, define_c) / resp. with discard-define(a, b, c)
    • Discard all combinations that would contain more than one of the tokens in require-exclusive() and nones without one of these tokens (resp., for discard-define(...), simply remove the defines to discard from the list of #defines extracted before computing all the combinations)

Support .vert.glsl & co

• Edit get_shader_stage in hooks/glslang_validator.py so it parses the extension before the name (doable in 7 lines, see the patch below)

Functional specifications

• Users should be able to specify directly in their shaders what #defines should be discarded (not added by the hook) and mutually exclusive combinations.
  • Mutually exclusive can be specified with //sheldon:glslang require-exclusive(define_a, define_b, define_c). Spaces don't matter. In that case, only one of and exactly one of define_a, define_b, define_c will be added at a time.
  • #defines that must not be added are can be specified with a similar comment, following the format // sheldon:glslang discard(define_d, define_e). None of the tokens specified will be considered by the hook.
• .vert.glsl / .frag.glsl & co. should be correctly compiled as vertex / fragment / ... shaders.

Technical specifications

all_pp_defines_combinations

• Parse the contents and look for comments in the following format:
  • // sheldon: glslang require-exclusive(...)
  • Regex:
    • Find the content of the comments starting by // sheldon: glslang with r'^(?://\s*sheldon\s*:\s*glslang\s*)(.*)'
    • Find the require-exclusive matches with r'(?:\s*require-exclusive\()(.*?)(?:\))'
    • Strict analogue for discard-define
  • Remove all tokens found in discard-define from the list of #defines to add to reduce the total number of combinations.
• Build the combinations.
• Remove ones containing combinations missing all tokens and ones containing more than one token in all_pp_defines_combinations, such as:

    for define in range(len(command_defines) + 1):
        for comb in itertools.combinations(command_defines, define):

            skip = False

            for exclusive_group in pp_defines.exclusive_defines:
                # Only keep combinations with exactly one
                if(len(set(comb).intersection(set(exclusive_group))) != 1):  # Also skip combinations with all missing
                    skip = True
                    break

            if(not skip):
                yield comb

get_shader_stage

7 extra lines + comments, which should do the job

glslang_file_extensions.patch

Test plan

• Remove affected shaders from .sheldonignore and look at the CI.
• Test cases for the regex testing require-exclusive:

// sheldon glslang: require-exclusive(aaa, bbb) // NOK - No match
// sheldon glslang: require-exclusive(aaa) // NOK - No match
// sheldon: glslang require-exclusive(aaa, bbb) // OK - 1 match
// sheldon: glslang require-exclusive(aaa) // OK - 1 match
// sheldon : glslang require-exclusive(aaa, bbb) // OK - 1 match
// sheldon : glslang require-exclusive(aaa) // OK - 1 match
// sheldon:    glslang require-exclusive(aaa, bbb) // OK - 1 match
// sheldon    :     glslang    require-exclusive(aaa) // OK - 1 match
//    sheldon: glslang require-exclusive(a, b, c,d ) // OK - 1 match
// sheldon: glslang require-exclusive(aaa, bbb) // OK - 1 match
// sheldon: glslang require-exclusive(aaa) // OK - 1 match
// sheldon: glslang require-exclusive(aaa, bbb), require-exclusive(aaa) // OK - 2 matches
// sheldon: glslang require-exclusive(aaa), require-exclusive(c) // OK - 2 matches
// sheldon: glslang require-exclusive(aaa), require-exclusive(c), require-exclusive(d) // OK - 3 matches
// sheldon: glslang require-exclusive(aaa),             require-exclusive(c),                       require-exclusive(d) // OK - 3 matches

require-exclusive(ffffff) // NOK, no match