First off you need to post the name of your controller! There is a huge difference if a controller is not supported & Someone on the forum may have already done the hard work and mapped your device as far as it can go.
If it’s not one of the supported controllers you are not going to be able to make use of the mode functions or other special features.
If it’s not a supported controller then you will only get minimum functionality ie. each button/control will only have one function that you can midi map it to (with the exception of the controller shift,djay shift & 4 deck buttons)
That means that your 4/8 keypads can only be assigned one function (cues etc) and pressing a different mode button will not allow you to assign more functions to those pads as it requires algoriddim to add code to djay for each supported controller (they add modifiers for each mode button). You would only be able to use the mode buttons for other single functions that don’t affect the pads.
If your pads don’t light up when you assign them to hot cues etc, it means that they don’t likely have the same midi number for the pad as for the led that is within it. Usually a button will have the same midi number and work automatically. When it’s different you have to go into the midi out advanced feature and set the “control” to the correct unique number for the pad led and then experiment with the max to find the colour and brightness you want when it’s turned on and min for the couloir and Brightness you want when it’s turned off.
Here is a tip I learnt when testing and finding the pad colour configurations.
If you don’t know the control number then you could potentially have to go through setting it to 127 different values and clicking on test to make it light up before finding the correct value.
The tip is to load a track and then set a cue point to the pad you want to light up. This means that Djay will now be trying to light that pad. Now just move the midi out "control’ slider along until you see the pad light up, that’s your control number. Then move the “max” to find the colour and intensity you want for that pad. Now delete the cue which will turn the pad to the off state, then use “min” to set the colour and intensity when it is off (or set it to 0/off). Ie bright red when on and dim red when off which looks great in the dark (I changed all my other buttons on my NS6ii from 0 on min to 1, so you can still see them in the dark even though their function is off, looks great)
Hope that helps