The effect you are seeing is typical of summertime performance. As heat enters the room from solar and other gains the fabric heats up. In doing so it takes heat from the room, which you see as a negative fabric gain. This is a dynamic effect which typically outweighs fabric heat transfer effects related to outside temperature (these are heavily damped as they pass through the external fabric). It is most pronounced in rooms with significant thermal mass, since it is this mass which, as it warms up, absorbs the heat.