James Madison made the statement in the context of a discourse on religious proclamations.
Religious proclamations by the Executive recommending thanksgivings & fasts are shoots from the same root with the legislative acts [the attempt in Virginia to establish a general assessment for the support of all Christian sects, the appointment of Chaplains to the two Houses of Congress and the Chaplainships for the army and navy] reviewed.
Altho' recommendations only, they imply a religious agency, making no part of the trust delegated to political rulers.
The objections to them are 1. that Govts ought not to interpose in relation to those subject to their authority but in cases where they can do it with effect. An advisory Govt is a contradiction in terms. 2. The members of a Govt as such can in no sense, be regarded as possessing an advisory trust from their Constituents in their religious capacities. They cannot form an ecclesiastical Assembly, Convocation, Council, or Synod, and as such issue decrees or injunctions addressed to the faith or the Consciences of the people. In their individual capacities, as distinct from their official station, they might unite in recommendations of any sort whatever, in the same manner as any other individuals might do. But then their recommendations ought to express the true character from which they emanate. 3. They seem to imply and certainly nourish the erronious idea of a national religion. The idea just as it related to the Jewish nation under a theocracy, having been improperly adopted by so many nations which have embraced Xnity, is too apt to lurk in the bosoms even of Americans, who in general are aware of the distinction between religious & political societies. The idea also of a union of all to form one nation under one Govt in acts of devotion to the God of all is an imposing idea.
