When we look in the Torah, we have good reason to assume that Sh'minni Atzeret is a part of chag Hassukot. After all, it is described as the "eighth day" of Sukkot. In addition, just as Pesach is a 7 day festival with a yom-tov at its beginning and end, similarly it would seem that sukkot - although 8 days long - also begins with a day of yom-tov (issur melacha) and ends in the same way. So it should be clear that Shminni Atzeret is simply an integrated part of chag haSukkot and will lack its own independent identity.
But certain questions remain.
1. What is "Atzeret"? Why does this day have a special title?
(If we compare this 8th day to the final day of Pesach, the 7th day of Pesach has no special title. Interestingly however, Chazal did give Shavuot the name of "Atzeret" - a name not used in Chumash for Shavuot - and Chazal were thereby declaring Shavuot as the "eighth day" of Pesach. Interesting!)
2. If you compare the Korbanot of Sukkot to Shminni Atzeret, you notice that Shmini Atzeret departs from the pattern of Sukkot . During Sukkot we offer Parim - oxen - in a decreasing scale from 13 to 7. Sh'minni Azeret does not continue the pattern, but instead has just a single Ox. (also 7 as opposed to 14 sheep.) Is this significant? It certainly indicates to us that Shminni Atzeret is distinct in some way.
Chazal distinguish a number of details in which Shminni Atzeret forms its own distinct chag (Found in Maechet Sukka 48a):
1. Birkhat She'hecheyanu
2. No Sukka
3. A different name
4. Distinct Korban
5. The song in the Beit Hamikdash was different. (Lamenatzeach al Hashminit!)
6. The ordering of the Kohanim in the mikdash was organised anew for this day and was not incorporated in the Sukkot order of things
(For more detail, see Rav S.Y. Zevin in his wonderful work Hamoadim Bahalacha.)
So to what extent is Sh'minni Atzeret part of Sukkot and to what extent is it an independent chag?
There is a certain "nafka mina" (practical consequence of a theoretical argument) here which has central importance regarding our nusach hatefilla (prayer text). This is an argument regarding the proper title of this chag in our tefilla.
One opinion calls it "shminni chag h'atzeret hazeh". This indicates that Sh'minni Atzeret has its own independent status of chag.
But the second "nusach" reads "shminni atzeret hachag hazeh" ; i.e. The eighth day - end of this (sukkot) festival. Thus sh'minni atzeret is subsidiary to Sukkot and thereby denied its own standing as a chag.
A resolution to this ancient problem is beyond my capability. The likelihood is that Sh'minni Atzeret contains certain independent element, and also is somehow connected and similar to Chag HaSukkot. However, let us look at Rashi's interpretation of the term "Atzeret" and we will see something rather interesting in Rashi's commentary.
"Atzeret: I have stopped you (from leaving and kept you) with me. It is comparable to a king who invited all his children to a banquet. The banquet continued for a number of days. When the time came for everyone to go home, the king said, 'my special child, I am making a special request for you to stay with me for a little more time. It is difficult for me to part from you.' " (Rashi in Vayikra 23)
"...The aggadic explanation: All the days of the (Sukkot) festival, they offered sacrifices in the name of the seventy nations of the world. They wished to go, God said to them, 'Please stay so that I can enjoy you!'" (Rashi on Bamidbar)
The imagery here of the party for the world and then an intimate get-together with Am Yisrael is taken from the korbanot of Sukkot. During Sukkot we sacrifice 70 oxen in total. On Sheminni Atzeret only one ox is offered. The number 70 is symbolic of the world community (70 nations). What might the single ox symbolise if not the Jewish nation - "a people who dwell alone and will not be counted amongst the nations." Indeed, Sukkot is portrayed by the prophet Zechariah (See the haftara for 1st day Sukkot - Zechariah Ch.14) as the time when in the future, all nations will arrive in Jerusalem to salute God and recognise Him as the Master of the Universe.
But the Midrashim are different. In the Vayikra quote, the banquet which Am Yisrael partcipate in is a CONTINUATION of the earlier party. It is not anew event with any distinct features. Israel simply stays an extra day because God doesn't want them to leave.
In Bamidbar however, Rashi's Midrash portrays Sheminni Atzeret as a DISTINCT and INDEPENDENT chag. Whereas Sukkot is a universal holiday according to this midrash, Sheminni Atzeret is chosen as a day fo intimacy between God and Israel. It is particularistic in nature, It is not a continuation of what came before but rather, a radical departure from the previous atmosphere. It is a small nuance in the Midrashim which distinguish between Shminni Atzeret as an independent event or rather, the tail-end of Sukkot.
THE NATURE OF SHEMINNI ATZERET.
If sh'mmini atzeret does have its own independent character, what is that character?
One option might follow Rashi's midrash and just see this day as a day of connection between God and Israel; a celebration of Israel's chosen-ness and legacy. In that case, Simchat Torah is certainly appropriately timed in that we celebrate the covenant between ourselves and God.
But I did find one commentator who focuses upon a particular focus of shminni atzeret.
The Targum Yonatan ben Uziel (Vayikra 23) says: "It shall be a gathering for you to pray for rain before God on the eighth day." Here, we see that Sh'minni Atzeret takes in a more sombre tone. In contrast to the rejoicing of Sukkot, Yonatan ben Uziel sees Shminni Atzeret as a day of supplication for the winter rains. This clearly relates to our practice of Tefillat Geshem on Shminni Atzeret. It also explains why the korbanot of Shmini Atzeret match those of Rosh Hashanna and Yom Kippur for these days all share something in common; they are days of judgement and prayer for life and death. Rain which controls agriculture is a life and death issue.
One further thing. If we do follow R. Yonatan ben Uziel, there is a further conrast between Sukkot and Shminni Atzeret. Sukkot is the Chag HeAssif - the ingathering festival. The ingathering festival celebrates the successes of the past season. It looks at the crops stored away in the barn and expresses that satisfaction of a year of prosperity in the form of Hallel and thanksgiving to God. Chag Ha'Asif looks backwards at the year just ended.
But if Sheminni Atzeret is a day of prayer for rain, then Sheminni Atzeret is not looking BACK in time. It looks FORWARD in time, towards the year ahead, praying to God that the coming year will be one of prosperity.