You might want to use an IN rather than multiple OR statements for the delimiters:

ALTER FUNCTION [dbo].[Casing]

(@instring VARCHAR(255))

RETURNS varchar(1000) AS

BEGIN

   --usage: SELECT dbo.Casing('I AM A DOG')

   DECLARE @strptr INT

   DECLARE @strChar char(1)

   DECLARE @outstring varchar(1000)

   SET @outstring = ''

   SET @strptr = 0

   WHILE @strPtr < len(RTRIM(@instring))

   BEGIN

SET @strptr = @strptr + 1

SET @strchar = SUBSTRING(@instring,@strptr,1)

IF @strptr = 1

SET @outstring = UPPER(@strchar)

ELSE IF SUBSTRING(@instring,(@strptr - 1),1) IN (',', ' ', '''', ']', '[')

SET @outstring = SUBSTRING(@outstring,1,@strptr) + UPPER(@strchar)

ELSE

SET @outstring = SUBSTRING(@outstring,1,@strptr) + LOWER(@strchar)

   END

   RETURN @outstring

END

It will properly format:

PRINT dbo.Casing('this is so cool''yeah,that I think [it] rox!')

We have a similar function, written in a SQL CLR method... It works well and handles things like roman numerals, and some other trickeries...