Inputs to lazyFileReader may look like text files (their contents
are essentially ASCII), but they may not necessarily obey the
platform rules for text files, such as terminating each line with
the CR LF control sequence on Windows platforms.
Also, the file position operations which may be performed on text
files are severely limited by the C and C++ language standards. For
instance, Microsoft documentation states that
1. ifstream::seekg will call basic_filebuf::seekpos.
basic_filebuf::seekpos is in turn implemented in terms of C fsetpos
on Windows.
https://msdn.microsoft.com/en-us/library/xx21a6ww(v=vs.110).aspx
2. The C standard specifies that it is undefined behavior if the
"fsetpos function is called to set a position that was not returned
by a previous successful call to the fgetpos function on a stream
associated with the same file (7.21.9.3)." [ISO/IEC 9899:2011]
3. Similarly, ifstream::unget() may call basic_filebuf::pbackfail()
which on Windows is implemented in terms of ungetc().
https://msdn.microsoft.com/en-us/library/x0kf6337(v=vs.110).aspxhttps://msdn.microsoft.com/en-us/library/29hykw2y(v=vs.110).aspx
4. Microsoft documents the following about how file positions are
maintained when ungetc() is used on text streams:
On a successful ungetc call against a text stream, the
file-position indicator is unspecified until all the
pushed-back characters are read or discarded.
5. In stepcode, unget() and seekg() are freely mixed without regard
to whether undefined behavior is encountered as a result of
[1-4]
The idiom
char c = ...;
_userMsg.append( &c );
is not correct C++, because it treats the address of 'c' as a NUL-
terminated C string. However, this is not guaranteed.
When building and testing on Debian Stretch with AddressSanitizer:
ASAN_OPTIONS="detect_leaks=false" CXX="clang++" CC=clang CXXFLAGS="-fsanitize=address" LDFLAGS="-fsanitize=address" cmake .. -DSC_ENABLE_TESTING=ON -DSC_BUILD_SCHEMAS="ifc2x3;ap214e3;ap209"
ASAN_OPTIONS="detect_leaks=false" make
ASAN_OPTIONS="detect_leaks=false" ctest . --output-on-failure
an error like the following is encountered:
==15739==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffeb2ca7621 at pc 0x00000043c943 bp 0x7ffeb2ca75d0 sp 0x7ffeb2ca6d80
READ of size 33 at 0x7ffeb2ca7621 thread T0
#0 0x43c942 in __interceptor_strlen.part.45 (/home/jepler/src/stepcode/build/bin/lazy_sdai_ap214e3+0x43c942)
#1 0x7fb9056e6143 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(char const*) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x11f143)
#2 0x7fb905b677c3 in ErrorDescriptor::AppendToDetailMsg(char) /home/jepler/src/stepcode/src/clutils/errordesc.cc:150:5
Address 0x7ffeb2ca7621 is located in stack of thread T0 at offset 33 in frame
#0 0x7fb905b676af in ErrorDescriptor::AppendToDetailMsg(char) /home/jepler/src/stepcode/src/clutils/errordesc.cc:149
This frame has 1 object(s):
[32, 33) '' <== Memory access at offset 33 overflows this variable
A similar problem with AppendToUserMsg is found by inspection.
After this change, all 200 tests pass under the AddressSanitizer
configuration
On Debian Stretch, when configuring stepcode like so:
ASAN_OPTIONS="detect_leaks=false" CXX="clang++" CXXFLAGS="-fsanitize=address" cmake ..
a fatal error would be detected:
==29661==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x62100001dca0 at pc 0x0000004435e3 bp 0x7ffed6d9cae0 sp 0x7ffed6d9c290
READ of size 4001 at 0x62100001dca0 thread T0
#0 0x4435e2 in __interceptor_strlen.part.45 (/home/jepler/src/stepcode/build/bin/schema_scanner+0x4435e2)
#1 0x501d7b in ERRORreport_with_symbol /home/jepler/src/stepcode/src/express/error.c:413
0x62100001dca0 is located 0 bytes to the right of 4000-byte region
[0x62100001cd00,0x62100001dca0)
allocated by thread T0 here:
#0 0x4c3ae8 in __interceptor_malloc (/home/jepler/src/stepcode/build/bin/schema_scanner+0x4c3ae8)
#1 0x5011fc in ERRORinitialize /home/jepler/src/stepcode/src/express/error.c:129
Operations on ERROR_string were unsafe, because they did not guard
against accesses beyond the end of the allocatd region.
This patch ensures that all accesses via *printf functions do respect
the end of the buffer; and encapsulates the routine for pointing
ERROR_string at the space for the next error text to start, if space is
available.
Finally, because it was found with search and replace, a stray manipulation
of ERROR_string within the print-to-file branch of the code is removed.
This stray line would have had the effect of moving ERROR_string one byte
further along at every warning-to-file, which could also have been a
cause of the problem here.
On Windows, concurrent access to files is severely restricted
compared to standard operating systems. When ctest is invoking
cmake, this causes it to write simultaneously to the same files in
each concurrent cmake invocation, leading to spurious test failures
like
error MSB3491: Could not write lines to file "...". The process
cannot access the file '...' because it is being used by another
process.
Explicitly ask for no parallelism with "-j1", even though it is
probably the default.
On this platform, TEST_NULLPTR fails, even though nullptr and
nullptr_t are supported:
/home/jepler/src/stepcode/build/CMakeFiles/CMakeTmp/src.cxx:4:23:
error: converting to 'bool' from 'std::nullptr_t'
requires direct-initialization [-fpermissive]
int main() {return !!f();}
~^~
Subsequent to this failure, the workaround definitions in sc_nullptr.h
prevent standard C++ headers (which must refer to real nullptr) to fail.
The failure occurs because the C++ standard apparently does not state
that operator! may be used on nullptr. Despite this, some compilers
have historically allowed it. g++ 6.3's behavior appears to be aligned
with the standard.
As requested by @brlcad, ensure that the function 'f' is used from main,
to avoid a clever (but not nullptr-supporting) compiler from somehow
skipping 'f' altogether, creating a false positive for nullptr support.
