On Windows, when a file is opened in "text" mode, but it actually
contains Unix-style line endings, the behavior of tellg() is
unexpected.
Consider this program which puts the (binary) contents "a\nb\n" in a
file, then opens it in text mode for reading. It prints each
character read, along with the value returned by tellg():
#include <iostream>
#include <fstream>
int main()
{
{
std::ofstream f("myfile.txt", std::ios::binary);
f << "a\nb\n";
}
std::ifstream f("myfile.txt");
for (char c=0; f.get(c);)
std::cout << f.tellg() << ' ' << int(c) << '\n';
}
On a UNIX platform which does not have a distinction between "text"
and "binary" files, the output will read
1 97
2 10
3 98
4 10
because the file position simply advances one position after each
byte is read.
On Windows with the Visual Studio C and C++ runtime, the result is
instead
-1 97
1 10
2 98
4 10
While it is impossible to say exactly what the Windows runtime is
doing here, it appears that it is trying to adjust for the mismatch
between "number of bytes read in byte oriented mode and "number of
bytes read in text mode".
Since "part21" files don't necessarily contain CRLF line endings
when viewed in binary mode, open the file in binary mode. This
fixes the test failure seen on appveyor ci running the
"test_inverse_attr3" test.
The idiom
char c = ...;
_userMsg.append( &c );
is not correct C++, because it treats the address of 'c' as a NUL-
terminated C string. However, this is not guaranteed.
When building and testing on Debian Stretch with AddressSanitizer:
ASAN_OPTIONS="detect_leaks=false" CXX="clang++" CC=clang CXXFLAGS="-fsanitize=address" LDFLAGS="-fsanitize=address" cmake .. -DSC_ENABLE_TESTING=ON -DSC_BUILD_SCHEMAS="ifc2x3;ap214e3;ap209"
ASAN_OPTIONS="detect_leaks=false" make
ASAN_OPTIONS="detect_leaks=false" ctest . --output-on-failure
an error like the following is encountered:
==15739==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffeb2ca7621 at pc 0x00000043c943 bp 0x7ffeb2ca75d0 sp 0x7ffeb2ca6d80
READ of size 33 at 0x7ffeb2ca7621 thread T0
#0 0x43c942 in __interceptor_strlen.part.45 (/home/jepler/src/stepcode/build/bin/lazy_sdai_ap214e3+0x43c942)
#1 0x7fb9056e6143 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(char const*) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x11f143)
#2 0x7fb905b677c3 in ErrorDescriptor::AppendToDetailMsg(char) /home/jepler/src/stepcode/src/clutils/errordesc.cc:150:5
Address 0x7ffeb2ca7621 is located in stack of thread T0 at offset 33 in frame
#0 0x7fb905b676af in ErrorDescriptor::AppendToDetailMsg(char) /home/jepler/src/stepcode/src/clutils/errordesc.cc:149
This frame has 1 object(s):
[32, 33) '' <== Memory access at offset 33 overflows this variable
A similar problem with AppendToUserMsg is found by inspection.
After this change, all 200 tests pass under the AddressSanitizer
configuration
On Debian Stretch, when configuring stepcode like so:
ASAN_OPTIONS="detect_leaks=false" CXX="clang++" CXXFLAGS="-fsanitize=address" cmake ..
a fatal error would be detected:
==29661==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x62100001dca0 at pc 0x0000004435e3 bp 0x7ffed6d9cae0 sp 0x7ffed6d9c290
READ of size 4001 at 0x62100001dca0 thread T0
#0 0x4435e2 in __interceptor_strlen.part.45 (/home/jepler/src/stepcode/build/bin/schema_scanner+0x4435e2)
#1 0x501d7b in ERRORreport_with_symbol /home/jepler/src/stepcode/src/express/error.c:413
0x62100001dca0 is located 0 bytes to the right of 4000-byte region
[0x62100001cd00,0x62100001dca0)
allocated by thread T0 here:
#0 0x4c3ae8 in __interceptor_malloc (/home/jepler/src/stepcode/build/bin/schema_scanner+0x4c3ae8)
#1 0x5011fc in ERRORinitialize /home/jepler/src/stepcode/src/express/error.c:129
Operations on ERROR_string were unsafe, because they did not guard
against accesses beyond the end of the allocatd region.
This patch ensures that all accesses via *printf functions do respect
the end of the buffer; and encapsulates the routine for pointing
ERROR_string at the space for the next error text to start, if space is
available.
Finally, because it was found with search and replace, a stray manipulation
of ERROR_string within the print-to-file branch of the code is removed.
This stray line would have had the effect of moving ERROR_string one byte
further along at every warning-to-file, which could also have been a
cause of the problem here.
In #359 I identify a race condition between multiple parallel invocations
of cmake, which can arise naturally during ctests. Now that the file
contents will not change without an intervening git commit, it is
sufficient to ensure that the parallel invocations use distinct temporary
file names with high probability.
As analyzed in #359, if the header contains the current time, it will
be updated while running the testsuite; this, in turn, causes multiple
cmake processes to attempt to update targets like lib/libexpress.so.2.0.0
at the same time, causing test failures.
This fixes the failure in test_inverse_attr3 seen on travis ci's osx
build.
Actually, only the change to sectionReader::getRealInstance is
needed to fix the test, but as the reason that 'unget' can fail is
unclear, I changed all instances of 'unget' to use the 'seekg' +
arithmetic method instead.
I failed to find a reason why 'unget' could fail in this way, or
reports of macos-specific failures in 'unget', but I was not
enlightened.
I do not know whether test_inverse_attr3 would *consistently* hang
on Appveyor, but after this change (and modifying .appveyor.yml
to not skip test_inverse_attr3) it did succeed on the first try.
On Windows, concurrent access to files is severely restricted
compared to standard operating systems. When ctest is invoking
cmake, this causes it to write simultaneously to the same files in
each concurrent cmake invocation, leading to spurious test failures
like
error MSB3491: Could not write lines to file "...". The process
cannot access the file '...' because it is being used by another
process.
Explicitly ask for no parallelism with "-j1", even though it is
probably the default.
On this platform, TEST_NULLPTR fails, even though nullptr and
nullptr_t are supported:
/home/jepler/src/stepcode/build/CMakeFiles/CMakeTmp/src.cxx:4:23:
error: converting to 'bool' from 'std::nullptr_t'
requires direct-initialization [-fpermissive]
int main() {return !!f();}
~^~
Subsequent to this failure, the workaround definitions in sc_nullptr.h
prevent standard C++ headers (which must refer to real nullptr) to fail.
The failure occurs because the C++ standard apparently does not state
that operator! may be used on nullptr. Despite this, some compilers
have historically allowed it. g++ 6.3's behavior appears to be aligned
with the standard.
As requested by @brlcad, ensure that the function 'f' is used from main,
to avoid a clever (but not nullptr-supporting) compiler from somehow
skipping 'f' altogether, creating a false positive for nullptr support.
