Inheritance diagram for utils.BaseCoreDumpArchiver:

Public Member Functions
	__init__ (self, search_dir, output_directory)

	__enter__ (self)

	__exit__ (self, *_)

Protected Member Functions
	_safe_cleanup (self)

	_archive (self, crashes)

	_is_shard (self)

	_report_missing_crashes (self, missing, throw=False)

	_get_file_name (self, file)

	_move (self, files)

	_tar (self, file)

	_upload (self, files)

	_find_all_coredumps (self)

	_find_unexpected_crashes (self)

	_cleanup (self)

Protected Attributes
	_bucket

	_binaries_dir

	_search_dir

	_output_directory

Static Protected Attributes
str	_UNEXPECTED_CRASHES_FILE = 'unexpected-crashes'

Detailed Description

This class reads coredumps file written by UnexpectedCrashDumpArchiver
into the current working directory and uploads all cores and binaries
listed in it into Cloud Storage (see
pkg/test_runner/lib/src/test_progress.dart).

Definition at line 610 of file utils.py.

Constructor & Destructor Documentation

◆ init()

utils.BaseCoreDumpArchiver.__init__	(	self,
		search_dir,
		output_directory
	)

Reimplemented in utils.LinuxCoreDumpArchiver, utils.MacOSCoreDumpArchiver, utils.WindowsCoreDumpArchiver, and utils.PosixCoreDumpArchiver.

Definition at line 620 of file utils.py.

    def __init__(self, search_dir, output_directory):
        self._bucket = 'dart-temp-crash-archive'
        self._binaries_dir = os.getcwd()
        self._search_dir = search_dir
        self._output_directory = output_directory
 

Member Function Documentation

◆ enter()

utils.BaseCoreDumpArchiver.__enter__ ( self )

Definition at line 633 of file utils.py.

    def __enter__(self):
        print('INFO: Core dump archiving is activated')
 
        # Cleanup any stale files
        if self._safe_cleanup():
            print('WARNING: Found and removed stale coredumps')
 

◆ exit()

utils.BaseCoreDumpArchiver.__exit__	(		self,
		*	_
	)

Reimplemented in utils.WindowsCoreDumpArchiver.

Definition at line 640 of file utils.py.

    def __exit__(self, *_):
        try:
            crashes = self._find_unexpected_crashes()
            if crashes:
                # If we get a ton of crashes, only archive 10 dumps.
                archive_crashes = crashes[:10]
                print('Archiving coredumps for crash (if possible):')
                for crash in archive_crashes:
                    print('----> {}'.format(crash))
 
                sys.stdout.flush()
 
                self._archive(archive_crashes)
            else:
                print('INFO: No unexpected crashes recorded')
                dumps = self._find_all_coredumps()
                if dumps:
                    print('INFO: However there are {} core dumps found'.format(
                        len(dumps)))
                    for dump in dumps:
                        print('INFO:        -> {}'.format(dump))
                    print()
        except Exception as error:
            print('ERROR: Failed to archive crashes: {}'.format(error))
            raise
 
        finally:
            self._safe_cleanup()
 

◆ _archive()

utils.BaseCoreDumpArchiver._archive	(	self,
		crashes
	)

protected

Definition at line 669 of file utils.py.

    def _archive(self, crashes):
        files = set()
        missing = []
        for crash in crashes:
            files.update(crash.binaries)
            core = self._find_coredump_file(crash)
            if core:
                files.add(core)
            else:
                missing.append(crash)
        if self._output_directory is not None and self._is_shard():
            print(
                "INFO: Moving collected dumps and binaries into output directory\n"
                "INFO: They will be uploaded to isolate server. Look for \"isolated"
                " out\" under the failed step on the build page.\n"
                "INFO: For more information see runtime/docs/infra/coredumps.md"
            )
            self._move(files)
        else:
            print(
                "INFO: Uploading collected dumps and binaries into Cloud Storage\n"
                "INFO: Use `gsutil.py cp from-url to-path` to download them.\n"
                "INFO: For more information see runtime/docs/infra/coredumps.md"
            )
            self._upload(files)
 
        if missing:
            self._report_missing_crashes(missing, throw=False)
 

◆ _cleanup()

utils.BaseCoreDumpArchiver._cleanup ( self )

protected

Reimplemented in utils.PosixCoreDumpArchiver, and utils.WindowsCoreDumpArchiver.

Definition at line 801 of file utils.py.

    def _cleanup(self):
        found = False
        if os.path.exists(BaseCoreDumpArchiver._UNEXPECTED_CRASHES_FILE):
            os.unlink(BaseCoreDumpArchiver._UNEXPECTED_CRASHES_FILE)
            found = True
        for binary in glob.glob(os.path.join(self._binaries_dir, 'binary.*')):
            found = True
            TryUnlink(binary)
 
        return found
 
 

◆ _find_all_coredumps()

utils.BaseCoreDumpArchiver._find_all_coredumps ( self )

protected

Return coredumps that were recorded (if supported by the platform).
This method will be overridden by concrete platform specific implementations.

Reimplemented in utils.WindowsCoreDumpArchiver.

Definition at line 781 of file utils.py.

    def _find_all_coredumps(self):
        """Return coredumps that were recorded (if supported by the platform).
        This method will be overridden by concrete platform specific implementations.
        """
        return []
 

◆ _find_unexpected_crashes()

utils.BaseCoreDumpArchiver._find_unexpected_crashes ( self )

protected

Load coredumps file. Each line has the following format:

test-name,pid,binary-file1,binary-file2,...

Definition at line 787 of file utils.py.

    def _find_unexpected_crashes(self):
        """Load coredumps file. Each line has the following format:
 
        test-name,pid,binary-file1,binary-file2,...
        """
        try:
            with open(BaseCoreDumpArchiver._UNEXPECTED_CRASHES_FILE) as f:
                return [
                    UnexpectedCrash(*ln.strip('\n').split(','))
                    for ln in f.readlines()
                ]
        except:
            return []
 

◆ _get_file_name()

utils.BaseCoreDumpArchiver._get_file_name	(	self,
		file
	)

protected

Definition at line 717 of file utils.py.

    def _get_file_name(self, file):
        # Sanitize the name: actual cores follow 'core.%d' pattern, crashed
        # binaries are copied next to cores and named
        # 'binary.<mode>_<arch>_<binary_name>'.
        # This should match the code in testing/dart/test_progress.dart
        name = os.path.basename(file)
        (prefix, suffix) = name.split('.', 1)
        is_binary = prefix == 'binary'
        if is_binary:
            (mode, arch, binary_name) = suffix.split('_', 2)
            name = binary_name
        return (name, is_binary)
 

◆ _is_shard()

utils.BaseCoreDumpArchiver._is_shard ( self )

protected

Definition at line 699 of file utils.py.

    def _is_shard(self):
        return 'BUILDBOT_BUILDERNAME' not in os.environ
 

◆ _move()

utils.BaseCoreDumpArchiver._move	(	self,
		files
	)

protected

Definition at line 730 of file utils.py.

    def _move(self, files):
        for file in files:
            print('+++ Moving {} to output_directory ({})'.format(
                file, self._output_directory))
            (name, is_binary) = self._get_file_name(file)
            destination = os.path.join(self._output_directory, name)
            shutil.move(file, destination)
            if is_binary and os.path.exists(file + '.pdb'):
                # Also move a PDB file if there is one.
                pdb = os.path.join(self._output_directory, name + '.pdb')
                shutil.move(file + '.pdb', pdb)
 

◆ _report_missing_crashes()

utils.BaseCoreDumpArchiver._report_missing_crashes	(	self,
		missing,
		throw = `False`
	)

protected

Reimplemented in utils.WindowsCoreDumpArchiver.

Definition at line 702 of file utils.py.

    def _report_missing_crashes(self, missing, throw=False):
        missing_as_string = ', '.join([str(c) for c in missing])
        other_files = list(glob.glob(os.path.join(self._search_dir, '*')))
        sys.stderr.write(
            "Could not find crash dumps for '{}' in search directory '{}'.\n"
            "Existing files which *did not* match the pattern inside the search "
            "directory are are:\n  {}\n".format(missing_as_string,
                                                self._search_dir,
                                                '\n  '.join(other_files)))
        # TODO: Figure out why windows coredump generation does not work.
        # See http://dartbug.com/36469
        if throw and GuessOS() != 'win32':
            raise Exception(
                'Missing crash dumps for: {}'.format(missing_as_string))
 

◆ _safe_cleanup()

utils.BaseCoreDumpArchiver._safe_cleanup ( self )

protected

Definition at line 626 of file utils.py.

    def _safe_cleanup(self):
        try:
            return self._cleanup()
        except Exception as error:
            print('ERROR: Failure during cleanup: {}'.format(error))
            return False
 

◆ _tar()

utils.BaseCoreDumpArchiver._tar	(	self,
		file
	)

protected

Definition at line 742 of file utils.py.

    def _tar(self, file):
        (name, is_binary) = self._get_file_name(file)
        tarname = '{}.tar.gz'.format(name)
 
        # Compress the file.
        tar = tarfile.open(tarname, mode='w:gz')
        tar.add(file, arcname=name)
        if is_binary and os.path.exists(file + '.pdb'):
            # Also add a PDB file if there is one.
            tar.add(file + '.pdb', arcname=name + '.pdb')
        tar.close()
        return tarname
 

◆ _upload()

utils.BaseCoreDumpArchiver._upload	(	self,
		files
	)

protected

Definition at line 755 of file utils.py.

    def _upload(self, files):
        bot_utils = GetBotUtils()
        gsutil = bot_utils.GSUtil()
        storage_path = '{}/{}/'.format(self._bucket, uuid.uuid4())
        gs_prefix = 'gs://{}'.format(storage_path)
        http_prefix = 'https://storage.cloud.google.com/{}'.format(storage_path)
 
        print('\n--- Uploading into {} ({}) ---'.format(gs_prefix, http_prefix))
        for file in files:
            tarname = self._tar(file)
 
            # Remove / from absolute path to not have // in gs path.
            gs_url = '{}{}'.format(gs_prefix, tarname)
            http_url = '{}{}'.format(http_prefix, tarname)
 
            try:
                gsutil.upload(tarname, gs_url)
                print('+++ Uploaded {} ({})'.format(gs_url, http_url))
            except Exception as error:
                print('!!! Failed to upload {}, error: {}'.format(
                    tarname, error))
 
            TryUnlink(tarname)
 
        print('--- Done ---\n')
 

Member Data Documentation

◆ _binaries_dir

utils.BaseCoreDumpArchiver._binaries_dir

protected

Definition at line 622 of file utils.py.

◆ _bucket

utils.BaseCoreDumpArchiver._bucket

protected

Definition at line 621 of file utils.py.

◆ _output_directory

utils.BaseCoreDumpArchiver._output_directory

protected

Definition at line 624 of file utils.py.

◆ _search_dir

utils.BaseCoreDumpArchiver._search_dir

protected

Definition at line 623 of file utils.py.

◆ _UNEXPECTED_CRASHES_FILE

str utils.BaseCoreDumpArchiver._UNEXPECTED_CRASHES_FILE = 'unexpected-crashes'

staticprotected

Definition at line 618 of file utils.py.

The documentation for this class was generated from the following file:

third_party/dart-lang/sdk/tools/utils.py

Public Member Functions

Protected Member Functions

Protected Attributes

Static Protected Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ __enter__()

◆ __exit__()

◆ _archive()

◆ _cleanup()

◆ _find_all_coredumps()

◆ _find_unexpected_crashes()

◆ _get_file_name()

◆ _is_shard()

◆ _move()

◆ _report_missing_crashes()

◆ _safe_cleanup()

◆ _tar()

◆ _upload()

Member Data Documentation

◆ _binaries_dir

◆ _bucket

◆ _output_directory

◆ _search_dir

◆ _UNEXPECTED_CRASHES_FILE

◆ init()

◆ enter()

◆ exit()