CINC2021#

Bases: PhysioNetDataBase

Will Two Do? Varying Dimensions in Electrocardiography: The PhysioNet/Computing in Cardiology Challenge 2021

ABOUT

goal: build an algorithm that can classify cardiac abnormalities from either
- twelve-lead (I, II, III, aVR, aVL, aVF, V1, V2, V3, V4, V5, V6)
- six-lead (I, II, III, aVL, aVR, aVF),
- four-lead (I, II, III, V2)
- three-lead (I, II, V2)
- two-lead (I, II)
tranches of data:
- CPSC2018 (tranches A and B of CinC2020, ref. [4]):
  contains 13,256 ECGs (6,877 from tranche A, 3,453 from tranche B), 10,330 ECGs shared as training data, 1,463 retained as validation data, and 1,463 retained as test data. Each recording is between 6 and 144 seconds long with a sampling frequency of 500 Hz
- INCARTDB (tranche C of CinC2020, ref. [4]):
  contains 75 annotated ECGs, all shared as training data, extracted from 32 Holter monitor recordings. Each recording is 30 minutes long with a sampling frequency of 257 Hz
- PTB (PTB and PTB-XL, tranches D and E of CinC2020, ref. [5] and [6]):
  contains 22,353 ECGs, 516 + 21,837, all shared as training data. Each recording is between 10 and 120 seconds long, with a sampling frequency of either 500 (PTB-XL) or 1,000 (PTB) Hz
- Georgia (tranche F of CinC2020, ref. [3]):
  contains 20,678 ECGs, 10,334 ECGs shared as training data, 5,167 retained as validation data, and 5,167 retained as test data. Each recording is between 5 and 10 seconds long with a sampling frequency of 500 Hz
- American (NEW, UNDISCLOSED):
  contains 10,000 ECGs, all retained as test data, geographically distinct from the Georgia database. Perhaps is the main part of the hidden test set of CinC2020
- CUSPHNFH (NEW, the Chapman University, Shaoxing People’s Hospital and Ningbo First Hospital database)
  contains 45,152 ECGS, all shared as training data. Each recording is 10 seconds long with a sampling frequency of 500 Hz this tranche contains two subsets:
  
  Chapman_Shaoxing: “JS00001” - “JS10646”
  
  Ningbo: “JS10647” - “JS45551”
All files can be downloaded from [8] or [9].
only a part of diagnosis_abbr (diseases that appear in the labels of the 6 tranches of training data) are used in the scoring function, while others are ignored. The scored diagnoses were chosen based on prevalence of the diagnoses in the training data, the severity of the diagnoses, and the ability to determine the diagnoses from ECG recordings. The ignored diagnosis_abbr can be put in a a “non-class” group.
the (updated) scoring function has a scoring matrix with nonzero off-diagonal elements. This scoring function reflects the clinical reality that some misdiagnoses are more harmful than others and should be scored accordingly. Moreover, it reflects the fact that confusing some classes is much less harmful than confusing other classes.

all data are recorded in the leads ordering of

["I", "II", "III", "aVR", "aVL", "aVF", "V1", "V2", "V3", "V4", "V5", "V6"]

using for example the following code:

>>> db_dir = "/media/cfs/wenhao71/data/CinC2021/"
>>> working_dir = "./working_dir"
>>> dr = CINC2021(db_dir=db_dir,working_dir=working_dir)
>>> set_leads = []
>>> for tranche, l_rec in dr.all_records.items():
...     for rec in l_rec:
...         ann = dr.load_ann(rec)
...         leads = ann["df_leads"]["lead_name"].values.tolist()
...     if leads not in set_leads:
...         set_leads.append(leads)

Challenge official website [1]. Webpage of the database on PhysioNet [2].

Note

The datasets have been roughly processed to have a uniform format, hence differ from their original resource (e.g. differe in sampling frequency, sample duration, etc.)
The original datasets might have richer metadata (especially those from PhysioNet), which can be fetched from corresponding reader’s docstring or website of the original source
Each sub-dataset might have its own organizing scheme of data, which should be carefully dealt with
There are few “absolute” diagnoses in 12 lead ECGs, where large discrepancies in the interpretation of the ECG can be found even inspected by experts. There is inevitably something lost in translation, especially when you do not have the context. This doesn”t mean making an algorithm isn’t important
The labels are noisy, which one has to deal with in all real world data
each line of the following classes are considered the same (in the scoring matrix):
- RBBB, CRBBB (NOT including IRBBB)
- PAC, SVPB
- PVC, VPB
unfortunately, the newly added tranches (C - F) have baseline drift and are much noisier. In contrast, CPSC data have had baseline removed and have higher SNR

on Aug. 1, 2020, adc gain (including “resolution”, “ADC”? in .hea files) of datasets INCART, PTB, and PTB-xl (tranches C, D, E) are corrected. After correction, (the .tar files of) the 3 datasets are all put in a “WFDB” subfolder. In order to keep the structures consistant, they are moved into “Training_StPetersburg”, “Training_PTB”, “WFDB” as previously. Using the following code, one can check the adc_gain and baselines of each tranche:

>>> db_dir = "/media/cfs/wenhao71/data/CinC2021/"
>>> working_dir = "./working_dir"
>>> dr = CINC2021(db_dir=db_dir,working_dir=working_dir)
>>> resolution = {tranche: set() for tranche in "ABCDEF"}
>>> baseline = {tranche: set() for tranche in "ABCDEF"}
>>> for tranche, l_rec in dr.all_records.items():
...     for rec in l_rec:
...         ann = dr.load_ann(rec)
...         resolution[tranche] = resolution[tranche].union(set(ann["df_leads"]["adc_gain"]))
...         baseline[tranche] = baseline[tranche].union(set(ann["df_leads"]["baseline"]))
>>> print(resolution, baseline)
{"A": {1000.0}, "B": {1000.0}, "C": {1000.0}, "D": {1000.0}, "E": {1000.0}, "F": {1000.0}} {"A": {0}, "B": {0}, "C": {0}, "D": {0}, "E": {0}, "F": {0}}

the .mat files all contain digital signals, which has to be converted to physical values using adc gain, basesline, etc. in corresponding .hea files. wfdb.rdrecord() has already done this conversion, hence greatly simplifies the data loading process. NOTE that there”s a difference when using wfdb.rdrecord(): data from loadmat are in “channel_first” format, while wfdb.rdrecord.p_signal produces data in the “channel_last” format
there are 3 equivalent (2 classes are equivalent if the corr. value in the scoring matrix is 1): (RBBB, CRBBB), (PAC, SVPB), (PVC, VPB)
in the newly (Feb., 2021) created dataset (ref. [7]), header files of each subset were gathered into one separate compressed file. This is due to the fact that updates on the dataset are almost always done in the header files. The correct usage of ref. [7], after uncompressing, is replacing the header files in the folder All_training_WFDB by header files from the 6 folders containing all header files from the 6 subsets. This procedure has to be done, since All_training_WFDB contains the very original headers with baselines: {“A”: {1000.0}, “B”: {1000.0}, “C”: {1000.0}, “D”: {2000000.0}, “E”: {200.0}, “F”: {4880.0}} (the last 3 are NOT correct)
IMPORTANT: organization of the total dataset: either one moves all training records into ONE folder, or at least one moves the subsets Chapman_Shaoxing (WFDB_ChapmanShaoxing) and Ningbo (WFDB_Ningbo) into ONE folder, or use the data WFDB_ShaoxingUniv which is the union of WFDB_ChapmanShaoxing and WFDB_Ningbo

Usage

ECG arrhythmia detection

Issues

reading the .hea files, baselines of all records are 0, however it is not the case if one plot the signal
about half of the LAD records satisfy the “2-lead” criteria, but fail for the “3-lead” criteria, which means that their axis is (-30°, 0°) which is not truely LAD
(Aug. 15, 2020; resolved, and changed to 1000) tranche F, the Georgia subset, has ADC gain 4880 which might be too high. Thus obtained voltages are too low. 1000 might be a suitable (correct) value of ADC gain for this tranche just as the other tranches.
“E04603” (all leads), “E06072” (chest leads, epecially V1-V3), “E06909” (lead V2), “E07675” (lead V3), “E07941” (lead V6), “E08321” (lead V6) has exceptionally large values at rpeaks, reading (load_data) these two records using wfdb would bring in nan values. One can check using the following code
```
>>> rec = "E04603"
>>> dr.plot(rec, dr.load_data(rec, backend="scipy", units="uv"))  # currently raising error
```
many records (headers) have duplicate labels. For example, many records in the Georgia subset has duplicate “PAC” (“284470004”) label
some records in tranche G has #Dx ending with “,” (at least “JS00344”), or consecutive “,” (at least “JS03287”) in corresponding .hea file
tranche G has 2 Dx (“251238007”, “6180003”) which are listed in neither of dx_mapping_scored.csv nor dx_mapping_unscored.csv
about 68 records from tranche G has nan values loaded via wfdb.rdrecord(), which might be caused by motion artefact in some leads
“Q0400”, “Q2961” are completely flat (constant), while many other records have flat leads, especially V1-V6 leads

References

Citation

10.23919/cinc53138.2021.9662687 10.13026/JZ9P-0M02

Parameters:

db_dir (path-like, optional) – Storage path of the database. If not specified, data will be fetched from Physionet.
working_dir (path-like, optional) – Working directory, to store intermediate files and log files.
verbose (int, default 1) – Level of logging verbosity.
kwargs (dict, optional) – Auxilliary key word arguments.

property all_records: Dict[str, List[str]]#: List of all records in the database.

property database_info: DataBaseInfo#: The DataBaseInfo object of the database.

property df_stats: DataFrame#: Dataframe of stats on the whole dataset.

property diagnoses_records_list: Dict[str, List[str]]#: List of all records for each diagnosis.

download() → None[source]#

Download the database from PhysioNet.

Parameters:

compressed (bool, default True) –
Whether to download the compressed database file.

Added in version 0.0.5.
use_s3 (bool, default True) –
Whether to download the database from AWS S3. NOTE: files on AWS S3 are not compressed (larger total size); use_s3 has higher priority than compressed; if AWS client is not available, the database will be downloaded from PhysioNet.

Added in version 0.0.31.

Return type:

None

get_absolute_path(rec: str | int, extension: str | None = None) → Path[source]#

Get the absolute path of the record.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
extension (str, optional) – Extension of the file.

Returns:

abs_fp – Absolute path of the file.

Return type:

pathlib.Path

get_ann_filepath(rec: str | int, with_ext: bool = True) → Path[source]#

Get the absolute file path of the header file of the record.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
with_ext (bool, default True) – If True, the returned file path comes with file extension, otherwise without file extension, which is useful for wfdb functions.

Returns:

Absolute file path of the header file of the record.

Return type:

pathlib.Path

get_data_filepath(rec: str | int, with_ext: bool = True) → Path[source]#

Get the absolute file path of the data file of the record.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
with_ext (bool, default True) – If True, the returned file path comes with file extension, otherwise without file extension, which is useful for wfdb functions.

Returns:

Absolute file path of the data file of the record.

Return type:

pathlib.Path

get_fs(rec: str | int, from_hea: bool = True) → Real[source]#

Get the sampling frequency of the record.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
from_hea (bool, default True) – If True, sampling frequency is read from corresponding header file of the record; otherwise, self.fs is used.

Returns:

fs – Sampling frequency of the record.

Return type:

numbers.Real

get_header_filepath(rec: str | int, with_ext: bool = True) → Path[source]#

Get the absolute file path of the header file of the record.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
with_ext (bool, default True) – If True, the returned file path comes with file extension, otherwise without file extension, which is useful for wfdb functions.

Returns:

Absolute file path of the header file of the record.

Return type:

pathlib.Path

get_labels(rec: str | int, scored_only: bool = True, fmt: str = 's', normalize: bool = True) → List[str][source]#

Get labels (diagnoses or arrhythmias) of the record.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
scored_only (bool, default True) – If True, only get the labels that are scored in the CINC2021 official phase.
fmt (str, default "s") –
Format of labels, one of the following (case insensitive):
- ”a”, abbreviations
- ”f”, full names
- ”s”, SNOMED CT Code
normalize (bool, default True) – If True, the labels will be transformed into their equavalents, which are defined in utils.utils_misc.cinc2021_aux_data.py.

Returns:

labels – The list of labels of the record.

Return type:

List[str]

get_subject_id(rec: str | int) → int[source]#

Attach a unique subject ID for the record.

Parameters:: rec (str or int) – Record name or index of the record in all_records.
Returns:: Subject ID associated with the record.
Return type:: int

get_subject_info(rec: str | int, items: List[str] | None = None) → dict[source]#

Get auxiliary information of a subject (a record) stored in the header files.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
items (List[str], optional) – Items of the subject’s information (e.g. sex, age, etc.).

Returns:

subject_info – Information about the subject, including “age”, “sex”, “medical_prescription”, “history”, “symptom_or_surgery”.

Return type:

dict

get_tranche_class_distribution(tranches: Sequence[str], scored_only: bool = True) → Dict[str, int][source]#

Compute class distribution in the tranches.

Parameters:

tranches (Sequence[str]) – Tranche symbols (A-G).
scored_only (bool, default True) – If True, only classes that are scored in the CINC2021 official phase are considered for computing the distribution.

Returns:

distribution – Distribution of classes in the tranches. Keys are abbrevations of the classes, and values are appearance of corr. classes in the tranche.

Return type:

dict

load_ann(rec: str | int, raw: bool = False) → dict | str[source]#

Load the annotations of the record.

The annotations (header) are stored in the .hea files.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
raw (bool, default False) – If True, the raw annotations without being parsed will be returned.

Returns:

ann_dict – The annotations with items listed in self.ann_items.

Return type:

dict or str

Load physical (converted from digital) ECG data.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
leads (str or int or Sequence[str] or Sequence[int], optional) – The leads to load, can be a single lead or a list of leads.
data_format (str, default "channel_first") – Format of the ECG data, “channel_last” (alias “lead_last”), or “channel_first” (alias “lead_first”)
backend ({"wfdb", "scipy"}, default "wfdb") – The backend data reader.
units (str or None, default "mV") – Units of the output signal, can also be “μV” (aliases “uV”, “muV”). None for digital data, without digital-to-physical conversion.
fs (numbers.Real, optional) – Sampling frequency of the output signal. If not None, the loaded data will be resampled to this frequency, otherwise, the original sampling frequency will be used.
return_fs (bool, default False) – Whether to return the sampling frequency of the output signal.

Returns:

data (numpy.ndarray) – The loaded ECG data.
data_fs (numbers.Real, optional) – Sampling frequency of the output signal. Returned if return_fs is True.

load_header(rec: str | int, raw: bool = False) → dict | str[source]#

Load the annotations of the record.

The annotations (header) are stored in the .hea files.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
raw (bool, default False) – If True, the raw annotations without being parsed will be returned.

Returns:

ann_dict – The annotations with items listed in self.ann_items.

Return type:

dict or str

load_raw_data(rec: str | int, backend: Literal['wfdb', 'scipy'] = 'scipy') → ndarray[source]#

Load raw data from corresponding files with no further processing.

This method facilitates feeding data into the run_12ECG_classifier function.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
backend ({"scipy", "wfdb"}, default "scipy") – The backend data reader. Note that “scipy” provides data in the format of “lead_first”, while “wfdb” provides data in the format of “lead_last”.

Returns:

raw_data – Raw data (d_signal) loaded from corresponding data file, without digital-to-analog conversion (DAC) and resampling.

Return type:

numpy.ndarray

load_resampled_data(rec: str | int, leads: str | List[str] | None = None, data_format: str = 'channel_first', siglen: int | None = None) → ndarray[source]#

Resample the data of rec to 500Hz, or load the resampled data in 500Hz, if the corr. data file already exists

Parameters:

rec (str or int) – Record name or index of the record in all_records.
leads (str or List[str], optional) – The leads of the ECG data to be loaded.
data_format (str, default "channel_first") – Format of the ECG data, “channel_last” (alias “lead_last”), or “channel_first” (alias “lead_first”).
siglen (int, optional) – Signal length, with units in number of samples. If is not None, signal with length longer will be sliced to the length of siglen. Used for preparing/doing model training for example.

Returns:

2D resampled (and perhaps sliced 3D) signal data.

Return type:

numpy.ndarray

Plot the signals of a record or external signals (units in μV), with metadata (fs, labels, tranche, etc.), possibly also along with wave delineations.

Parameters:

rec (str or int) – Record name or index of the record in all_records.
data (numpy.ndarray, optional) – (12-lead) ECG signal to plot, should be of the format “channel_first”, and compatible with leads. If is not None, data of rec will not be used. This is useful when plotting filtered data.
ann (dict, optional) – Annotations for data, with 2 items: “scored”, “all”. Ignored if data is None.
ticks_granularity (int, default 0) – Granularity to plot axis ticks, the higher the more ticks. 0 (no ticks) –> 1 (major ticks) –> 2 (major + minor ticks)
leads (str or List[str], optional) – The leads of the ECG signal to plot.
same_range (bool, default False) – If True, all leads are forced to have the same y range.
waves (dict, optional) – Indices of the wave critical points, including “p_onsets”, “p_peaks”, “p_offsets”, “q_onsets”, “q_peaks”, “r_peaks”, “s_peaks”, “s_offsets”, “t_onsets”, “t_peaks”, “t_offsets”.
kwargs (dict, optional) – Additional keyword arguments to pass to matplotlib.pyplot.plot().

TODO

Slice too long records, and plot separately for each segment.
Plot waves using matplotlib.pyplot.axvspan().

Note

Locator of plt has default MAXTICKS of 1000. If not modifying this number, at most 40 seconds of signal could be plotted once.

Contributors: Jeethan, and WEN Hao

property url: List[str]#: URL of the database index page for downloading.

Added in version 0.0.4.